#790: Matroska audio oracle checks the encoded signal, robust to older ffmpeg

ffmpeg's CLI handles Matroska priming differently by version: 9.x trims
CodecDelay, 2023-24 static builds keep the priming samples (AAC 1024,
AC-3 / E-AC-3 256, MP3 1105), so a +-40-sample best-lag search against
its decode landed on a spurious minimum (4 or 24 samples).

matroska_audio_oracle now has ffmpeg encode a reference signal we
generate (low-passed xorshift noise) and asserts our decode lines up
with that signal to the sample, the stream's own CodecDelay trimmed:
version-independent, and it fails on the old ms-rounded trim (-16).
ffmpeg's decode is compared after a best-lag search bounded by the
CodecDelay; its lag is reported, the waveform must match (< 1e-3).
E-AC-3 joins the cases. eac3_oracle's .mkv case accepts lag 0 or the
256-sample priming; MP4 / MOV / TS stay strictly at lag 0.

Verified with ffmpeg 9.0.2 and the static N-71064 (2024-06) build.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

crates/codecs/tests/eac3_oracle.rs

@@ -53,12 +53,18 @@ fn eac3_in_mp4_matroska_and_transport_streams() {
         };
         let ours = src.audio(start, n, 48_000).unwrap().channels;
         assert_eq!(ours.len(), ch, "{name}");
-        // Matroska stores CodecDelay (the encoder's 256-sample priming) in ns and timestamps in
+        // Matroska stores the encoder's 256-sample priming as CodecDelay, which we trim exactly
-        // ms, and we place packets at the rounded value (240 samples): a shared Matroska timing
+        // (checked to the sample against the encoded signal in `matroska_audio_oracle`). Whether
-        // limit (AAC and AC-3 alike), not a decoding one. Elsewhere the samples line up exactly.
+        // ffmpeg's CLI trims it too depends on its version (9.x does; 2023-24 builds keep the
-        let (worst, lag) = (-32..=32).map(|lag| (snr(&ours, &want, ch, lag), lag)).fold((f64::MIN, 0), |a, b| if b.0 > a.0 { b } else { a });
+        // priming samples), so for .mkv its decode may start 256 samples earlier than ours.
-        let max_lag = if name.ends_with(".mkv") { 16 } else { 0 };
+        // Elsewhere the samples line up exactly, whatever the version.
-        assert!(lag.abs() <= max_lag, "{name}: decode is {lag} samples off ffmpeg's");
+        let lags: Vec<i64> = match name.ends_with(".mkv") {
+            true => (-32..=32).chain(-256 - 32..=-256 + 32).collect(),
+            false => (-32..=32).collect(),
+        };
+        let (worst, lag) = lags.into_iter().map(|lag| (snr(&ours, &want, ch, lag), lag)).fold((f64::MIN, 0), |a, b| if b.0 > a.0 { b } else { a });
+        let ok = if name.ends_with(".mkv") { lag == 0 || lag == -256 } else { lag == 0 };
+        assert!(ok, "{name}: decode is {lag} samples off ffmpeg's");
         // a window read on a fresh source (random access) gives the same samples
         let fresh = filmcraft_codecs::open_bytes(name, bytes(&f)).unwrap();
         let at = n / 3;

crates/codecs/tests/matroska_audio_oracle.rs

@@ -49,45 +49,95 @@ fn matroska_audio_plays_without_holes() {
     }
 }
 
-/// The lag in `-max..=max` at which `ours` best matches `theirs` (least mean |difference| over
+/// Mean |difference| between `ours[i]` and `theirs[i + lag]` over `from..from + len`.
-/// `from..from + len`).
+fn mean_err(ours: &[f32], theirs: &[f32], from: usize, len: usize, lag: i64) -> f64 {
-fn best_lag(ours: &[f32], theirs: &[f32], from: usize, len: usize, max: i64) -> i64 {
+    (from..from + len).map(|i| (ours[i] as f64 - theirs[(i as i64 + lag) as usize] as f64).abs()).sum::<f64>() / len as f64
-    let err = |lag: i64| -> f64 { (from..from + len).map(|i| (ours[i] as f64 - theirs[(i as i64 + lag) as usize] as f64).abs()).sum::<f64>() / len as f64 };
+}
-    (-max..=max).min_by(|&a, &b| err(a).total_cmp(&err(b))).unwrap()
+
+/// The lag in `lags` at which `ours` best matches `theirs` (least [`mean_err`]), and that error.
+fn best_lag(ours: &[f32], theirs: &[f32], from: usize, len: usize, lags: std::ops::RangeInclusive<i64>) -> (i64, f64) {
+    lags.map(|lag| (lag, mean_err(ours, theirs, from, len, lag))).min_by(|a, b| a.1.total_cmp(&b.1)).unwrap()
+}
+
+/// A reference signal we control: low-passed white noise from a fixed xorshift seed. Its
+/// autocorrelation dies out within a few samples, so even a one-sample misalignment shows.
+fn noise(n: usize) -> Vec<f32> {
+    let (mut state, mut y) = (0x9E37_79B9_7F4A_7C15u64, 0f32);
+    (0..n)
+        .map(|_| {
+            state ^= state << 13;
+            state ^= state >> 7;
+            state ^= state << 17;
+            let white = (state >> 40) as f32 / (1u64 << 24) as f32 - 0.5;
+            y = 0.5 * y + 0.4 * white;
+            y
+        })
+        .collect()
 }
 
 /// #790: ffmpeg stores the encoder priming as `CodecDelay` (1024 samples = 21.333 ms for AAC at
 /// 48 kHz, 256 = 5.805 ms for AC-3 at 44.1 kHz, 1105 = 25.057 ms for MP3), and block timestamps to
 /// the millisecond. Taking the rounded delay as the start of the first packet played the whole
-/// track up to 16 samples late; the decode must line up with ffmpeg's to the sample.
+/// track up to 16 samples late.
+///
+/// The strict check is version-independent: ffmpeg encodes a signal we generate, and our decode
+/// must line up with that signal to the sample (the stream's `CodecDelay` trimmed exactly). ffmpeg's
+/// own decode is compared only after a bounded best-lag search: how its CLI handles Matroska
+/// priming depends on the version (9.x trims `CodecDelay`, 2023-24 builds keep the priming
+/// samples), so its lag is reported, not asserted.
 #[test]
 fn matroska_audio_is_sample_aligned_with_ffmpeg() {
     let ff = filmcraft_testkit::require_ffmpeg!();
-    let cases: [(&str, u32, &[&str]); 5] = [
+    let cases: [(&str, u32, &[&str]); 6] = [
-        ("noise_aac_48k.mkv", 48_000, &["-c:a", "aac", "-b:a", "256k"]),
+        ("ref_aac_48k.mkv", 48_000, &["-c:a", "aac", "-b:a", "256k"]),
-        ("noise_aac_44k.mkv", 44_100, &["-c:a", "aac", "-b:a", "256k"]),
+        ("ref_aac_44k.mkv", 44_100, &["-c:a", "aac", "-b:a", "256k"]),
-        ("noise_ac3_44k.mkv", 44_100, &["-c:a", "ac3", "-b:a", "448k"]),
+        ("ref_ac3_44k.mkv", 44_100, &["-c:a", "ac3", "-b:a", "448k"]),
-        ("noise_ac3_48k.mkv", 48_000, &["-c:a", "ac3", "-b:a", "448k"]),
+        ("ref_ac3_48k.mkv", 48_000, &["-c:a", "ac3", "-b:a", "448k"]),
-        ("noise_mp3_44k.mkv", 44_100, &["-c:a", "libmp3lame", "-b:a", "320k"]),
+        ("ref_eac3_48k.mkv", 48_000, &["-c:a", "eac3", "-b:a", "384k"]),
+        ("ref_mp3_44k.mkv", 44_100, &["-c:a", "libmp3lame", "-b:a", "320k"]),
     ];
     for (name, rate, codec) in cases {
-        let input = format!("anoisesrc=color=pink:seed=7:amplitude=0.5:sample_rate={rate}:duration=2");
+        let reference = noise(rate as usize * 2);
-        let args: Vec<&str> = ["-f", "lavfi", "-i", input.as_str(), "-ac", "1"].iter().copied().chain(codec.iter().copied()).collect();
+        let raw = format!("ref_noise_{rate}.f32");
+        let raw_path = filmcraft_testkit::fixtures::generate(&dir().join(&raw), |tmp| {
+            std::fs::write(tmp, reference.iter().flat_map(|v| v.to_le_bytes()).collect::<Vec<u8>>()).is_ok()
+        })
+        .unwrap();
+        let (raw_arg, rate_arg) = (raw_path.to_str().unwrap().to_string(), rate.to_string());
+        let args: Vec<&str> = ["-f", "f32le", "-ar", &rate_arg, "-ac", "1", "-i", &raw_arg].iter().copied().chain(codec.iter().copied()).collect();
         let Some(path) = fixture(&ff, name, &args) else {
             eprintln!("SKIPPED ({name}): this ffmpeg can't encode it");
             continue;
         };
-        let src = filmcraft_codecs::open_bytes(name, bytes(&path)).unwrap();
+        let file = bytes(&path);
+        let mkv = filmcraft_matroska::open(file.clone()).unwrap();
+        let delay_ns = mkv.tracks.first().map_or(0, |t| t.codec_delay_ns) as i64;
+        let delay = (delay_ns * rate as i64 + 500_000_000) / 1_000_000_000;
+        assert!(delay > 0, "{name}: no CodecDelay");
+        let src = filmcraft_codecs::open_bytes(name, file).unwrap();
         let frames = rate as usize * 3 / 2;
         let mut ours = Vec::with_capacity(frames);
         while ours.len() < frames {
             let n = 1000.min(frames - ours.len());
             ours.extend_from_slice(&src.audio(ours.len() as i64, n, rate).unwrap().channels[0]);
         }
+        let (from, len, bound) = (rate as usize / 4, rate as usize / 4, delay + 32);
+        // strict: the encoder's input, to the sample
+        let (lag, err) = best_lag(&ours, &reference, from, len, -bound..=bound);
+        let level = reference[from..from + len].iter().map(|v| v.abs() as f64).sum::<f64>() / len as f64;
+        assert_eq!(lag, 0, "{name}: our decode is {lag} samples off the encoded signal (CodecDelay {delay} samples)");
+        assert!(err < 0.25 * level, "{name}: decode error {err:.2e} against a signal level of {level:.2e}");
+        // ffmpeg's decode, wherever its version puts the priming
         let theirs = ffmpeg_audio_f32(&ff, &path, &[]);
         assert!(theirs.len() >= frames, "{name}: ffmpeg decoded {} samples", theirs.len());
-        let lag = best_lag(&ours, &theirs, rate as usize / 4, rate as usize, 40);
+        let (ff_lag, ff_err) = best_lag(&ours, &theirs, from, len, -bound..=bound);
-        assert_eq!(lag, 0, "{name}: our decode is {lag} samples off ffmpeg's");
+        let priming = match ff_lag {
+            0 => "trims CodecDelay".to_string(),
+            l if l == delay => "keeps the priming samples".to_string(),
+            l => format!("is {l} samples off"),
+        };
+        println!("{name}: CodecDelay {delay} samples; vs the encoded signal lag 0, error {err:.2e} (level {level:.2e}); ffmpeg {priming}, error {ff_err:.2e}");
+        assert!(ff_err < 1e-3, "{name}: differs from ffmpeg's decode by {ff_err:.2e} at its best lag {ff_lag}");
         // read from the middle, as a seek does
         let at = rate as i64 / 2;
         let mid = src.audio(at, 2000, rate).unwrap().channels[0].clone();

docs/file-format-parity.md

@@ -39,7 +39,7 @@ only). "Tested" names the evidence.
 | QuickTime MOV | read + write | yes | yes (H.264, HEVC, ProRes, DNxHR, APV, MJPEG, PCM) | ffprobe | 90% |
 | MXF OP1a / OP-Atom | read + write (XDCAM, AVC-Intra, P2, AS-10 / AS-11, JPEG 2000, DNx) | yes (AVC, DNx, ProRes, MPEG-2, PCM / AES3, timecode) | OP1a (DNxHR, ProRes, H.264), OP-Atom | index-table seeking tests | 70% |
 | MPEG TS / PS (AVCHD `.mts`, `.m2ts`, `.ts`, `.mpg`, `.vob`, `.mod`) | read; write MPEG-2 / Blu-ray / DVD | yes | **no** | fixtures | 65% |
-| Matroska / WebM | read (limited) | yes (H.264, HEVC, VP9, AV1, ProRes, MJPEG + AAC, Opus, FLAC, MP3, Vorbis, PCM); WebM black report (#432), MKV multi-audio (#601) | no | fixtures; audio sample-aligned with ffmpeg (#790) | 75% |
+| Matroska / WebM | read (limited) | yes (H.264, HEVC, VP9, AV1, ProRes, MJPEG + AAC, Opus, FLAC, MP3, Vorbis, PCM); WebM black report (#432), MKV multi-audio (#601) | no | fixtures; audio sample-aligned with the encoded signal (#790) | 75% |
 | **AVI** | read + write | **no** (extension listed, no demuxer, #598) | no | — | 0% |
 | WMV / ASF | read; write on Windows | no | no | — | 0% |
 | FLV / F4V | read + write | no | no | — | 0% |

docs/gaps.md

@@ -172,8 +172,8 @@ platforms** G4, G9, G17; **performance** G16; **ecosystem** G11; **measurement**
 - **Estimate:** 120–200 h (camera RAW alone 60–100 h, and some SDKs have licences we cannot use:
   clean-room from public specs only where specs exist). Docs: [codec-parity.md](codec-parity.md),
   [file-format-parity.md](file-format-parity.md).
-- **Have:** Matroska AAC / MP3 / AC-3 sample-aligned with ffmpeg's decode (the exact
+- **Have:** Matroska AAC / MP3 / AC-3 / E-AC-3 sample-aligned with the signal the encoder was
-  `CodecDelay` is trimmed, not the 1 ms-rounded one; #790).
+  given (the exact `CodecDelay` is trimmed, not the 1 ms-rounded one; #790).
 
 ## G11. No plugin hosting