filmcraft/docs/testing.md

Testing

cargo test --workspace                          # everything; oracle tests skip without ffmpeg
cargo test -p filmcraft-h264                    # one crate
cargo test --release -p filmcraft-hevc          # codec tests are much faster in release
cargo xtask ci                                  # all gates (fmt, clippy, tests, layers, assets, wasm)
cargo xtask fixtures                            # pre-generate the ffmpeg fixture matrix
FILMCRAFT_REQUIRE_ORACLES=1 cargo test --workspace   # CI: missing ffmpeg fails instead of skipping

1. Kinds of tests

Kind	Where	Examples
Unit	src/ #[cfg(test)] modules, src/tests.rs	CABAC engines, VLC tables (prefix-freeness, Kraft sums), colour matrices, frame cache
Property (proptest)	time, bitstream, edit, isobmff/tests/roundtrip.rs, interchange/tests/*	tick↔frame round trips for every rate, DF timecode, edit invariants (no overlaps, durations conserved), mux→demux round trips, interchange export→import
Synthetic streams	h264/src/synth_tests.rs, hevc/src/synth_tests.rs	hand-built bitstreams for features the reference encoders don't emit; expected output known exactly
Engine / command	crates/engine/src/tests.rs	run commands on the demo project, assert the sequence, undo/redo, disabled cases
Render	crates/render/src/tests.rs	compositing, opacity, Motion, cross dissolve midpoint, ½-res vs full, GPU plan vs reference, audio mix, audio-effect continuity
Oracle	crates/*/tests/*oracle*.rs, conformance.rs	compare with ffmpeg/ffprobe (§2)
Robustness / fuzz	*/tests/robustness.rs, */tests/fuzz.rs	seeded mutation and truncation of real and synthetic files; nothing may panic
Performance	*/tests/perf.rs (#[ignore])	§5

Commit *.proptest-regressions files so failing cases are re-run.

2. ffmpeg oracle tests

ffmpeg and ffprobe are external processes used to generate fixtures and to check results. They are never linked or shipped (AGENTS.md §2).

• The tools are found by filmcraft-testkit (a dev-dependency-only crate, crates/testkit), in this order: FILMCRAFT_FFMPEG / FILMCRAFT_FFPROBE (explicit paths; a wrong path is an error), then every directory on PATH (ffmpeg and ffmpeg.exe), then well-known install directories (/opt/homebrew/bin, /usr/local/bin, /usr/bin, C:\ffmpeg\bin, …).
• If a tool is missing the test prints SKIPPED (<test>): ffmpeg not found … and passes. Set FILMCRAFT_REQUIRE_ORACLES=1 (CI) to make every such skip a failure. In new tests use let ff = filmcraft_testkit::require_ffmpeg!(); (also require_ffprobe!, require_oracles!).
• Fixtures are generated on first use into <workspace>/target/fixtures/<crate>/ (filmcraft_testkit::fixtures_dir; independent of CARGO_TARGET_DIR, so agents with private target dirs share them; FILMCRAFT_FIXTURES_DIR overrides the root) and reused afterwards. Generators write to a per-process/thread temporary name (testkit::temp_path) and rename it into place, so concurrent tests never see half-written files. Delete the directory to regenerate. Never commit media.
• Every crate's tests/common/mod.rs delegates discovery and fixture paths to testkit, except crates/codecs/src/tests.rs, which still has its own lookup (the crate was being edited concurrently; migrate it when convenient).
• cargo xtask fixtures [crate…] pre-generates the whole fixture matrix up front (useful before a parallel test run or on a fresh machine). It runs each crate's ignored generate_fixtures test — the same generators the oracle tests call — for h264, hevc, isobmff, matroska and prores, and prints one made / cached / skipped line per fixture plus a summary. The aac, opus and h264enc oracles generate small per-test signals on demand and are not part of the matrix.
• Fixture sources are synthetic: testsrc2, mandelbrot, SMPTE bars, noise and fades, sine tones. H.264 and HEVC fixtures need ffmpeg built with libx264 and libx265. VideoToolbox fixtures are generated only on macOS.

Pass criteria per codec

Crate	Oracle check	Criterion
h264	decode every fixture single-threaded and frame-threaded; compare with ffmpeg -f rawvideo -pix_fmt yuv420p	bit-exact, every frame
hevc	same, yuv420p / yuv420p10le	bit-exact (8- and 10-bit)
h264enc	ffmpeg -ec 0 decodes our stream	no errors, exact frame count, bit-identical to the encoder's reconstruction; B-frame order checked with ffprobe; rate-control targets
prores decode	ffmpeg prores_ks / prores_aw fixtures, compared at native depth	within ±1 LSB (different integer IDCT); alpha bit-exact
prores encode	ffmpeg decodes with -xerror	no errors; agrees with our decoder within ±1
aac decode	ffmpeg's decode of the same stream	max abs error ~1e-7 (float)
aac encode	ffmpeg decodes our stream	no errors, no clipping, CBR within ±5% of target; SNR reported
isobmff	ffprobe -show_packets on ffmpeg-made MP4/MOV	packet offsets, sizes, pts/dts, durations, key flags and stream parameters equal; remuxed files decode with ffmpeg -v error silent
matroska	ffprobe -show_packets	every packet (stream, size, key flag, pts, duration) equal; seeks land on the latest keyframe ≤ target
audio-dsp loudness	tests/loudness_oracle.rs: signals generated in Rust (997 Hz sine, pink noise at 48/96 kHz, speech-like bursts at 48/44.1 kHz, stereo with silence and sub-gate passages, an fs/4 inter-sample-peak tone) written as float WAV and measured with ffmpeg -af ebur128=peak=true:metadata=1	momentary and short-term every 100 ms ±0.1 LU, integrated ±0.1 LU, LRA ±0.5 LU, true peak ±0.2 dB (against the analytic value when one exists). Measured: ΔM/ΔS ≤ 0.0005 LU, ΔI ≤ 0.007 LU, ΔLRA ≤ 0.04 LU, ΔTP ≤ 0.045 dB against ffmpeg. On the fs/4 tone ffmpeg's own true peak is +0.6 dB high (−0.32 vs the analytic −0.92 dBTP); ours is −0.05 dB
export	our own demuxer/decoder reads the file back; ffprobe counts frames when present	expected size, duration, colour, audio level; exact frame count

Each codec README has the full fixture matrix and the measured results.

3. Rendering: CPU reference and GPU parity

• The CPU compositor in filmcraft-render is the reference. Its tests check computed values (opacity blends, dissolve midpoints, Motion placement, ½-resolution against downsampled full resolution) and that render::plan matches render_sequence.
• crates/gpu/src/tests.rs composites the same frame plan (a YUV layer plus a transformed, semi-transparent RGBA layer) on the GPU and on the CPU. It requires a 99th-percentile channel difference of ≤ 6 and a mean of < 1.5 (8-bit levels). It skips when no GPU adapter is available.
• Effects implemented only on the CPU need no GPU test: the plan pre-renders those layers on the CPU.
• Golden images (crates/golden, a test-only crate): tests/golden.rs builds small procedural projects (demo-generator footage, bars, no media files) at 320×180 and renders one frame of each through the CPU compositor: Motion transform + opacity, four blend modes (Multiply, Screen, Overlay, Difference), Gaussian Blur, Lumetri basic correction, Crop, Cross Dissolve at 50 %, Dip to Black at 25 %, Wipe at 50 %, and Timecode / Clip Name burn-in text. Each frame is compared with crates/golden/goldens/<scene>.png by filmcraft_testkit::golden: PSNR ≥ 45 dB, max abs ≤ 12, 99th-percentile per-pixel max channel difference ≤ 2 (8-bit sRGB levels; Tolerance::RENDER). On failure the actual frame and a ×8 difference image go to <workspace>/target/golden-failures/. scenes_are_distinct guards against blank or duplicate scenes.
• Blessing: FILMCRAFT_BLESS=1 cargo test -p filmcraft-golden rewrites the references (and writes a .attribution sidecar for any new one; it prints the ATTRIBUTION.md row to add). References are original work rendered by FilmCraft, must stay under 50 KB (enforced on bless), and every one needs its sidecar and index row (cargo xtask assets). Look at a re-blessed PNG before committing it.
• The same scenes run through plan_frame + GpuCompositor when a GPU adapter exists (gpu_matches_cpu_on_golden_scenes, same p99 ≤ 6 / mean < 1.5 criterion; skipped otherwise). This found a real bug: the GPU upload cache was keyed by pixel-buffer address without keeping the buffer alive, so a new frame allocated at a freed frame's address was drawn with the stale texture (fixed; regression test upload_cache_keeps_buffers_alive).

4. UI: control channel and screenshots

There is no automated UI test suite yet. UI work is checked by driving the real app:

cargo run --release -p filmcraft -- --control 9876

Then script it over the control channel or MCP: run commands, click by automation id, ui.inspect / ui.elements to assert state, and ui.screenshot {"path": "…"} to capture the window or a single panel ({"panel": "Timeline"}). Look at every screenshot. Examples are in agents.md. The headless path (filmcraft-cli run script.jsonl, MCP --demo) covers engine behaviour without a window.

5. Performance

Benchmark	Command
H.264 decode, 1080p, threads sweep	cargo test --release -p filmcraft-h264 --test perf -- --ignored --nocapture
HEVC decode, 1080p / 2160p	cargo test --release -p filmcraft-hevc --test perf -- --ignored --nocapture
ProRes decode/encode	cargo test --release -p filmcraft-prores --test perf -- --ignored --nocapture
Encoder speed / PSNR / bitrate	cargo run --release -p filmcraft-h264enc --example h264enc_synth -- …
Decode a real file through the media stack	cargo run --release -p filmcraft-cli -- bench-decode file.mp4 --frames 120
Coding-tool coverage of the fixtures	cargo test --release -p filmcraft-h264 --test conformance coverage_report -- --ignored --nocapture (same for hevc)

The perf tests check bit-exactness before they time anything. Results go in the crate README's performance table, with machine and thread count. Headline numbers go in ROADMAP.md. Measure on an idle machine: parallel agent builds distort timings.