filmcraft/docs/testing.md

Testing

cargo test --workspace                          # everything; oracle tests skip without ffmpeg
cargo test -p filmcraft-h264                    # one crate
cargo test --release -p filmcraft-hevc          # codec tests are much faster in release
cargo xtask ci                                  # all gates (fmt, clippy, tests, layers, assets, wasm)

1. Kinds of tests

Kind	Where	Examples
Unit	src/ #[cfg(test)] modules, src/tests.rs	CABAC engines, VLC tables (prefix-freeness, Kraft sums), colour matrices, frame cache
Property (proptest)	time, bitstream, edit, isobmff/tests/roundtrip.rs, interchange/tests/*	tick↔frame round trips for every rate, DF timecode, edit invariants (no overlaps, durations conserved), mux→demux round trips, interchange export→import
Synthetic streams	h264/src/synth_tests.rs, hevc/src/synth_tests.rs	hand-built bitstreams for features the reference encoders don't emit; expected output known exactly
Engine / command	crates/engine/src/tests.rs	run commands on the demo project, assert the sequence, undo/redo, disabled cases
Render	crates/render/src/tests.rs	compositing, opacity, Motion, cross dissolve midpoint, ½-res vs full, GPU plan vs reference, audio mix, audio-effect continuity
Oracle	crates/*/tests/*oracle*.rs, conformance.rs	compare with ffmpeg/ffprobe (§2)
Robustness / fuzz	*/tests/robustness.rs, */tests/fuzz.rs	seeded mutation and truncation of real and synthetic files; nothing may panic
Performance	*/tests/perf.rs (#[ignore])	§5

Commit *.proptest-regressions files so failing cases are re-run.

2. ffmpeg oracle tests

ffmpeg and ffprobe are external processes used to generate fixtures and to check results. They are never linked or shipped (AGENTS.md §2).

• The tools are found by filmcraft-testkit (a dev-dependency-only crate, crates/testkit), in this order: FILMCRAFT_FFMPEG / FILMCRAFT_FFPROBE (explicit paths; a wrong path is an error), then every directory on PATH (ffmpeg and ffmpeg.exe), then well-known install directories (/opt/homebrew/bin, /usr/local/bin, /usr/bin, C:\ffmpeg\bin, …).
• If a tool is missing the test prints SKIPPED (<test>): ffmpeg not found … and passes. Set FILMCRAFT_REQUIRE_ORACLES=1 (CI) to make every such skip a failure. In new tests use let ff = filmcraft_testkit::require_ffmpeg!(); (also require_ffprobe!, require_oracles!).
• Fixtures are generated on first use into <workspace>/target/fixtures/<crate>/ (filmcraft_testkit::fixtures_dir; independent of CARGO_TARGET_DIR, so agents with private target dirs share them; FILMCRAFT_FIXTURES_DIR overrides the root) and reused afterwards. Generators write to a per-process/thread temporary name (testkit::temp_path) and rename it into place, so concurrent tests never see half-written files. Delete the directory to regenerate. Never commit media.
• Every crate's tests/common/mod.rs delegates discovery and fixture paths to testkit, except crates/codecs/src/tests.rs, which still has its own lookup (the crate was being edited concurrently; migrate it when convenient).
• Fixture sources are synthetic: testsrc2, mandelbrot, SMPTE bars, noise and fades, sine tones. H.264 and HEVC fixtures need ffmpeg built with libx264 and libx265. VideoToolbox fixtures are generated only on macOS.

Pass criteria per codec

Crate	Oracle check	Criterion
h264	decode every fixture single-threaded and frame-threaded; compare with ffmpeg -f rawvideo -pix_fmt yuv420p	bit-exact, every frame
hevc	same, yuv420p / yuv420p10le	bit-exact (8- and 10-bit)
h264enc	ffmpeg -ec 0 decodes our stream	no errors, exact frame count, bit-identical to the encoder's reconstruction; B-frame order checked with ffprobe; rate-control targets
prores decode	ffmpeg prores_ks / prores_aw fixtures, compared at native depth	within ±1 LSB (different integer IDCT); alpha bit-exact
prores encode	ffmpeg decodes with -xerror	no errors; agrees with our decoder within ±1
aac decode	ffmpeg's decode of the same stream	max abs error ~1e-7 (float)
aac encode	ffmpeg decodes our stream	no errors, no clipping, CBR within ±5% of target; SNR reported
isobmff	ffprobe -show_packets on ffmpeg-made MP4/MOV	packet offsets, sizes, pts/dts, durations, key flags and stream parameters equal; remuxed files decode with ffmpeg -v error silent
matroska	ffprobe -show_packets	every packet (stream, size, key flag, pts, duration) equal; seeks land on the latest keyframe ≤ target
export	our own demuxer/decoder reads the file back; ffprobe counts frames when present	expected size, duration, colour, audio level; exact frame count

Each codec README has the full fixture matrix and the measured results.

3. Rendering: CPU reference and GPU parity

• The CPU compositor in filmcraft-render is the reference. Its tests check computed values (opacity blends, dissolve midpoints, Motion placement, ½-resolution against downsampled full resolution) and that render::plan matches render_sequence.
• crates/gpu/src/tests.rs composites the same frame plan (a YUV layer plus a transformed, semi-transparent RGBA layer) on the GPU and on the CPU. It requires a 99th-percentile channel difference of ≤ 6 and a mean of < 1.5 (8-bit levels). It skips when no GPU adapter is available.
• Effects implemented only on the CPU need no GPU test: the plan pre-renders those layers on the CPU.
• Golden-image files (rendered PNGs compared against committed references) are not used yet. If you add them, generate the references from FilmCraft itself, give each one an attribution sidecar, and document the tolerance (PSNR in dB, or max/percentile error).

4. UI: control channel and screenshots

There is no automated UI test suite yet. UI work is checked by driving the real app:

cargo run --release -p filmcraft -- --control 9876

Then script it over the control channel or MCP: run commands, click by automation id, ui.inspect / ui.elements to assert state, and ui.screenshot {"path": "…"} to capture the window or a single panel ({"panel": "Timeline"}). Look at every screenshot. Examples are in agents.md. The headless path (filmcraft-cli run script.jsonl, MCP --demo) covers engine behaviour without a window.

5. Performance

Benchmark	Command
H.264 decode, 1080p, threads sweep	cargo test --release -p filmcraft-h264 --test perf -- --ignored --nocapture
HEVC decode, 1080p / 2160p	cargo test --release -p filmcraft-hevc --test perf -- --ignored --nocapture
ProRes decode/encode	cargo test --release -p filmcraft-prores --test perf -- --ignored --nocapture
Encoder speed / PSNR / bitrate	cargo run --release -p filmcraft-h264enc --example h264enc_synth -- …
Decode a real file through the media stack	cargo run --release -p filmcraft-cli -- bench-decode file.mp4 --frames 120
Coding-tool coverage of the fixtures	cargo test --release -p filmcraft-h264 --test conformance coverage_report -- --ignored --nocapture (same for hevc)

The perf tests check bit-exactness before they time anything. Results go in the crate README's performance table, with machine and thread count. Headline numbers go in ROADMAP.md. Measure on an idle machine: parallel agent builds distort timings.