filmcraft/crates/vp9/README.md

filmcraft-vp9

Clean-room, pure-Rust (no unsafe) VP9 decoder, implemented from the public VP9 Bitstream & Decoding Process Specification v0.6 (WebM project). It is an L0 crate: it depends only on filmcraft-bitstream, thiserror, and optionally rayon (default feature threads), and builds for wasm32-unknown-unknown with and without threads.

The constant tables (probability defaults, scans, coefficient bands, quantizer lookups, Pareto table, interpolation filters, mv_ref_blocks...) were extracted from the specification text by tools/extract_tables.py into src/spec_tables.rs. The 1D inverse DCT / ADST butterflies are generated by tools/gen_transforms.py, which symbolically executes the step lists of spec section 8.7.1 into src/transform_gen.rs. No code was taken or ported from libvpx, FFmpeg or any other decoder; ffmpeg (with libvpx-vp9) is used only as an external fixture generator and test oracle.

Status

Area	Status
Profiles 0 / 2 (8, 10, 12-bit 4:2:0)	done, bit-exact
Profiles 1 / 3 (4:2:2, 4:4:0, 4:4:4; 8 / 10 / 12-bit)	done, bit-exact
Uncompressed + compressed headers, superframe index (Annex B)	done
Boolean decoder, forward updates (diff_update_prob, MV probs), backward adaptation (8.4)	done
Segmentation (tree / temporal prediction, all four features), loop filter (levels, sharpness, ref / mode deltas, segment levels)	done
All 10 intra modes incl. edge rules, sub-8x8 blocks	done
Inter: 8-tap regular / smooth / sharp + bilinear, compound prediction, MV prediction (incl. previous-frame MVs, sub-8x8 append), reference scaling (8.5.2.3)	done
DCT / ADST 4..32, WHT (lossless)	done
Tiles (columns and rows); tile columns decode in parallel	done
Hidden frames, show_existing_frame, intra-only frames, resolution changes, error resilient / frame parallel modes	done
Loop filter in parallel (superblock wavefront)	done
Frame-level threading (overlapping consecutive frames)	not yet

API

use filmcraft_vp9::{Decoder, Plane};

let mut dec = Decoder::new(); // or Decoder::with_threads(n)
for (pts, chunk) in chunks {  // one IVF / WebM block / MP4 sample each (frame or superframe)
    for pic in dec.decode(chunk, pts)? {
        // pic.width / pic.height (frame size), pic.render_width / render_height (render_size),
        // pic.bit_depth, pic.subsampling_x / subsampling_y, pic.chroma_width / chroma_height,
        // pic.y / pic.u / pic.v: Plane::U8(Vec<u8>) for 8-bit, Plane::U16(Vec<u16>) above,
        // pic.y_stride / pic.uv_stride, pic.pts, pic.key, pic.intra_only,
        // pic.color { color_space (0..7 as coded), full_range }
    }
}
let _ = dec.flush(); // VP9 has no reordering: always empty, kept for API symmetry
dec.reset(); // forget references / contexts (seek), keeps the thread pool
let stats = dec.stats(); // what the stream exercised (frame kinds, tiles, compound, scaling...)

// Random access points: does a container sample start with a key frame (and its parameters)?
let key: bool = filmcraft_vp9::is_keyframe(chunk);
let info: Option<filmcraft_vp9::KeyframeInfo> = filmcraft_vp9::keyframe_info(chunk);

filmcraft-codecs wires the decoder into Matroska / WebM (V_VP9, profile / bit depth from CodecPrivate, colour from Colour) and MP4 (vp09 + vpcC) import; its GOP cache asks is_keyframe before starting a decode at a container sync sample, so wrongly flagged samples (e.g. an MP4 without stss) still seek correctly.

A superframe produces every shown frame it contains (normally one); hidden frames produce none.

Architecture

The calling thread parses the uncompressed and compressed headers and manages the eight reference slots, the four probability contexts, segmentation map and previous-frame motion vectors. Each tile column is decoded by one TileDecoder (rayon pool) into a private strip of samples and mode info (tile rows of a column are decoded in order — they share above contexts); strips are then assembled, the loop filter runs as a superblock wavefront (superblock (r, c) waits for (r-1, c+1); each superblock filters a private copy of its area plus 8 samples above / left and writes it back), counts are merged for backward adaptation and the frame is stored. Samples are u16 internally for every bit depth.

Modules: header (6.2 / 6.3, superframes), boolcoder (9.2), probs (frame contexts, counts, 8.4), tile (partition / mode info / MV prediction), tile::recon (residual, tokens, intra / inter prediction calls, reconstruction), intra, inter, transform, loopfilter, decoder.

Tests

cargo test --release -p filmcraft-vp9. Fixtures are generated on first use into target/fixtures/vp9/ with ffmpeg + libvpx-vp9 (tests print a message and skip when ffmpeg or libvpx is unavailable). Every frame is compared sample-exactly with ffmpeg -c:v vp9 -f rawvideo output, single-threaded and with the default thread pool; the first mismatch reports frame, plane, position and superblock.

• Unit tests: boolean decoder against a literal model of 9.2, tables / trees, inverse DCT / ADST against floating-point bases, DC-only shortcut vs. full transform, scaled MC path vs. unscaled, superframe index, inv_remap_prob, probability merging.
• Conformance (tests/conformance.rs): the fixture matrix below plus a stream concatenating two sizes. compare_detects_mismatch guards the harness. coverage_report (ignored) prints the decoder statistics for each fixture.
• Synthetic (tests/synthetic.rs): headers of libvpx output are rewritten to produce what ffmpeg's libvpx wrapper cannot: inter frames of a stream encoded at another size spliced after a 352x288 key frame with explicit frame sizes, i.e. scaled reference prediction (downscale 256x208, upscale 480x400, odd 301x199; a same-size splice is the control); a key frame rewritten as a hidden intra-only frame (reset_frame_context 3 / 2 / 0, various refresh masks) shown with show_existing_frame; show_existing_frame of every slot, also inside a superframe.
• Robustness (tests/robustness.rs): bit flips, byte replacement and truncation of real streams, truncated frames, random data and oversized frames never panic (debug and release) or hang, single- and multi-threaded.

Fixture matrix (all bit-exact, 1 thread and thread pool)

Backward adaptation is on (-frame-parallel 0) unless noted.

fixture	content	configuration
intra_only_keyframes	352x288	-g 1 (key frames only, frame parallel)
default_good	352x288, 20 frames	800 kb/s
profile2_10bit / profile2_12bit	352x288 / 176x144	profile 2, 10 / 12-bit
profile1_444 / _422 / _440	176x144	profile 1, 4:4:4 / 4:2:2 / 4:4:0
profile3_444_10bit / profile3_422_12bit	176x144	profile 3
lossless / lossless_10bit	176x144	-lossless 1 (WHT)
tiles_cols4	1280x720	4 tile columns
tiles_rows_cols / tiles_rows4	1280x720 / 640x480	tile columns + rows / 4 tile rows
altref_hidden / altref_10bit / altref_tiles	352x288 / 720p	2-pass, -auto-alt-ref 1: hidden frames, superframes, compound prediction
frame_parallel	352x288	-frame-parallel 1 (no backward adaptation)
error_resilient	352x288	-error-resilient 1
odd_size / tiny_odd	347x251 / 33x17	odd sizes, partial superblocks
row_mt	640x360	-row-mt 1, 2 tile columns
speed0 .. speed8	176x144 / 352x288	every speed (0 two-pass, 5..8 realtime)
cq_low / cq_q4 / cq_high / cq_10bit_low / cq_12bit_high		-crf 0 / 4 / 63 / 2 / 60
aq_segmentation / aq_variance / aq_complexity	352x288	-aq-mode 3 / 1 / 2 (segmentation, temporal segment-id prediction)
sharpness	352x288	-sharpness 5
fade_intra_heavy	352x288	fade-in, key frame every 8
hd_1080p	1920x1080	realtime speed 5
resize_concatenated	352x288 then 347x251	size change at a key frame

Performance

cargo test --release -p filmcraft-vp9 --test perf -- --ignored --nocapture (bit-exactness is verified first), or VP9_THREADS=n VP9_LOOPS=k cargo run --release -p filmcraft-vp9 --example vp9dec -- in.ivf (decodes the file k times and prints the statistics).

Apple M4 Pro (14 cores), 1080p 8-bit bench_1080p fixture (libvpx speed 4, 4 MB/s, 4 tile columns), 120 frames per run, before / after binaries alternated. The machine was shared with other builds (load average 80-110 during the multi-thread runs, up to 200 earlier), so wall-clock numbers are the best of 3 runs and still well below what an idle machine gives; single-thread CPU time is the stable figure.

	before (5e9d7d9)	now
1 thread, CPU time	52-55 fps	98-102 fps
14 threads, wall clock (best of 3, load ~90)	82-89 fps	143-145 fps

What made the difference:

• Loop filter: each group of 8 samples along an edge is filtered as 8 lanes at once (16-bit lanes for 8 / 10-bit, 32-bit for 12-bit); the mask / flat / flat2 decisions are lane-wise and every filter result is selected per lane, so the arithmetic vectorizes. Per-group decisions use masks instead of divisions and skip groups with no edge before looking up levels. The parallel loop filter (superblock wavefront) locks per superblock-row band only while copying, instead of holding one frame-wide lock.
• Motion compensation: the 8-tap filters run tap-outer over fixed-width rows (4..64) and no longer zero a 10 KB edge window per block; the 2D inverse transform buffers are sized per transform.
• Memory: strip / frame planes and mode-info grids are recycled through a pool (no per-frame multi-megabyte allocations and page faults); assembling tile strips into the frame and the cropping / 8-bit narrowing of output pictures run in parallel chunks of rows.

The loop filter is still the largest single cost (~30% single-threaded), followed by inter prediction and coefficient decoding.

Known gaps

• No frame-level threading: parallelism is tile columns + loop filter wavefront + row-parallel copies, so streams with a single tile column (narrower than 512 pixels or encoded without tile columns) decode mostly on one thread, and 1080p (at most 4 tile columns) uses 4 threads for coefficient decoding and prediction.
• Scaled references combined with compound prediction are not covered by a fixture (both are verified separately; libvpx through ffmpeg cannot change size mid-stream and the synthetic splice cannot keep the compound streams' probability contexts intact).
• Frames larger than 16384x16384 are rejected (Error::Invalid) to bound memory.
• Damaged streams are decoded as far as the data allows (the boolean decoder reads zeros past the end, as specified); there is no concealment beyond that, and inter frames whose reference slots are empty return Error::MissingReference.