Merge branch 'main' into worktree-agent-a8349582751c0ec31
This commit is contained in:
Brandon Thomas
commit
d5e3d7e2ef
11
ROADMAP.md
11
ROADMAP.md
|
|
@ -2,7 +2,7 @@
|
|||
|
||||
Progress toward feature parity with Adobe Premiere Pro, with estimates. Updated as milestones land.
|
||||
|
||||
**Last updated:** 2026-10-02 · **Overall parity:** ~81% (measured scorecard below) · **Code:** 34 crates, 1432 tests
|
||||
**Last updated:** 2026-10-02 · **Overall parity:** ~84% (measured scorecard below) · **Code:** 34 crates, 1488 tests
|
||||
|
||||
## Parity scorecard
|
||||
|
||||
|
|
@ -19,11 +19,11 @@ matched against `filmcraft-cli commands` and the UI command table. Effects: the
|
|||
| Panels and UI fidelity | 12% | all main panels incl. Metadata, Timecode, Events, Progress, Reference Monitor, full Lumetri Scopes, audio effect editors; Project panel (List / Icon / Freeform, Metadata Display, view presets, bin tabs and windows) and Media Browser (tree, Favorites, history, thumbnails, Edit Columns) at parity; Libraries-style panels thin | ~75% |
|
||||
| Audio | 10% | mixer, automation (track + clip), Essential Sound, meters, all 53 effects, 5.1 tracks/buses/panner/export, voice-over record, Remix | ~88% |
|
||||
| Colour | 8% | Lumetri complete, LUTs, colour management, HDR, all Lumetri Scopes (+ `scopes.read` for agents) | ~88% |
|
||||
| Graphics and captions | 8% | text engine, shapes, captions, transcripts; no MOGRT, rolls/crawls, responsive design | ~60% |
|
||||
| Graphics and captions | 8% | text engine, shapes, per-character styles, our own graphics templates (.fcgt, 8 built-ins, export/install/edit), rolls/crawls, responsive pins + time, captions SRT/VTT/SCC/MCC/STL/TTML/DFXP, transcripts | ~85% |
|
||||
| Export | 8% | own H.264/AAC, ProRes, DNxHR, image sequences, GIF, WAV; preset library + user presets, full Export-mode settings (loudness normalization, overlays, captions), export queue, Quick Export; no AAF/OMF, MXF export | ~80% |
|
||||
| Preferences and project management | 5% | Settings dialog with 16 categories (most settings wired), project settings, scratch disks, search bins, templates | ~80% |
|
||||
| Performance | 5% | 1080p real-time, 3×1080p; HEVC 2.5× cheaper, catch-up decoding halves 4K CPU/frame, `cargo xtask bench` + `perf.stats`; 4K/8K and AV1 not yet real-time on a loaded machine | ~55% |
|
||||
| **Weighted total** | | | **~81%** |
|
||||
| Performance | 5% | 1080p real-time, 3×1080p; 4K H.264 real-time at load ≤140 (−23% decoder cycles, opt-in draft decoding at 1/2–1/4); HEVC 2.5× cheaper, catch-up decoding, `cargo xtask bench` + `perf.stats`; 8K and AV1 not yet real-time on a loaded machine | ~62% |
|
||||
| **Weighted total** | | | **~84%** |
|
||||
|
||||
## Estimate to parity
|
||||
|
||||
|
|
@ -77,6 +77,9 @@ Status: ✅ done · 🟡 in progress · ⬜ not started. Estimates are remaining
|
|||
|
||||
## Log
|
||||
|
||||
- **2026-10-03 (later):** graphics & captions (M10.7): `.fcgt` graphics templates (export/install/apply/edit, 8 original built-ins; Adobe .mogrt refused), Essential Graphics Browse/Edit, rolls/crawls, responsive design pins and time, per-character styles, Upgrade Caption to Graphic / to Source Graphic, MCC/EBU STL/TTML/DFXP, Replace Fonts; schema v12. 1479 tests.
|
||||
- **2026-10-03:** 4K H.264 playback (M4.9): vectorised deblocking, branch-free CABAC bins and column-parallel inverse transforms cut H.264 decoding by 23 % (cycles per frame, bit-exact; 4K cold seeks −27 % CPU); opt-in draft decoding (Settings ▸ Playback) skips deblocking of non-reference pictures and uploads decimated planes while playing at 1/2–1/4, never for paused frames or exports; conformance runs 1, 3 and all threads. 4K plays 192/0 at load 100–140 (before: only at load ~90).
|
||||
|
||||
- **2026-10-03:** keyboard parity (M3.12: 143 new commands on Premiere's default keys, table validated against the registry), audio (M7.8: 5.1 tracks/submixes/Mix, 5.1 panner, BS.775 downmix, 5.1 WAV/MOV/AAC export, voice-over record with punch-in, Remix, Clip Mixer Latch/Touch/Write). 1432 tests.
|
||||
|
||||
- **2026-10-02 (late):** export parity (M6.5): built-in preset library + user presets, full Export-mode settings (loudness normalization, image/name/timecode overlays, captions, range), export queue, Quick Export, Premiere-style image-sequence export, `filmcraft-cli export --preset`; media import (M9.14): MXF OP1a/OP-Atom (new `crates/mxf`, H.264/DNx/ProRes/PCM, timecode), Ogg Opus (new `crates/ogg`), image sequences as clips, BWF timecode. 1373 tests.
|
||||
|
|
|
|||
|
|
@ -31,6 +31,8 @@ pub struct GopStats {
|
|||
pub decode_ns: u64,
|
||||
/// Non-reference samples left out while catching up (frames already late).
|
||||
pub skipped: u64,
|
||||
/// Frames decoded in draft mode (reduced-resolution playback, [`VideoDecoder::set_draft`]).
|
||||
pub draft: u64,
|
||||
}
|
||||
|
||||
impl GopStats {
|
||||
|
|
@ -53,6 +55,7 @@ static DECODED: AtomicU64 = AtomicU64::new(0);
|
|||
static EVICTED: AtomicU64 = AtomicU64::new(0);
|
||||
static DECODE_NS: AtomicU64 = AtomicU64::new(0);
|
||||
static SKIPPED: AtomicU64 = AtomicU64::new(0);
|
||||
static DRAFT: AtomicU64 = AtomicU64::new(0);
|
||||
|
||||
/// Run a decoder call, adding its wall time to the process-wide counter.
|
||||
fn timed<R>(f: impl FnOnce() -> R) -> R {
|
||||
|
|
@ -72,6 +75,7 @@ pub fn gop_stats() -> GopStats {
|
|||
evicted: EVICTED.load(Ordering::Relaxed),
|
||||
decode_ns: DECODE_NS.load(Ordering::Relaxed),
|
||||
skipped: SKIPPED.load(Ordering::Relaxed),
|
||||
draft: DRAFT.load(Ordering::Relaxed),
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -86,6 +90,7 @@ impl std::ops::Sub for GopStats {
|
|||
evicted: self.evicted - o.evicted,
|
||||
decode_ns: self.decode_ns - o.decode_ns,
|
||||
skipped: self.skipped - o.skipped,
|
||||
draft: self.draft - o.draft,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -116,9 +121,23 @@ struct State {
|
|||
out_max: i64,
|
||||
/// Decoded frames by presentation pts (bounded).
|
||||
frames: BTreeMap<i64, Arc<VideoFrame>>,
|
||||
/// The cached frames that were decoded in draft mode: served to draft requests only.
|
||||
drafts: std::collections::BTreeSet<i64>,
|
||||
bytes: usize,
|
||||
}
|
||||
|
||||
impl State {
|
||||
/// The cached frame at `pts`, unless it is a draft frame and the request wants the exact one.
|
||||
fn cached(&self, pts: i64, draft_ok: bool) -> Option<&Arc<VideoFrame>> {
|
||||
self.frames.get(&pts).filter(|_| draft_ok || !self.drafts.contains(&pts))
|
||||
}
|
||||
|
||||
/// The nearest usable frame at or before `pts` (robust to decoder pts quirks).
|
||||
fn at_or_before(&self, pts: i64, draft_ok: bool) -> Option<Arc<VideoFrame>> {
|
||||
self.cached(pts, draft_ok).or_else(|| self.frames.range(..=pts).rev().find(|(p, _)| draft_ok || !self.drafts.contains(p)).map(|(_, f)| f)).cloned()
|
||||
}
|
||||
}
|
||||
|
||||
thread_local! {
|
||||
/// Caches whose lock this thread holds while decoding (see [`GopCache::frame`]).
|
||||
static DECODING: std::cell::RefCell<Vec<usize>> = const { std::cell::RefCell::new(Vec::new()) };
|
||||
|
|
@ -177,6 +196,7 @@ impl GopCache {
|
|||
start: 0,
|
||||
out_max: i64::MIN,
|
||||
frames: BTreeMap::new(),
|
||||
drafts: Default::default(),
|
||||
bytes: 0,
|
||||
}),
|
||||
explicit_color,
|
||||
|
|
@ -193,8 +213,14 @@ impl GopCache {
|
|||
}
|
||||
}
|
||||
|
||||
fn store(&self, st: &mut State, pts: i64, mut f: VideoFrame) {
|
||||
fn store(&self, st: &mut State, pts: i64, mut f: VideoFrame, draft: bool) {
|
||||
self.apply_color(&mut f);
|
||||
if draft {
|
||||
st.drafts.insert(pts);
|
||||
DRAFT.fetch_add(1, Ordering::Relaxed);
|
||||
} else {
|
||||
st.drafts.remove(&pts);
|
||||
}
|
||||
let budget = self.budget.max(MIN_FRAMES * f.byte_size());
|
||||
st.bytes += f.byte_size();
|
||||
if let Some(old) = st.frames.insert(pts, Arc::new(f)) {
|
||||
|
|
@ -206,6 +232,7 @@ impl GopCache {
|
|||
let last = *st.frames.keys().next_back().expect("non-empty");
|
||||
let victim = if pts - first > last - pts { first } else { last };
|
||||
if let Some(v) = st.frames.remove(&victim) {
|
||||
st.drafts.remove(&victim);
|
||||
st.bytes -= v.byte_size();
|
||||
EVICTED.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
|
|
@ -216,7 +243,7 @@ impl GopCache {
|
|||
fn store_output(&self, st: &mut State, out: Vec<crate::video::DecodedFrame>) {
|
||||
for d in out {
|
||||
st.out_max = st.out_max.max(d.pts);
|
||||
self.store(st, d.pts, d.frame);
|
||||
self.store(st, d.pts, d.frame, d.draft);
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -234,6 +261,10 @@ impl GopCache {
|
|||
/// late. Non-reference samples of late frames are left out on the way to the wanted frame
|
||||
/// (no other picture depends on them, so the wanted frame decodes exactly as it would
|
||||
/// otherwise; a later request for a skipped frame re-seeks).
|
||||
///
|
||||
/// Inside [`filmcraft_media::cancel::with_draft`]`(true, …)` the decoder runs in draft mode
|
||||
/// ([`VideoDecoder::set_draft`]); its draft frames are cached for draft requests only, and an
|
||||
/// exact request for one re-decodes it.
|
||||
pub fn frame_late(&self, s: &dyn VideoSamples, target: i64, late_before: Option<i64>) -> crate::Result<Arc<VideoFrame>> {
|
||||
let n = s.count();
|
||||
let i = s.sample_at(target.max(0)).or_else(|| (n > 0).then(|| n - 1)).ok_or_else(|| CodecError::Decode("empty track".into()))?;
|
||||
|
|
@ -242,9 +273,10 @@ impl GopCache {
|
|||
if DecodingGuard::active(me) {
|
||||
return self.private_frame(s, i, want_pts, n);
|
||||
}
|
||||
let draft = filmcraft_media::cancel::draft();
|
||||
let mut st = self.state.lock().unwrap_or_else(|e| e.into_inner());
|
||||
let _decoding = DecodingGuard::enter(me);
|
||||
if let Some(f) = st.frames.get(&want_pts) {
|
||||
if let Some(f) = st.cached(want_pts, draft) {
|
||||
HITS.fetch_add(1, Ordering::Relaxed);
|
||||
return Ok(f.clone());
|
||||
}
|
||||
|
|
@ -293,6 +325,7 @@ impl GopCache {
|
|||
}
|
||||
let limit = (i.max(st.next) + 64).min(n);
|
||||
let late = late_before.unwrap_or(i64::MIN).min(want_pts);
|
||||
st.decoder.as_mut().expect("decoder").set_draft(draft);
|
||||
while st.next < limit {
|
||||
if filmcraft_media::cancel::cancelled() {
|
||||
// The decoder state stays consistent (`next`, `out_max`): a later request continues.
|
||||
|
|
@ -309,21 +342,17 @@ impl GopCache {
|
|||
st.next += 1;
|
||||
DECODED.fetch_add(1, Ordering::Relaxed);
|
||||
self.store_output(&mut st, out);
|
||||
if st.frames.contains_key(&want_pts) {
|
||||
if st.cached(want_pts, draft).is_some() {
|
||||
break;
|
||||
}
|
||||
}
|
||||
if !st.frames.contains_key(&want_pts) {
|
||||
if st.cached(want_pts, draft).is_none() {
|
||||
let out = timed(|| st.decoder.as_mut().expect("decoder").flush());
|
||||
self.store_output(&mut st, out);
|
||||
st.next = usize::MAX;
|
||||
}
|
||||
// nearest decoded frame at or before the wanted pts (robust to decoder pts quirks)
|
||||
st.frames
|
||||
.get(&want_pts)
|
||||
.cloned()
|
||||
.or_else(|| st.frames.range(..=want_pts).next_back().map(|(_, f)| f.clone()))
|
||||
.ok_or_else(|| CodecError::Decode("frame not produced".into()))
|
||||
st.at_or_before(want_pts, draft).ok_or_else(|| CodecError::Decode("frame not produced".into()))
|
||||
}
|
||||
|
||||
/// A nested request (see [`Self::frame`]): decode from the sync sample with a fresh decoder,
|
||||
|
|
@ -407,13 +436,9 @@ impl GopCache {
|
|||
st.spare.push(dec);
|
||||
}
|
||||
for d in res? {
|
||||
self.store(&mut st, d.pts, d.frame);
|
||||
self.store(&mut st, d.pts, d.frame, d.draft);
|
||||
}
|
||||
st.frames
|
||||
.get(&want_pts)
|
||||
.cloned()
|
||||
.or_else(|| st.frames.range(..=want_pts).next_back().map(|(_, f)| f.clone()))
|
||||
.ok_or_else(|| CodecError::Decode("frame not produced".into()))
|
||||
st.at_or_before(want_pts, true).ok_or_else(|| CodecError::Decode("frame not produced".into()))
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -438,20 +463,24 @@ mod tests {
|
|||
struct Dec {
|
||||
delay: usize,
|
||||
intra: bool,
|
||||
held: VecDeque<i64>,
|
||||
/// (pts, decoded as a draft picture)
|
||||
held: VecDeque<(i64, bool)>,
|
||||
draft: bool,
|
||||
resets: Arc<AtomicUsize>,
|
||||
decodes: Arc<AtomicUsize>,
|
||||
}
|
||||
|
||||
fn picture(pts: i64) -> DecodedFrame {
|
||||
/// A picture whose pixel encodes its index (and, in blue, whether it is a draft picture).
|
||||
fn picture((pts, draft): (i64, bool)) -> DecodedFrame {
|
||||
let i = (pts / 1000) as u32;
|
||||
DecodedFrame { pts, frame: VideoFrame::rgba8(1, 1, vec![i as u8, (i >> 8) as u8, 0, 255]) }
|
||||
DecodedFrame { pts, frame: VideoFrame::rgba8(1, 1, vec![i as u8, (i >> 8) as u8, draft as u8, 255]), draft }
|
||||
}
|
||||
|
||||
impl VideoDecoder for Dec {
|
||||
fn decode(&mut self, _sample: &[u8], pts: i64) -> crate::Result<Vec<DecodedFrame>> {
|
||||
fn decode(&mut self, sample: &[u8], pts: i64) -> crate::Result<Vec<DecodedFrame>> {
|
||||
self.decodes.fetch_add(1, Ordering::Relaxed);
|
||||
self.held.push_back(pts);
|
||||
// like H.264 draft mode: only non-reference (odd) pictures are approximate
|
||||
self.held.push_back((pts, self.draft && sample[0] % 2 == 1));
|
||||
let mut out = Vec::new();
|
||||
while self.held.len() > self.delay {
|
||||
out.push(picture(self.held.pop_front().expect("held")));
|
||||
|
|
@ -465,6 +494,9 @@ mod tests {
|
|||
self.resets.fetch_add(1, Ordering::Relaxed);
|
||||
self.held.clear();
|
||||
}
|
||||
fn set_draft(&mut self, on: bool) {
|
||||
self.draft = on;
|
||||
}
|
||||
fn name(&self) -> &str {
|
||||
"test"
|
||||
}
|
||||
|
|
@ -495,7 +527,14 @@ mod tests {
|
|||
Ok(vec![i as u8])
|
||||
}
|
||||
fn make_decoder(&self) -> crate::Result<Box<dyn VideoDecoder>> {
|
||||
Ok(Box::new(Dec { delay: self.delay, intra: self.intra, held: VecDeque::new(), resets: self.resets.clone(), decodes: self.decodes.clone() }))
|
||||
Ok(Box::new(Dec {
|
||||
delay: self.delay,
|
||||
intra: self.intra,
|
||||
held: VecDeque::new(),
|
||||
draft: false,
|
||||
resets: self.resets.clone(),
|
||||
decodes: self.decodes.clone(),
|
||||
}))
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -592,6 +631,43 @@ mod tests {
|
|||
assert_eq!(s3.resets.load(Ordering::Relaxed), 1);
|
||||
}
|
||||
|
||||
fn is_draft(f: &VideoFrame) -> bool {
|
||||
match &f.data {
|
||||
filmcraft_frame::PixelData::Rgba8(d) => d[2] == 1,
|
||||
_ => unreachable!(),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn draft_frames_are_served_to_draft_requests_only() {
|
||||
let s = samples(300, 250, 3, false);
|
||||
let c = GopCache::new(None);
|
||||
let before = gop_stats().draft;
|
||||
// reduced-resolution playback: odd (non-reference) pictures come out as drafts
|
||||
for i in 0..12usize {
|
||||
let f = filmcraft_media::cancel::with_draft(true, || c.frame(&s, i as i64 * 1000)).expect("frame");
|
||||
assert_eq!(index_of(&f), i);
|
||||
assert_eq!(is_draft(&f), i % 2 == 1, "frame {i}");
|
||||
}
|
||||
assert!(gop_stats().draft - before >= 6);
|
||||
assert_eq!(s.resets.load(Ordering::Relaxed), 1);
|
||||
// paused / export (no draft hint): an even frame is exact and cached
|
||||
let f = c.frame(&s, 4_000).expect("frame");
|
||||
assert!(!is_draft(&f) && index_of(&f) == 4);
|
||||
assert_eq!(s.resets.load(Ordering::Relaxed), 1, "exact frames are served from the cache");
|
||||
// a draft frame is decoded again, exactly
|
||||
let f = c.frame(&s, 5_000).expect("frame");
|
||||
assert!(!is_draft(&f) && index_of(&f) == 5);
|
||||
assert_eq!(s.resets.load(Ordering::Relaxed), 2, "re-decoded from the sync sample");
|
||||
// and a draft request takes the exact frame now cached
|
||||
let f = filmcraft_media::cancel::with_draft(true, || c.frame(&s, 5_000)).expect("frame");
|
||||
assert!(!is_draft(&f));
|
||||
// without the hint nothing is ever a draft
|
||||
let s2 = samples(300, 250, 3, false);
|
||||
let c2 = GopCache::new(None);
|
||||
assert!((0..12).all(|i| !is_draft(&c2.frame(&s2, i * 1000).expect("frame"))));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cancelled_request_stops_decoding_and_keeps_state() {
|
||||
let s = samples(300, 250, 0, false);
|
||||
|
|
|
|||
|
|
@ -9,6 +9,9 @@ use crate::{CodecError, Result};
|
|||
pub struct DecodedFrame {
|
||||
pub pts: i64,
|
||||
pub frame: VideoFrame,
|
||||
/// Decoded in draft mode ([`VideoDecoder::set_draft`]): approximate, for reduced-resolution
|
||||
/// playback only.
|
||||
pub draft: bool,
|
||||
}
|
||||
|
||||
/// A stateful video decoder. Samples are fed in decode order; pictures come out in
|
||||
|
|
@ -37,6 +40,11 @@ pub trait VideoDecoder: Send {
|
|||
fn is_disposable(&self, _sample: &[u8]) -> bool {
|
||||
false
|
||||
}
|
||||
/// Draft decoding for reduced-resolution playback (off by default; see
|
||||
/// [`filmcraft_media::cancel::with_draft`]): samples fed from now on may be decoded with
|
||||
/// shortcuts that only change pictures nothing references, which come out flagged
|
||||
/// [`DecodedFrame::draft`]. Decoders without such a mode ignore it.
|
||||
fn set_draft(&mut self, _on: bool) {}
|
||||
}
|
||||
|
||||
/// NAL unit headers of a length-prefixed (avcC / hvcC) sample: the first two bytes of each unit.
|
||||
|
|
@ -132,7 +140,7 @@ impl VideoDecoder for MjpegDecoder {
|
|||
let img = image::load_from_memory_with_format(sample, image::ImageFormat::Jpeg).map_err(|e| CodecError::Decode(e.to_string()))?;
|
||||
let rgba = img.to_rgba8();
|
||||
let (w, h) = rgba.dimensions();
|
||||
Ok(vec![DecodedFrame { pts, frame: VideoFrame::rgba8(w, h, rgba.into_raw()) }])
|
||||
Ok(vec![DecodedFrame { pts, frame: VideoFrame::rgba8(w, h, rgba.into_raw()), draft: false }])
|
||||
}
|
||||
fn flush(&mut self) -> Vec<DecodedFrame> {
|
||||
Vec::new()
|
||||
|
|
@ -155,20 +163,22 @@ pub struct H264Decoder {
|
|||
avcc: Vec<u8>,
|
||||
dec: filmcraft_h264::Decoder,
|
||||
length_size: usize,
|
||||
draft: bool,
|
||||
}
|
||||
|
||||
impl H264Decoder {
|
||||
pub fn new(avcc: Vec<u8>) -> Result<Self> {
|
||||
let dec = filmcraft_h264::Decoder::from_avcc(&avcc).map_err(|e| CodecError::Decode(e.to_string()))?;
|
||||
let length_size = avcc.get(4).map_or(4, |b| (b & 3) as usize + 1);
|
||||
Ok(Self { avcc, dec, length_size })
|
||||
Ok(Self { avcc, dec, length_size, draft: false })
|
||||
}
|
||||
/// A decoder for Annex B byte-stream samples (start codes, in-band parameter sets: MXF, TS).
|
||||
pub fn annexb() -> Self {
|
||||
Self { avcc: Vec::new(), dec: filmcraft_h264::Decoder::new(), length_size: 0 }
|
||||
Self { avcc: Vec::new(), dec: filmcraft_h264::Decoder::new(), length_size: 0, draft: false }
|
||||
}
|
||||
fn convert(p: filmcraft_h264::Picture) -> DecodedFrame {
|
||||
use std::sync::Arc;
|
||||
let draft = p.draft;
|
||||
let (w, h) = (p.width as usize, p.height as usize);
|
||||
let (cw, ch) = (p.chroma_width as usize, p.chroma_height as usize);
|
||||
let y = tight_plane(p.y, p.y_stride, w, h);
|
||||
|
|
@ -193,7 +203,7 @@ impl H264Decoder {
|
|||
par,
|
||||
pts: filmcraft_time::Tick::ZERO,
|
||||
};
|
||||
DecodedFrame { pts: p.pts, frame }
|
||||
DecodedFrame { pts: p.pts, frame, draft }
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -211,10 +221,15 @@ impl VideoDecoder for H264Decoder {
|
|||
} else if let Ok(d) = filmcraft_h264::Decoder::from_avcc(&self.avcc) {
|
||||
self.dec = d;
|
||||
}
|
||||
self.dec.set_draft(self.draft);
|
||||
}
|
||||
fn name(&self) -> &str {
|
||||
"FilmCraft H.264"
|
||||
}
|
||||
fn set_draft(&mut self, on: bool) {
|
||||
self.draft = on;
|
||||
self.dec.set_draft(on);
|
||||
}
|
||||
fn is_disposable(&self, sample: &[u8]) -> bool {
|
||||
if self.length_size == 0 {
|
||||
return h264_disposable_annexb(sample);
|
||||
|
|
@ -227,6 +242,11 @@ impl VideoDecoder for H264Decoder {
|
|||
}
|
||||
}
|
||||
|
||||
/// Worker threads each H.264 decoder uses (frame threading; 1 on wasm).
|
||||
pub fn h264_threads() -> usize {
|
||||
filmcraft_h264::default_threads()
|
||||
}
|
||||
|
||||
pub fn h264_factory(e: &SampleEntry) -> Option<Result<Box<dyn VideoDecoder>>> {
|
||||
match &e.codec {
|
||||
CodecConfig::Avc(a) => Some(H264Decoder::new(a.to_bytes()).map(|d| Box::new(d) as Box<dyn VideoDecoder>)),
|
||||
|
|
@ -287,7 +307,7 @@ impl HevcDecoder {
|
|||
}
|
||||
let par = if p.sar.0 > 0 && p.sar.1 > 0 { (p.sar.0 as u32, p.sar.1 as u32) } else { (1, 1) };
|
||||
let frame = VideoFrame { width: p.width, height: p.height, data, color, par, pts: filmcraft_time::Tick::ZERO };
|
||||
DecodedFrame { pts: p.pts, frame }
|
||||
DecodedFrame { pts: p.pts, frame, draft: false }
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -388,7 +408,7 @@ impl Vp9Decoder {
|
|||
}
|
||||
let mut frame = VideoFrame::rgba8(p.width, p.height, rgba);
|
||||
frame.color = filmcraft_color::ColorInfo { range: Range::Full, transfer: filmcraft_color::Transfer::Srgb, ..color };
|
||||
return DecodedFrame { pts, frame };
|
||||
return DecodedFrame { pts, frame, draft: false };
|
||||
}
|
||||
// 4:4:0 has no frame format of its own: chroma rows are repeated to 4:4:4.
|
||||
let (chroma, rows_440) = match (p.subsampling_x, p.subsampling_y) {
|
||||
|
|
@ -425,7 +445,7 @@ impl Vp9Decoder {
|
|||
// render_size (the intended display size) is not applied: the container's display
|
||||
// dimensions / pixel aspect describe the presentation.
|
||||
let par = (1, 1);
|
||||
DecodedFrame { pts, frame: VideoFrame { width: p.width, height: p.height, data, color, par, pts: filmcraft_time::Tick::ZERO } }
|
||||
DecodedFrame { pts, frame: VideoFrame { width: p.width, height: p.height, data, color, par, pts: filmcraft_time::Tick::ZERO }, draft: false }
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -509,7 +529,7 @@ impl Av1Decoder {
|
|||
} else {
|
||||
filmcraft_frame::PixelData::Yuv16 { planes: [Arc::new(y), Arc::new(u), Arc::new(v)], chroma, bits: p.bit_depth as u32, alpha: None }
|
||||
};
|
||||
DecodedFrame { pts, frame: VideoFrame { width: p.width, height: p.height, data, color, par: (1, 1), pts: filmcraft_time::Tick::ZERO } }
|
||||
DecodedFrame { pts, frame: VideoFrame { width: p.width, height: p.height, data, color, par: (1, 1), pts: filmcraft_time::Tick::ZERO }, draft: false }
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -591,7 +611,7 @@ impl VideoDecoder for ProResDecoder {
|
|||
par: (1, 1),
|
||||
pts: filmcraft_time::Tick::ZERO,
|
||||
};
|
||||
Ok(vec![DecodedFrame { pts, frame }])
|
||||
Ok(vec![DecodedFrame { pts, frame, draft: false }])
|
||||
}
|
||||
fn flush(&mut self) -> Vec<DecodedFrame> {
|
||||
Vec::new()
|
||||
|
|
@ -663,7 +683,7 @@ pub fn dnx_to_video_frame(f: filmcraft_dnx::Frame) -> VideoFrame {
|
|||
impl VideoDecoder for DnxDecoder {
|
||||
fn decode(&mut self, sample: &[u8], pts: i64) -> Result<Vec<DecodedFrame>> {
|
||||
let f = filmcraft_dnx::decode_frame(sample).map_err(|e| CodecError::Decode(e.to_string()))?;
|
||||
Ok(vec![DecodedFrame { pts, frame: dnx_to_video_frame(f) }])
|
||||
Ok(vec![DecodedFrame { pts, frame: dnx_to_video_frame(f), draft: false }])
|
||||
}
|
||||
fn flush(&mut self) -> Vec<DecodedFrame> {
|
||||
Vec::new()
|
||||
|
|
|
|||
|
|
@ -0,0 +1,68 @@
|
|||
//! H.264 draft decoding (opt-in, reduced-resolution playback) through the media stack: it is off
|
||||
//! by default, its approximate frames only ever reach draft requests, and decoding for an export
|
||||
//! after draft playback (same source, same GOP cache) still matches ffmpeg bit for bit.
|
||||
|
||||
mod common;
|
||||
|
||||
use common::*;
|
||||
use filmcraft_media::FrameRequest;
|
||||
use filmcraft_media::cancel::with_draft;
|
||||
|
||||
#[test]
|
||||
fn draft_playback_never_reaches_exports() {
|
||||
let ff = filmcraft_testkit::require_ffmpeg!();
|
||||
// B-pyramid: references and non-reference B pictures (the ones draft mode leaves unfiltered)
|
||||
let Some(file) = fixture(
|
||||
&ff,
|
||||
"draft_h264.mp4",
|
||||
&[
|
||||
"-f",
|
||||
"lavfi",
|
||||
"-i",
|
||||
"testsrc2=s=320x240:r=24:d=2,noise=alls=10:allf=t",
|
||||
"-c:v",
|
||||
"libx264",
|
||||
"-preset",
|
||||
"fast",
|
||||
"-x264-params",
|
||||
"bframes=3:b-pyramid=normal:keyint=48",
|
||||
"-pix_fmt",
|
||||
"yuv420p",
|
||||
],
|
||||
) else {
|
||||
return;
|
||||
};
|
||||
let raw = ffmpeg_frames(&ff, &file, "yuv420p");
|
||||
let (w, h) = (320usize, 240usize);
|
||||
let fsize = w * h * 3 / 2;
|
||||
let n = raw.len() / fsize;
|
||||
assert!(n >= 40, "{n} frames");
|
||||
let reference = |i: usize| raw_planes(&raw[i * fsize..(i + 1) * fsize], w, h, w / 2, h / 2, 1);
|
||||
let src = filmcraft_codecs::open_bytes("draft_h264.mp4", bytes(&file)).expect("open");
|
||||
let rate = src.info().frame_rate();
|
||||
|
||||
// Default: nothing is a draft, every frame exact.
|
||||
let g0 = filmcraft_codecs::gop_stats();
|
||||
for i in 0..n {
|
||||
let f = src.video_frame(FrameRequest { time: rate.tick_of(i as i64), scale: 0.5 }).expect("frame");
|
||||
assert_eq!(max_diff(&planes(&f), &reference(i)), 0, "frame {i} without draft mode");
|
||||
}
|
||||
assert_eq!(filmcraft_codecs::gop_stats().draft, g0.draft, "draft decoding is off by default");
|
||||
|
||||
// Draft playback at 1/2 resolution on a fresh source: non-reference pictures are drafts.
|
||||
let src = filmcraft_codecs::open_bytes("draft_h264.mp4", bytes(&file)).expect("open");
|
||||
let g1 = filmcraft_codecs::gop_stats();
|
||||
let mut approximate = 0;
|
||||
for i in 0..n {
|
||||
let f = with_draft(true, || src.video_frame(FrameRequest { time: rate.tick_of(i as i64), scale: 0.5 })).expect("frame");
|
||||
approximate += (max_diff(&planes(&f), &reference(i)) > 0) as usize;
|
||||
}
|
||||
let drafts = filmcraft_codecs::gop_stats().draft - g1.draft;
|
||||
assert!(drafts > 0 && approximate > 0 && approximate as u64 <= drafts, "{drafts} draft frames, {approximate} differ");
|
||||
|
||||
// Then an export from the same source (no draft hint): every frame is exact again.
|
||||
for i in 0..n {
|
||||
let f = src.video_frame(FrameRequest::full(rate.tick_of(i as i64))).expect("frame");
|
||||
assert_eq!(max_diff(&planes(&f), &reference(i)), 0, "export frame {i} after draft playback");
|
||||
}
|
||||
}
|
||||
|
|
@ -78,6 +78,10 @@ pub struct PlaybackPrefs {
|
|||
pub pause_encoder_queue: bool,
|
||||
pub enable_transmit: bool,
|
||||
pub disable_video_in_background: bool,
|
||||
/// Draft decoding while playing at 1/2 resolution or lower: H.264 non-reference pictures
|
||||
/// skip deblocking (faster; those frames are approximate). Never used for exports, renders or
|
||||
/// a paused frame. Off by default.
|
||||
pub draft_decode: bool,
|
||||
}
|
||||
|
||||
impl Default for PlaybackPrefs {
|
||||
|
|
@ -89,6 +93,7 @@ impl Default for PlaybackPrefs {
|
|||
pause_encoder_queue: true,
|
||||
enable_transmit: false,
|
||||
disable_video_in_background: true,
|
||||
draft_decode: false,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -17,6 +17,8 @@ pub fn decode_json() -> Value {
|
|||
"seeks": g.seeks,
|
||||
"samplesDecoded": g.decoded,
|
||||
"samplesSkipped": g.skipped,
|
||||
"draftFrames": g.draft,
|
||||
"h264Threads": filmcraft_codecs::video::h264_threads(),
|
||||
"evicted": g.evicted,
|
||||
"decodeMs": g.decode_ns as f64 / 1e6,
|
||||
"decodeMsPerSample": g.decode_ms_per_sample(),
|
||||
|
|
@ -46,7 +48,7 @@ mod tests {
|
|||
s.execute("file.openDemoProject", json!({})).unwrap();
|
||||
let undo = s.history.undo.len();
|
||||
let v = s.execute("perf.stats", json!({})).unwrap();
|
||||
for k in ["requests", "cacheHitRate", "seeks", "samplesDecoded", "samplesSkipped", "decodeMs", "decodeMsPerSample"] {
|
||||
for k in ["requests", "cacheHitRate", "seeks", "samplesDecoded", "samplesSkipped", "draftFrames", "h264Threads", "decodeMs", "decodeMsPerSample"] {
|
||||
assert!(v["decode"][k].is_number(), "decode.{k} in {v}");
|
||||
}
|
||||
assert!(v["media"]["openSources"].is_number());
|
||||
|
|
|
|||
|
|
@ -951,6 +951,7 @@ static CATEGORIES: &[Category] = &[
|
|||
b("playback.pauseEncoderQueue", "Pause Media Encoder queue during playback", false),
|
||||
b("playback.enableTransmit", "Enable Mercury Transmit", false),
|
||||
b("playback.disableVideoInBackground", "Disable video output when in the background", false),
|
||||
b("playback.draftDecode", "Draft decoding at reduced playback resolution (H.264: faster, some frames less filtered)", true),
|
||||
],
|
||||
},
|
||||
Category {
|
||||
|
|
|
|||
|
|
@ -98,6 +98,51 @@ impl VideoFrame {
|
|||
}
|
||||
}
|
||||
|
||||
/// A planar Y'CbCr frame reduced by `n` (2, 4, 8…) in each direction: every sample (luma,
|
||||
/// chroma at its own resolution, alpha) is the rounded mean of an `n`×`n` block, chroma format
|
||||
/// and bit depth unchanged. Reduced-resolution playback hands the GPU this instead of the full
|
||||
/// picture (a quarter / sixteenth of the upload). None for RGBA frames or `n` < 2.
|
||||
pub fn box_decimated(&self, n: usize) -> Option<VideoFrame> {
|
||||
if n < 2 {
|
||||
return None;
|
||||
}
|
||||
let (w, h) = (self.width as usize, self.height as usize);
|
||||
let (ow, oh) = ((w / n).max(1), (h / n).max(1));
|
||||
let data = match &self.data {
|
||||
PixelData::Yuv8 { planes, chroma, alpha } => {
|
||||
let (sx, sy) = chroma.shifts();
|
||||
let (cw, ch) = (w.div_ceil(1 << sx), h.div_ceil(1 << sy));
|
||||
let (ocw, och) = (ow.div_ceil(1 << sx), oh.div_ceil(1 << sy));
|
||||
PixelData::Yuv8 {
|
||||
planes: [
|
||||
Arc::new(box_plane(&planes[0], w, h, ow, oh, n)),
|
||||
Arc::new(box_plane(&planes[1], cw, ch, ocw, och, n)),
|
||||
Arc::new(box_plane(&planes[2], cw, ch, ocw, och, n)),
|
||||
],
|
||||
chroma: *chroma,
|
||||
alpha: alpha.as_ref().map(|a| Arc::new(box_plane(a, w, h, ow, oh, n))),
|
||||
}
|
||||
}
|
||||
PixelData::Yuv16 { planes, chroma, bits, alpha } => {
|
||||
let (sx, sy) = chroma.shifts();
|
||||
let (cw, ch) = (w.div_ceil(1 << sx), h.div_ceil(1 << sy));
|
||||
let (ocw, och) = (ow.div_ceil(1 << sx), oh.div_ceil(1 << sy));
|
||||
PixelData::Yuv16 {
|
||||
planes: [
|
||||
Arc::new(box_plane(&planes[0], w, h, ow, oh, n)),
|
||||
Arc::new(box_plane(&planes[1], cw, ch, ocw, och, n)),
|
||||
Arc::new(box_plane(&planes[2], cw, ch, ocw, och, n)),
|
||||
],
|
||||
chroma: *chroma,
|
||||
bits: *bits,
|
||||
alpha: alpha.as_ref().map(|a| Arc::new(box_plane(a, w, h, ow, oh, n))),
|
||||
}
|
||||
}
|
||||
PixelData::Rgba8(_) | PixelData::RgbaF32(_) => return None,
|
||||
};
|
||||
Some(VideoFrame { width: ow as u32, height: oh as u32, data, color: self.color, par: self.par, pts: self.pts })
|
||||
}
|
||||
|
||||
/// Convert to premultiplied linear RGBA f32 (the compositor's working format).
|
||||
pub fn to_linear_f32(&self) -> Vec<f32> {
|
||||
self.to_linear_f32_decimated(1).2
|
||||
|
|
@ -376,8 +421,59 @@ pub fn default_matrix(width: u32, height: u32) -> Matrix {
|
|||
if width <= 1024 && height <= 576 { Matrix::Bt601 } else { Matrix::Bt709 }
|
||||
}
|
||||
|
||||
/// `n`×`n` box mean of a `w`×`h` plane into `ow`×`oh` (blocks clamped at the right / bottom).
|
||||
fn box_plane<T: Copy + Into<u32> + TryFrom<u32> + Send + Sync + Default>(src: &[T], w: usize, h: usize, ow: usize, oh: usize, n: usize) -> Vec<T> {
|
||||
let mut out = vec![T::default(); ow * oh];
|
||||
if w == 0 || h == 0 || src.len() < w * h {
|
||||
return out;
|
||||
}
|
||||
out.par_chunks_mut(ow).enumerate().for_each(|(oy, row)| {
|
||||
let mut acc = vec![0u32; ow];
|
||||
let mut cnt = vec![0u32; ow];
|
||||
for y in (oy * n..(oy + 1) * n).filter(|&y| y < h) {
|
||||
let line = &src[y * w..y * w + w];
|
||||
for ((a, c), chunk) in acc.iter_mut().zip(cnt.iter_mut()).zip(line.chunks(n)) {
|
||||
*a += chunk.iter().map(|&v| v.into()).sum::<u32>();
|
||||
*c += chunk.len() as u32;
|
||||
}
|
||||
}
|
||||
for ((o, a), c) in row.iter_mut().zip(&acc).zip(&cnt) {
|
||||
let c = (*c).max(1);
|
||||
*o = T::try_from((a + c / 2) / c).unwrap_or_default();
|
||||
}
|
||||
});
|
||||
out
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
|
||||
#[test]
|
||||
fn box_decimation_averages_blocks_per_plane() {
|
||||
// 6x4 4:2:0: luma ramp, chroma 3x2
|
||||
let y: Vec<u8> = (0..24).map(|i| (i * 10) as u8).collect();
|
||||
let u = vec![10, 20, 30, 40, 50, 60];
|
||||
let v = vec![200; 6];
|
||||
let f = VideoFrame {
|
||||
width: 6,
|
||||
height: 4,
|
||||
data: PixelData::Yuv8 { planes: [Arc::new(y), Arc::new(u), Arc::new(v)], chroma: Chroma::C420, alpha: None },
|
||||
color: ColorInfo::REC709,
|
||||
par: (1, 1),
|
||||
pts: Tick::ZERO,
|
||||
};
|
||||
let d = f.box_decimated(2).expect("yuv");
|
||||
assert_eq!((d.width, d.height), (3, 2));
|
||||
let PixelData::Yuv8 { planes, .. } = &d.data else { panic!() };
|
||||
// luma block (0,0): 0, 10, 60, 70 -> 35
|
||||
assert_eq!(planes[0][0], 35);
|
||||
assert_eq!(planes[0].len(), 6);
|
||||
// chroma 3x2 -> 2x1: blocks {10,20,40,50} = 30 and the clamped edge block {30,60} = 45
|
||||
assert_eq!(&planes[1][..], &[30, 45]);
|
||||
assert_eq!(&planes[2][..], &[200, 200]);
|
||||
assert!(f.box_decimated(1).is_none());
|
||||
assert!(VideoFrame::rgba8(2, 2, vec![0; 16]).box_decimated(2).is_none());
|
||||
}
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
|
|
|
|||
|
|
@ -34,6 +34,8 @@ oracles and fixture generators.
|
|||
- Output: cropped 8-bit planar 4:2:0 pictures in output order with the caller's `pts`, POC, key
|
||||
flag, VUI colour description (range, primaries, transfer, matrix) and sample aspect ratio.
|
||||
- Frame-level multithreading (see below).
|
||||
- Draft mode for reduced-resolution playback (off by default): non-reference pictures skip the
|
||||
deblocking filter and are flagged `Picture::draft`; every other picture stays bit-exact.
|
||||
|
||||
## API
|
||||
|
||||
|
|
@ -41,6 +43,7 @@ oracles and fixture generators.
|
|||
use filmcraft_h264::Decoder;
|
||||
|
||||
let mut dec = Decoder::new(); // or Decoder::with_threads(n), Decoder::from_avcc(&avcc)?
|
||||
// dec.threads(): worker threads in use; dec.set_draft(true): draft mode (playback only)
|
||||
for (pts, access_unit) in access_units {
|
||||
for pic in dec.decode(access_unit, pts)? {
|
||||
// pic.width / pic.height (cropped), pic.y / pic.u / pic.v, pic.y_stride / pic.uv_stride,
|
||||
|
|
@ -67,6 +70,29 @@ bit-exact. Without the `threads` feature (or on wasm, or with `with_threads(1)`)
|
|||
With threads, `decode` returns pictures once their job has finished, so output lags input by up to
|
||||
roughly the number of threads.
|
||||
|
||||
Deblocking is row-parallel in this pipelined form: each picture's job filters macroblock row r as
|
||||
soon as row r + 1 is reconstructed (intra prediction of row r + 1 reads the unfiltered bottom
|
||||
line of row r) and publishes it one row later, while the jobs of other pictures run on other
|
||||
threads. A picture waiting on a reference therefore waits only for that reference's rows its
|
||||
motion vectors reach plus about two rows, so a chain of reference pictures (a seek from an IDR
|
||||
picture) advances a few rows behind its predecessor instead of a whole picture behind it. A
|
||||
separate deblocking thread per picture would shorten that lag by about one row and would need its
|
||||
own copy of the reconstructed rows and macroblock state; it is not worth it while the pool has
|
||||
more pictures in flight than cores. The hot loops are written to vectorise without `unsafe` or
|
||||
intrinsics: the edge filters process all 16 (luma) or 8 (chroma) lines of an edge at once with
|
||||
per-lane masks, the inverse transforms do their column pass across all columns, and the CABAC
|
||||
engine decodes a context-coded bin without branching on its outcome.
|
||||
|
||||
### Draft mode
|
||||
|
||||
`Decoder::set_draft(true)` (reduced-resolution playback only; FilmCraft enables it with Settings ▸
|
||||
Playback ▸ Draft decoding while the Program monitor plays at 1/2 or 1/4 resolution) skips the
|
||||
deblocking filter (8.7) of non-reference pictures (`nal_ref_idc` 0). No picture is predicted from
|
||||
a non-reference picture, and deblocking never changes the motion data that direct prediction
|
||||
reads, so every other picture is unchanged; the skipped pictures are flagged `Picture::draft` and
|
||||
the media stack never hands them to a paused monitor, a render or an export. On the 4K fixture
|
||||
below it saves ~9 % of the decoding work (the B-pyramid's non-reference half of the B pictures).
|
||||
|
||||
## Tests
|
||||
|
||||
`cargo test -p filmcraft-h264` (fixtures are generated on first use into `target/fixtures/h264/`
|
||||
|
|
@ -82,10 +108,16 @@ with ffmpeg + libx264; tests print a message and skip when `/opt/homebrew/bin/ff
|
|||
explicit bi-prediction. The expected output is known exactly and is also checked against ffmpeg.
|
||||
- Robustness: avcC input with pts round trip; randomly corrupted, truncated and garbage input never
|
||||
panics or deadlocks (single- and multi-threaded).
|
||||
- Conformance (`tests/conformance.rs`): every fixture is decoded single-threaded and frame-threaded
|
||||
and compared byte-for-byte with `ffmpeg -f rawvideo -pix_fmt yuv420p`; on mismatch the first
|
||||
differing frame, plane, sample position and macroblock are reported. Output pts/POC order is
|
||||
checked too.
|
||||
- Conformance (`tests/conformance.rs`): every fixture is decoded single-threaded, with 3 threads
|
||||
and with every core, and compared byte-for-byte with `ffmpeg -f rawvideo -pix_fmt yuv420p`; on
|
||||
mismatch the first differing frame, plane, sample position and macroblock are reported. Output
|
||||
pts/POC order is checked too, and that no picture is flagged draft without draft mode. Draft
|
||||
mode (`draft_mode_changes_only_flagged_non_reference_pictures`): on five B-pyramid / B-frame
|
||||
fixtures, single- and frame-threaded, every unflagged picture is bit-exact and only flagged
|
||||
pictures differ.
|
||||
- Vectorised kernels against their straightforward forms on random input: the deblocking edge
|
||||
filter against the per-line filter, the 4x4 / 8x8 inverse transforms against the per-column
|
||||
formulation.
|
||||
|
||||
### Fixture matrix (all bit-exact, single- and multi-threaded)
|
||||
|
||||
|
|
@ -146,9 +178,26 @@ Pro (14 cores) while the machine was heavily shared (load average 40-180), so th
|
|||
| 1 | ~105-122 (≈ 33 Mcycles per frame) |
|
||||
| 14 | ~500-600 |
|
||||
|
||||
4K High profile (testsrc2 + grain, 3840x2160, CABAC, B-pyramid, ~170 Mbit/s; the first 48 frames
|
||||
of the playback benchmark's `a2160.mp4`), CPU cycles per frame counted by the kernel
|
||||
(`proc_pid_rusage`; independent of machine load, all threads summed; bit-exact with ffmpeg):
|
||||
|
||||
| build | Mcycles / frame | instructions / frame |
|
||||
|---|---|---|
|
||||
| before M4.9 | 505 | 1232 M |
|
||||
| M4.9 | **390** (−23 %) | 1195 M |
|
||||
| M4.9, draft mode | **356** (−30 %) | 1068 M |
|
||||
|
||||
The same on the benchmark's 1080p fixture (`dec_h264_1080.mp4`, 120 frames): 132 → 100 Mcycles
|
||||
per frame (−24 %), 90 in draft mode. Single-threaded 4K time (`sample` profile) splits into CABAC
|
||||
residual decoding 41 %, deblocking 16 % (25-30 % before), other macroblock-layer parsing 11 %,
|
||||
inverse transforms and dequantisation 9 %, motion compensation 8 %, intra prediction 6 %, row and
|
||||
picture copies 4 %. At ~4.5 GHz a 4K frame takes ~85 ms of one
|
||||
core, so 23.976 fps needs about two cores' worth of frame threads.
|
||||
|
||||
The example `cargo run --release -p filmcraft-h264 --example h264dec -- in.h264 out.yuv` decodes a
|
||||
file; `H264_BENCH_ITERS=n` (and `H264_THREADS=t`) turns it into a benchmark, `H264_STATS=1` prints
|
||||
the coverage counters.
|
||||
file; `H264_BENCH_ITERS=n` (and `H264_THREADS=t`, `H264_DRAFT=1`) turns it into a benchmark,
|
||||
`H264_STATS=1` prints the coverage counters.
|
||||
|
||||
## Known gaps
|
||||
|
||||
|
|
@ -159,7 +208,8 @@ the coverage counters.
|
|||
- FMO (slice groups), SP/SI slices, data partitioning (Extended profile), MVC/SVC NAL units.
|
||||
- Error concealment is minimal: undecodable slices leave the prediction/previous content; missing
|
||||
references are replaced by other references.
|
||||
- Single-threaded speed has no SIMD-specific code yet (the hot loops are written to auto-vectorise).
|
||||
- No SIMD intrinsics (the hot loops are written to auto-vectorise); CABAC residual decoding is
|
||||
inherently serial and now dominates.
|
||||
|
||||
### Extending to High 10 and 4:2:2
|
||||
|
||||
|
|
|
|||
|
|
@ -1,5 +1,6 @@
|
|||
//! Decode an Annex-B H.264 file to raw yuv420p: `h264dec in.h264 [out.yuv]`.
|
||||
//! Prints timing information to stderr.
|
||||
//! Prints timing information to stderr. `H264_BENCH_ITERS=n` (with `H264_THREADS=t`, and
|
||||
//! `H264_DRAFT=1` for draft mode) turns it into a benchmark.
|
||||
|
||||
use filmcraft_h264::Decoder;
|
||||
use std::io::Write;
|
||||
|
|
@ -18,6 +19,7 @@ fn main() {
|
|||
let threads = std::env::var("H264_THREADS").ok().and_then(|v| v.parse::<usize>().ok());
|
||||
for _ in 0..n {
|
||||
let mut dec = threads.map(Decoder::with_threads).unwrap_or_default();
|
||||
dec.set_draft(std::env::var_os("H264_DRAFT").is_some());
|
||||
let t0 = std::time::Instant::now();
|
||||
frames = dec.decode(&data, 0).expect("decode").len() + dec.flush().len();
|
||||
best = best.min(t0.elapsed().as_secs_f64());
|
||||
|
|
|
|||
|
|
@ -145,35 +145,34 @@ impl<'a> Cabac<'a> {
|
|||
|
||||
#[inline(always)]
|
||||
fn renorm(&mut self) {
|
||||
if self.range < 256 {
|
||||
let shift = self.range.leading_zeros() - 23;
|
||||
self.range <<= shift;
|
||||
if self.bits < shift {
|
||||
self.refill();
|
||||
}
|
||||
self.bits -= shift;
|
||||
// codIRange < 512 always, so the shift is 0 when no renormalisation is needed: no branch on
|
||||
// the (unpredictable) range, only on the rare refill.
|
||||
let shift = self.range.leading_zeros() - 23;
|
||||
self.range <<= shift;
|
||||
if self.bits < shift {
|
||||
self.refill();
|
||||
}
|
||||
self.bits -= shift;
|
||||
}
|
||||
|
||||
/// 9.3.3.2.1, branch-free: the MPS / LPS outcome of a context-coded bin is close to random
|
||||
/// at high bit rates, so it selects values instead of jumping.
|
||||
#[inline(always)]
|
||||
pub fn decode_decision(&mut self, ctx_idx: usize) -> u32 {
|
||||
let s = self.ctx[ctx_idx] as usize;
|
||||
// Every ctxIdx is < 1024 and every state < 128: the masks only let the compiler drop the
|
||||
// bounds checks from the per-bin path.
|
||||
let ctx_idx = ctx_idx & (NUM_CTX - 1);
|
||||
let s = (self.ctx[ctx_idx] & 127) as usize;
|
||||
let q = ((self.range >> 6) & 3) as usize;
|
||||
let lps = RANGE_TAB_LPS[s >> 1][q] as u32;
|
||||
self.range -= lps;
|
||||
let scaled = (self.range as u64) << self.bits;
|
||||
let bin;
|
||||
if self.value >= scaled {
|
||||
bin = 1 - (s & 1) as u32;
|
||||
self.value -= scaled;
|
||||
self.range = lps;
|
||||
self.ctx[ctx_idx] = NEXT_STATE[s][1];
|
||||
} else {
|
||||
bin = (s & 1) as u32;
|
||||
self.ctx[ctx_idx] = NEXT_STATE[s][0];
|
||||
}
|
||||
let mps_range = self.range - lps;
|
||||
let scaled = (mps_range as u64) << self.bits;
|
||||
let is_lps = self.value >= scaled;
|
||||
self.value -= scaled & (is_lps as u64).wrapping_neg();
|
||||
self.range = if is_lps { lps } else { mps_range };
|
||||
self.ctx[ctx_idx] = NEXT_STATE[s][is_lps as usize];
|
||||
self.renorm();
|
||||
bin
|
||||
(s & 1) as u32 ^ is_lps as u32
|
||||
}
|
||||
|
||||
#[inline(always)]
|
||||
|
|
@ -183,12 +182,9 @@ impl<'a> Cabac<'a> {
|
|||
}
|
||||
self.bits -= 1;
|
||||
let scaled = (self.range as u64) << self.bits;
|
||||
if self.value >= scaled {
|
||||
self.value -= scaled;
|
||||
1
|
||||
} else {
|
||||
0
|
||||
}
|
||||
let one = self.value >= scaled;
|
||||
self.value -= scaled & (one as u64).wrapping_neg();
|
||||
one as u32
|
||||
}
|
||||
|
||||
pub fn decode_terminate(&mut self) -> u32 {
|
||||
|
|
|
|||
|
|
@ -72,13 +72,14 @@ fn motion_bs(p: &Motion, q: &Motion) -> u8 {
|
|||
}
|
||||
}
|
||||
|
||||
#[inline(always)]
|
||||
#[cfg(test)]
|
||||
fn clip3(lo: i32, hi: i32, v: i32) -> i32 {
|
||||
v.clamp(lo, hi)
|
||||
}
|
||||
|
||||
/// Filter one set of eight samples p3 p2 p1 p0 | q0 q1 q2 q3 across an edge (8.7.2.3 / 8.7.2.4).
|
||||
#[inline(always)]
|
||||
/// Filter one set of eight samples p3 p2 p1 p0 | q0 q1 q2 q3 across an edge (8.7.2.3 / 8.7.2.4):
|
||||
/// the per-line reference for [`filter_lanes`].
|
||||
#[cfg(test)]
|
||||
fn filter8(v: &mut [u8; 8], bs: u8, alpha: i32, beta: i32, tc0: i32, chroma: bool) {
|
||||
let p0 = v[3] as i32;
|
||||
let q0 = v[4] as i32;
|
||||
|
|
@ -143,51 +144,130 @@ struct EdgeParams {
|
|||
beta: i32,
|
||||
}
|
||||
|
||||
/// Filter a vertical edge whose q0 column is at `x`, for `n` lines starting at row `y`.
|
||||
/// Filter one edge for `N` lines at once (lane `i` = line `i` across the edge). `s[k][i]` holds
|
||||
/// sample p3 p2 p1 p0 q0 q1 q2 q3 (k = 0..8) of line `i`. Branch-free per lane, so the loops
|
||||
/// vectorise; bit-exact with [`filter8`] applied to every line (8.7.2.3 / 8.7.2.4).
|
||||
#[inline(always)]
|
||||
fn filter_vertical(pix: &mut [u8], stride: usize, x: usize, y: usize, n: usize, ep: &EdgeParams, chroma: bool) {
|
||||
let per = n / 4;
|
||||
for i in 0..n {
|
||||
let k = i / per;
|
||||
let b = ep.bs[k];
|
||||
if b == 0 {
|
||||
continue;
|
||||
fn filter_lanes<const N: usize>(s: &mut [[i16; N]; 8], ep: &EdgeParams, chroma: bool) {
|
||||
let per = N / 4;
|
||||
let (alpha, beta) = (ep.alpha as i16, ep.beta as i16);
|
||||
let mut bs_on = [false; N];
|
||||
let mut tc0 = [0i16; N];
|
||||
for k in 0..4 {
|
||||
for i in k * per..(k + 1) * per {
|
||||
bs_on[i] = ep.bs[k] != 0;
|
||||
tc0[i] = ep.tc0[k] as i16;
|
||||
}
|
||||
}
|
||||
let [p3, p2, p1, p0, q0, q1, q2, q3] = *s;
|
||||
// Masks are 0 / -1 per lane, selects are bitwise, and clipping to per-lane bounds uses
|
||||
// min / max (`clamp` would assert its bounds per lane), so every loop below vectorises.
|
||||
let lt = |x: i16, y: i16| -((x < y) as i16);
|
||||
let sel = |m: i16, a: i16, b: i16| (a & m) | (b & !m);
|
||||
let mut on = [0i16; N];
|
||||
let mut any = 0i16;
|
||||
for i in 0..N {
|
||||
on[i] = -(bs_on[i] as i16) & lt((p0[i] - q0[i]).abs(), alpha) & lt((p1[i] - p0[i]).abs(), beta) & lt((q1[i] - q0[i]).abs(), beta);
|
||||
any |= on[i];
|
||||
}
|
||||
if any == 0 {
|
||||
return;
|
||||
}
|
||||
// bS 4 is only ever assigned to all four segments of an edge
|
||||
let strong_edge = ep.bs[0] >= 4;
|
||||
if chroma {
|
||||
for i in 0..N {
|
||||
let (np0, nq0) = if strong_edge {
|
||||
((2 * p1[i] + p0[i] + q1[i] + 2) >> 2, (2 * q1[i] + q0[i] + p1[i] + 2) >> 2)
|
||||
} else {
|
||||
let tc = tc0[i] + 1;
|
||||
let delta = ((((q0[i] - p0[i]) << 2) + (p1[i] - q1[i]) + 4) >> 3).max(-tc).min(tc);
|
||||
((p0[i] + delta).clamp(0, 255), (q0[i] - delta).clamp(0, 255))
|
||||
};
|
||||
s[3][i] = sel(on[i], np0, p0[i]);
|
||||
s[4][i] = sel(on[i], nq0, q0[i]);
|
||||
}
|
||||
return;
|
||||
}
|
||||
if strong_edge {
|
||||
let lim = (alpha >> 2) + 2;
|
||||
for i in 0..N {
|
||||
let (a, b, c, d, e, f, g, h) = (p3[i], p2[i], p1[i], p0[i], q0[i], q1[i], q2[i], q3[i]);
|
||||
let strong = lt((d - e).abs(), lim);
|
||||
let sp = on[i] & lt((b - d).abs(), beta) & strong;
|
||||
let sq = on[i] & lt((g - e).abs(), beta) & strong;
|
||||
let wp0 = (2 * c + d + f + 2) >> 2;
|
||||
let wq0 = (2 * f + e + c + 2) >> 2;
|
||||
let sp0 = (b + 2 * c + 2 * d + 2 * e + f + 4) >> 3;
|
||||
let sq0 = (c + 2 * d + 2 * e + 2 * f + g + 4) >> 3;
|
||||
s[3][i] = sel(sp, sp0, sel(on[i], wp0, d));
|
||||
s[2][i] = sel(sp, (b + c + d + e + 2) >> 2, c);
|
||||
s[1][i] = sel(sp, (2 * a + 3 * b + c + d + e + 4) >> 3, b);
|
||||
s[4][i] = sel(sq, sq0, sel(on[i], wq0, e));
|
||||
s[5][i] = sel(sq, (d + e + f + g + 2) >> 2, f);
|
||||
s[6][i] = sel(sq, (2 * h + 3 * g + f + e + d + 4) >> 3, g);
|
||||
}
|
||||
} else {
|
||||
for i in 0..N {
|
||||
let ap = lt((p2[i] - p0[i]).abs(), beta);
|
||||
let aq = lt((q2[i] - q0[i]).abs(), beta);
|
||||
let t0 = tc0[i];
|
||||
let tc = t0 - ap - aq;
|
||||
let delta = ((((q0[i] - p0[i]) << 2) + (p1[i] - q1[i]) + 4) >> 3).max(-tc).min(tc);
|
||||
let avg = (p0[i] + q0[i] + 1) >> 1;
|
||||
let np1 = p1[i] + ((p2[i] + avg - (p1[i] << 1)) >> 1).max(-t0).min(t0);
|
||||
let nq1 = q1[i] + ((q2[i] + avg - (q1[i] << 1)) >> 1).max(-t0).min(t0);
|
||||
s[3][i] = sel(on[i], (p0[i] + delta).clamp(0, 255), p0[i]);
|
||||
s[4][i] = sel(on[i], (q0[i] - delta).clamp(0, 255), q0[i]);
|
||||
s[2][i] = sel(on[i] & ap, np1, p1[i]);
|
||||
s[5][i] = sel(on[i] & aq, nq1, q1[i]);
|
||||
}
|
||||
let o = (y + i) * stride + x - 4;
|
||||
let v: &mut [u8; 8] = (&mut pix[o..o + 8]).try_into().unwrap();
|
||||
filter8(v, b, ep.alpha, ep.beta, ep.tc0[k], chroma);
|
||||
}
|
||||
}
|
||||
|
||||
/// Filter a horizontal edge whose q0 row is `y`, for `n` columns starting at `x`.
|
||||
/// Filter a vertical edge whose q0 column is at `x`, for `N` lines starting at row `y`.
|
||||
#[inline(always)]
|
||||
fn filter_horizontal(pix: &mut [u8], stride: usize, x: usize, y: usize, n: usize, ep: &EdgeParams, chroma: bool) {
|
||||
let base = (y - 4) * stride + x;
|
||||
let rest = &mut pix[base..];
|
||||
let (r0, rest) = rest.split_at_mut(stride);
|
||||
let (r1, rest) = rest.split_at_mut(stride);
|
||||
let (r2, rest) = rest.split_at_mut(stride);
|
||||
let (r3, rest) = rest.split_at_mut(stride);
|
||||
let (r4, rest) = rest.split_at_mut(stride);
|
||||
let (r5, rest) = rest.split_at_mut(stride);
|
||||
let (r6, rest) = rest.split_at_mut(stride);
|
||||
let r7 = &mut rest[..n];
|
||||
let (r0, r1, r2, r3, r4, r5, r6) = (&mut r0[..n], &mut r1[..n], &mut r2[..n], &mut r3[..n], &mut r4[..n], &mut r5[..n], &mut r6[..n]);
|
||||
let per = n / 4;
|
||||
for c in 0..n {
|
||||
let k = c / per;
|
||||
let b = ep.bs[k];
|
||||
if b == 0 {
|
||||
continue;
|
||||
fn filter_vertical<const N: usize>(pix: &mut [u8], stride: usize, x: usize, y: usize, ep: &EdgeParams, chroma: bool) {
|
||||
// Each line's eight samples as one little-endian word: byte k = sample k (p3 .. q3).
|
||||
let mut w = [0u64; N];
|
||||
for (i, w) in w.iter_mut().enumerate() {
|
||||
let o = (y + i) * stride + x - 4;
|
||||
*w = u64::from_le_bytes(pix[o..o + 8].try_into().unwrap());
|
||||
}
|
||||
let mut s = [[0i16; N]; 8];
|
||||
for (k, row) in s.iter_mut().enumerate() {
|
||||
for i in 0..N {
|
||||
row[i] = ((w[i] >> (8 * k)) & 0xff) as i16;
|
||||
}
|
||||
}
|
||||
filter_lanes(&mut s, ep, chroma);
|
||||
for (i, w) in w.iter_mut().enumerate() {
|
||||
*w = (0..8).fold(0u64, |acc, k| acc | ((s[k][i] as u8 as u64) << (8 * k)));
|
||||
}
|
||||
for (i, w) in w.iter().enumerate() {
|
||||
let o = (y + i) * stride + x - 4;
|
||||
pix[o..o + 8].copy_from_slice(&w.to_le_bytes());
|
||||
}
|
||||
}
|
||||
|
||||
/// Filter a horizontal edge whose q0 row is `y`, for `N` columns starting at `x`.
|
||||
#[inline(always)]
|
||||
fn filter_horizontal<const N: usize>(pix: &mut [u8], stride: usize, x: usize, y: usize, ep: &EdgeParams, chroma: bool) {
|
||||
let base = (y - 4) * stride + x;
|
||||
let mut s = [[0i16; N]; 8];
|
||||
for (k, row) in s.iter_mut().enumerate() {
|
||||
let src: &[u8; N] = pix[base + k * stride..base + k * stride + N].try_into().unwrap();
|
||||
for i in 0..N {
|
||||
row[i] = src[i] as i16;
|
||||
}
|
||||
}
|
||||
filter_lanes(&mut s, ep, chroma);
|
||||
let (k0, k1) = if chroma { (3, 5) } else { (1, 7) };
|
||||
for (k, row) in s.iter().enumerate().take(k1).skip(k0) {
|
||||
let dst: &mut [u8; N] = (&mut pix[base + k * stride..base + k * stride + N]).try_into().unwrap();
|
||||
for i in 0..N {
|
||||
dst[i] = row[i] as u8;
|
||||
}
|
||||
let mut v = [r0[c], r1[c], r2[c], r3[c], r4[c], r5[c], r6[c], r7[c]];
|
||||
filter8(&mut v, b, ep.alpha, ep.beta, ep.tc0[k], chroma);
|
||||
r1[c] = v[1];
|
||||
r2[c] = v[2];
|
||||
r3[c] = v[3];
|
||||
r4[c] = v[4];
|
||||
r5[c] = v[5];
|
||||
r6[c] = v[6];
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -287,10 +367,15 @@ pub fn deblock_mb(pic: &mut PicState, addr: usize, mb_w: usize) {
|
|||
}
|
||||
let qpp = if e == 0 { p_qp[dir].map(|p| p.0).unwrap_or(q_qp) } else { q_qp };
|
||||
let ep = params(qpp, q_qp, bs[dir][e]);
|
||||
if ep.alpha == 0 || ep.beta == 0 {
|
||||
// filterSamplesFlag needs |p0 - q0| < alpha and |p1 - p0| < beta (8.7.2.2): no
|
||||
// sample of the edge changes (low QPs)
|
||||
continue;
|
||||
}
|
||||
if dir == 0 {
|
||||
filter_vertical(&mut pic.planes.y, width, mx * 16 + e * 4, my * 16, 16, &ep, false);
|
||||
filter_vertical::<16>(&mut pic.planes.y, width, mx * 16 + e * 4, my * 16, &ep, false);
|
||||
} else {
|
||||
filter_horizontal(&mut pic.planes.y, width, mx * 16, my * 16 + e * 4, 16, &ep, false);
|
||||
filter_horizontal::<16>(&mut pic.planes.y, width, mx * 16, my * 16 + e * 4, &ep, false);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -304,13 +389,156 @@ pub fn deblock_mb(pic: &mut PicState, addr: usize, mb_w: usize) {
|
|||
}
|
||||
let qpp = if e == 0 { p_qp[dir].map(|p| p.1[c]).unwrap_or(q_qpc[c]) } else { q_qpc[c] } as i32;
|
||||
let ep = params(qpp, q_qpc[c] as i32, bs[dir][e]);
|
||||
if ep.alpha == 0 || ep.beta == 0 {
|
||||
continue;
|
||||
}
|
||||
let plane = if c == 0 { &mut pic.planes.cb } else { &mut pic.planes.cr };
|
||||
if dir == 0 {
|
||||
filter_vertical(plane, cwidth, mx * 8 + ce * 4, my * 8, 8, &ep, true);
|
||||
filter_vertical::<8>(plane, cwidth, mx * 8 + ce * 4, my * 8, &ep, true);
|
||||
} else {
|
||||
filter_horizontal(plane, cwidth, mx * 8, my * 8 + ce * 4, 8, &ep, true);
|
||||
filter_horizontal::<8>(plane, cwidth, mx * 8, my * 8 + ce * 4, &ep, true);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
/// The vectorised edge filter equals the per-line reference [`filter8`] on random edges
|
||||
/// (small differences across the edge so that every filter branch is taken).
|
||||
#[test]
|
||||
fn lane_filter_matches_per_line_reference() {
|
||||
let mut rng = 0x9e37_79b9_7f4a_7c15u64;
|
||||
let mut next = move || {
|
||||
rng ^= rng << 13;
|
||||
rng ^= rng >> 7;
|
||||
rng ^= rng << 17;
|
||||
rng
|
||||
};
|
||||
for iter in 0..20_000 {
|
||||
let chroma = iter % 3 == 0;
|
||||
let strong = iter % 4 == 1;
|
||||
let bs: [u8; 4] = if strong { [4; 4] } else { std::array::from_fn(|_| (next() % 4) as u8) };
|
||||
let index_a = (next() % 52) as usize;
|
||||
let index_b = (next() % 52) as usize;
|
||||
let tc = |b: u8| if (1..4).contains(&b) { TC0[index_a][b as usize - 1] as i32 } else { 0 };
|
||||
let ep = EdgeParams { bs, tc0: bs.map(tc), alpha: ALPHA[index_a] as i32, beta: BETA[index_b] as i32 };
|
||||
let base = (next() % 256) as i32;
|
||||
let spread = 1 + (next() % 24) as i32;
|
||||
let mut lines = [[0u8; 8]; 16];
|
||||
for l in lines.iter_mut() {
|
||||
for v in l.iter_mut() {
|
||||
*v = (base + (next() % (2 * spread as u64 + 1)) as i32 - spread).clamp(0, 255) as u8;
|
||||
}
|
||||
}
|
||||
let n = if chroma { 8 } else { 16 };
|
||||
let mut want = lines;
|
||||
for (i, l) in want.iter_mut().enumerate().take(n) {
|
||||
let k = i / (n / 4);
|
||||
if bs[k] != 0 {
|
||||
filter8(l, bs[k], ep.alpha, ep.beta, ep.tc0[k], chroma);
|
||||
}
|
||||
}
|
||||
let got: Vec<[u8; 8]> = if chroma {
|
||||
let mut s = [[0i16; 8]; 8];
|
||||
for i in 0..8 {
|
||||
for k in 0..8 {
|
||||
s[k][i] = lines[i][k] as i16;
|
||||
}
|
||||
}
|
||||
filter_lanes(&mut s, &ep, true);
|
||||
(0..8).map(|i| std::array::from_fn(|k| s[k][i] as u8)).collect()
|
||||
} else {
|
||||
let mut s = [[0i16; 16]; 8];
|
||||
for i in 0..16 {
|
||||
for k in 0..8 {
|
||||
s[k][i] = lines[i][k] as i16;
|
||||
}
|
||||
}
|
||||
filter_lanes(&mut s, &ep, false);
|
||||
(0..16).map(|i| std::array::from_fn(|k| s[k][i] as u8)).collect()
|
||||
};
|
||||
assert_eq!(&got[..], &want[..n], "iteration {iter}: bs {bs:?} indexA {index_a} indexB {index_b}");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod speed {
|
||||
use super::*;
|
||||
|
||||
/// Rough timing of the edge filters (`cargo test --release -p filmcraft-h264 --lib edge_filter_speed -- --ignored --nocapture`).
|
||||
#[test]
|
||||
#[ignore]
|
||||
fn edge_filter_speed() {
|
||||
let (w, h) = (1024usize, 256usize);
|
||||
let mut pix: Vec<u8> = (0..w * h).map(|i| (128 + ((i * 7919) % 13) as i32 - 6) as u8).collect();
|
||||
let eps = [
|
||||
EdgeParams { bs: [4; 4], tc0: [0; 4], alpha: 40, beta: 9 },
|
||||
EdgeParams { bs: [3; 4], tc0: [2; 4], alpha: 40, beta: 9 },
|
||||
EdgeParams { bs: [2, 1, 0, 2], tc0: [1, 1, 0, 1], alpha: 40, beta: 9 },
|
||||
];
|
||||
let t0 = std::time::Instant::now();
|
||||
let mut n = 0;
|
||||
for _ in 0..20 {
|
||||
for my in 1..h / 16 {
|
||||
for mx in 1..w / 16 {
|
||||
for (j, ep) in eps.iter().enumerate() {
|
||||
filter_horizontal::<16>(&mut pix, w, mx * 16, my * 16 + j * 4, ep, false);
|
||||
n += 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
let dt = t0.elapsed();
|
||||
let tv = std::time::Instant::now();
|
||||
for _ in 0..20 {
|
||||
for my in 1..h / 16 {
|
||||
for mx in 1..w / 16 {
|
||||
for (j, ep) in eps.iter().enumerate() {
|
||||
filter_vertical::<16>(&mut pix, w, mx * 16 + j * 4, my * 16, ep, false);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
println!("vertical: {:.1} ns/edge", tv.elapsed().as_nanos() as f64 / n as f64);
|
||||
let t1 = std::time::Instant::now();
|
||||
for _ in 0..20 {
|
||||
for my in 1..h / 16 {
|
||||
for mx in 1..w / 16 {
|
||||
for ep in eps.iter() {
|
||||
for (vert, x, y) in [(true, mx * 16, my * 16), (false, mx * 16, my * 16)] {
|
||||
for i in 0..16 {
|
||||
let b = ep.bs[i / 4];
|
||||
if b == 0 {
|
||||
continue;
|
||||
}
|
||||
let mut v = [0u8; 8];
|
||||
for k in 0..8 {
|
||||
v[k] = if vert { pix[(y + i) * w + x - 4 + k] } else { pix[(y - 4 + k) * w + x + i] };
|
||||
}
|
||||
filter8(&mut v, b, ep.alpha, ep.beta, ep.tc0[i / 4], false);
|
||||
for k in 1..7 {
|
||||
if vert {
|
||||
pix[(y + i) * w + x - 4 + k] = v[k];
|
||||
} else {
|
||||
pix[(y - 4 + k) * w + x + i] = v[k];
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
println!("scalar reference (both): {:.1} ns/edge", t1.elapsed().as_nanos() as f64 / (2 * n) as f64);
|
||||
println!(
|
||||
"horizontal: {n} luma edges in {dt:?}: {:.1} ns/edge (checksum {})",
|
||||
dt.as_nanos() as f64 / n as f64,
|
||||
pix.iter().map(|&v| v as u64).sum::<u64>()
|
||||
);
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -63,6 +63,8 @@ pub struct DecodeStats {
|
|||
pub frame_num_gaps: u64,
|
||||
pub weighted_slices: u64,
|
||||
pub temporal_direct_slices: u64,
|
||||
/// Non-reference pictures decoded in draft mode (deblocking skipped).
|
||||
pub draft_pictures: u64,
|
||||
}
|
||||
|
||||
/// State shared with decoding jobs.
|
||||
|
|
@ -93,6 +95,9 @@ pub struct Decoder {
|
|||
pool: Option<rayon::ThreadPool>,
|
||||
in_flight: VecDeque<FrameRef>,
|
||||
max_in_flight: usize,
|
||||
threads: usize,
|
||||
/// Draft mode: skip deblocking of non-reference pictures ([`Decoder::set_draft`]).
|
||||
draft: bool,
|
||||
}
|
||||
|
||||
impl Default for Decoder {
|
||||
|
|
@ -122,7 +127,9 @@ fn is_new_picture(prev: &SliceHeader, sh: &SliceHeader, sps: &Sps) -> bool {
|
|||
false
|
||||
}
|
||||
|
||||
fn default_threads() -> usize {
|
||||
/// The worker threads [`Decoder::new`] uses: the available cores (at most 16) with the `threads`
|
||||
/// feature on native targets, 1 otherwise.
|
||||
pub fn default_threads() -> usize {
|
||||
#[cfg(all(feature = "threads", not(target_arch = "wasm32")))]
|
||||
{
|
||||
std::thread::available_parallelism().map(|n| n.get()).unwrap_or(1).min(16)
|
||||
|
|
@ -148,6 +155,10 @@ impl Decoder {
|
|||
} else {
|
||||
None
|
||||
};
|
||||
#[cfg(feature = "threads")]
|
||||
let workers = if pool.is_some() { threads } else { 1 };
|
||||
#[cfg(not(feature = "threads"))]
|
||||
let workers = 1;
|
||||
Decoder {
|
||||
spss: vec![None; 32],
|
||||
ppss: vec![None; 256],
|
||||
|
|
@ -165,9 +176,30 @@ impl Decoder {
|
|||
pool,
|
||||
in_flight: VecDeque::new(),
|
||||
max_in_flight: threads.max(1) + 2,
|
||||
threads: workers,
|
||||
draft: false,
|
||||
}
|
||||
}
|
||||
|
||||
/// Worker threads decoding pictures (1: on the calling thread).
|
||||
pub fn threads(&self) -> usize {
|
||||
self.threads
|
||||
}
|
||||
|
||||
/// Draft mode (off by default) for reduced-resolution playback: pictures submitted from now
|
||||
/// on that are non-reference pictures (`nal_ref_idc` 0) skip the deblocking filter (8.7).
|
||||
/// Nothing predicts from a non-reference picture (and deblocking never changes the motion
|
||||
/// data used for direct prediction), so every other picture stays bit-exact; the draft
|
||||
/// pictures themselves are approximate and flagged [`Picture::draft`]. Never use it for
|
||||
/// export or for a frame shown while paused.
|
||||
pub fn set_draft(&mut self, on: bool) {
|
||||
self.draft = on;
|
||||
}
|
||||
|
||||
pub fn draft(&self) -> bool {
|
||||
self.draft
|
||||
}
|
||||
|
||||
/// Configure from an `avcC` (AVCDecoderConfigurationRecord) box payload.
|
||||
pub fn from_avcc(avcc: &[u8]) -> Result<Self> {
|
||||
let mut d = Decoder::new();
|
||||
|
|
@ -450,19 +482,20 @@ impl Decoder {
|
|||
fn submit_pending(&mut self) {
|
||||
let Some(p) = self.pending.take() else { return };
|
||||
let PendingPic { frame, sps, first, poc, pts, key, slices } = p;
|
||||
self.dispatch(frame.clone(), slices);
|
||||
let draft = self.draft && first.nal_ref_idc == 0;
|
||||
self.dispatch(frame.clone(), slices, draft);
|
||||
self.poc_state.update(&first, &poc);
|
||||
if first.nal_ref_idc != 0 {
|
||||
self.prev_ref_frame_num = if first.has_mmco5() { 0 } else { first.frame_num };
|
||||
}
|
||||
let meta = Arc::new(output_meta(&sps, pts, key));
|
||||
let meta = Arc::new(OutputMeta { draft, ..output_meta(&sps, pts, key) });
|
||||
let mut outs = Vec::new();
|
||||
let fpoc = frame.poc;
|
||||
self.dpb.store_picture(&first, frame, fpoc, sps.max_frame_num(), sps.max_num_ref_frames as usize, meta, &mut outs);
|
||||
self.emit(outs);
|
||||
}
|
||||
|
||||
fn dispatch(&mut self, frame: FrameRef, slices: Vec<SliceJob>) {
|
||||
fn dispatch(&mut self, frame: FrameRef, slices: Vec<SliceJob>, draft: bool) {
|
||||
let shared = self.shared.clone();
|
||||
#[cfg(feature = "threads")]
|
||||
if let Some(pool) = &self.pool {
|
||||
|
|
@ -473,10 +506,10 @@ impl Decoder {
|
|||
}
|
||||
}
|
||||
self.in_flight.push_back(frame.clone());
|
||||
pool.spawn_fifo(move || run_job(frame, slices, &shared));
|
||||
pool.spawn_fifo(move || run_job(frame, slices, &shared, draft));
|
||||
return;
|
||||
}
|
||||
run_job(frame, slices, &shared);
|
||||
run_job(frame, slices, &shared, draft);
|
||||
}
|
||||
|
||||
fn emit(&mut self, outs: Vec<Output>) {
|
||||
|
|
@ -485,7 +518,7 @@ impl Decoder {
|
|||
}
|
||||
|
||||
/// Decode all slices of one picture and publish its rows.
|
||||
fn run_job(frame: FrameRef, slices: Vec<SliceJob>, shared: &Shared) {
|
||||
fn run_job(frame: FrameRef, slices: Vec<SliceJob>, shared: &Shared, draft: bool) {
|
||||
let reuse = shared.pool.lock().ok().and_then(|mut p| p.pop());
|
||||
let mut pic = match reuse {
|
||||
Some(mut p) if p.mb_w == frame.mb_w && p.mb_h == frame.mb_h() => {
|
||||
|
|
@ -494,6 +527,7 @@ fn run_job(frame: FrameRef, slices: Vec<SliceJob>, shared: &Shared) {
|
|||
}
|
||||
_ => PicState::new(frame.clone()),
|
||||
};
|
||||
pic.skip_deblock = draft;
|
||||
let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
|
||||
let mut first_err = None;
|
||||
for s in &slices {
|
||||
|
|
@ -521,6 +555,7 @@ fn run_job(frame: FrameRef, slices: Vec<SliceJob>, shared: &Shared) {
|
|||
};
|
||||
if let Ok(mut s) = shared.stats.lock() {
|
||||
s.pictures += 1;
|
||||
s.draft_pictures += draft as u64;
|
||||
for st in &pic.mbs {
|
||||
match st.kind {
|
||||
MbKind::I4x4 => s.mb_i4x4 += 1,
|
||||
|
|
@ -570,6 +605,7 @@ fn output_meta(sps: &Sps, pts: i64, key: bool) -> OutputMeta {
|
|||
transfer_characteristics: vui.transfer_characteristics,
|
||||
matrix_coefficients: vui.matrix_coefficients,
|
||||
sar: vui.sar,
|
||||
draft: false,
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -598,5 +634,6 @@ fn make_picture(o: &Output) -> Picture {
|
|||
matrix: o.meta.matrix_coefficients,
|
||||
},
|
||||
sar: o.meta.sar,
|
||||
draft: o.meta.draft,
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -24,6 +24,8 @@ pub struct OutputMeta {
|
|||
pub transfer_characteristics: u8,
|
||||
pub matrix_coefficients: u8,
|
||||
pub sar: (u16, u16),
|
||||
/// Decoded without deblocking (draft mode, non-reference picture).
|
||||
pub draft: bool,
|
||||
}
|
||||
|
||||
pub struct DpbEntry {
|
||||
|
|
|
|||
|
|
@ -41,7 +41,7 @@ mod synth_tests;
|
|||
mod tables;
|
||||
mod transform;
|
||||
|
||||
pub use decoder::{DecodeStats, Decoder};
|
||||
pub use decoder::{DecodeStats, Decoder, default_threads};
|
||||
pub use error::{Error, Result};
|
||||
|
||||
/// Colour description from the VUI (ITU-T H.273 code points).
|
||||
|
|
@ -77,4 +77,8 @@ pub struct Picture {
|
|||
pub color: ColorInfo,
|
||||
/// Sample aspect ratio (0, 0 when unspecified).
|
||||
pub sar: (u16, u16),
|
||||
/// Decoded in draft mode ([`Decoder::set_draft`]): a non-reference picture whose deblocking
|
||||
/// filter was skipped. Its samples are approximate (not the conforming output); no other
|
||||
/// picture is affected.
|
||||
pub draft: bool,
|
||||
}
|
||||
|
|
|
|||
|
|
@ -40,6 +40,8 @@ pub struct PicState {
|
|||
rows_recon: usize,
|
||||
rows_deblocked: usize,
|
||||
rows_published: usize,
|
||||
/// Draft mode for a non-reference picture: rows are published without deblocking.
|
||||
pub skip_deblock: bool,
|
||||
}
|
||||
|
||||
impl PicState {
|
||||
|
|
@ -58,6 +60,7 @@ impl PicState {
|
|||
rows_recon: 0,
|
||||
rows_deblocked: 0,
|
||||
rows_published: 0,
|
||||
skip_deblock: false,
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -76,6 +79,7 @@ impl PicState {
|
|||
self.rows_recon = 0;
|
||||
self.rows_deblocked = 0;
|
||||
self.rows_published = 0;
|
||||
self.skip_deblock = false;
|
||||
}
|
||||
|
||||
/// Record a decoded macroblock and deblock / publish rows that became final.
|
||||
|
|
@ -100,8 +104,10 @@ impl PicState {
|
|||
let deblock_limit = if finished { self.mb_h } else { self.rows_recon.saturating_sub(1) };
|
||||
while self.rows_deblocked < deblock_limit {
|
||||
let r = self.rows_deblocked;
|
||||
for addr in r * self.mb_w..(r + 1) * self.mb_w {
|
||||
deblock::deblock_mb(self, addr, self.mb_w);
|
||||
if !self.skip_deblock {
|
||||
for addr in r * self.mb_w..(r + 1) * self.mb_w {
|
||||
deblock::deblock_mb(self, addr, self.mb_w);
|
||||
}
|
||||
}
|
||||
self.rows_deblocked += 1;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -88,28 +88,34 @@ fn clip_u8(v: i32) -> u8 {
|
|||
}
|
||||
|
||||
/// Inverse 4x4 transform of raster-order `d` and add to `dst` (8.5.12.2, 8.5.14).
|
||||
///
|
||||
/// Rows first, then columns (the spec's order: the `>> 1` steps make it matter). The column
|
||||
/// pass and the reconstruction run across all columns at once so they vectorise.
|
||||
pub fn idct4_add(d: &[i32; 16], dst: &mut [u8], stride: usize) {
|
||||
let mut t = [0i32; 16];
|
||||
for i in 0..4 {
|
||||
let mut t = [[0i32; 4]; 4];
|
||||
for (i, row) in t.iter_mut().enumerate() {
|
||||
let r = &d[i * 4..i * 4 + 4];
|
||||
let e = r[0] + r[2];
|
||||
let f = r[0] - r[2];
|
||||
let g = (r[1] >> 1) - r[3];
|
||||
let h = r[1] + (r[3] >> 1);
|
||||
t[i * 4] = e + h;
|
||||
t[i * 4 + 1] = f + g;
|
||||
t[i * 4 + 2] = f - g;
|
||||
t[i * 4 + 3] = e - h;
|
||||
*row = [e + h, f + g, f - g, e - h];
|
||||
}
|
||||
let mut o = [[0i32; 4]; 4];
|
||||
for j in 0..4 {
|
||||
let e = t[j] + t[8 + j];
|
||||
let f = t[j] - t[8 + j];
|
||||
let g = (t[4 + j] >> 1) - t[12 + j];
|
||||
let h = t[4 + j] + (t[12 + j] >> 1);
|
||||
let col = [e + h, f + g, f - g, e - h];
|
||||
for i in 0..4 {
|
||||
let p = &mut dst[i * stride + j];
|
||||
*p = clip_u8(*p as i32 + ((col[i] + 32) >> 6));
|
||||
let e = t[0][j] + t[2][j];
|
||||
let f = t[0][j] - t[2][j];
|
||||
let g = (t[1][j] >> 1) - t[3][j];
|
||||
let h = t[1][j] + (t[3][j] >> 1);
|
||||
o[0][j] = e + h;
|
||||
o[1][j] = f + g;
|
||||
o[2][j] = f - g;
|
||||
o[3][j] = e - h;
|
||||
}
|
||||
for (i, row) in o.iter().enumerate() {
|
||||
let line: &mut [u8; 4] = (&mut dst[i * stride..i * stride + 4]).try_into().unwrap();
|
||||
for (p, &v) in line.iter_mut().zip(row) {
|
||||
*p = clip_u8(*p as i32 + ((v + 32) >> 6));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -146,18 +152,24 @@ fn idct8_1d(d: [i32; 8]) -> [i32; 8] {
|
|||
[b0 + b7, b2 + b5, b4 + b3, b6 + b1, b6 - b1, b4 - b3, b2 - b5, b0 - b7]
|
||||
}
|
||||
|
||||
/// Inverse 8x8 transform of raster-order `d` and add to `dst` (8.5.13.2).
|
||||
/// Inverse 8x8 transform of raster-order `d` and add to `dst` (8.5.13.2): rows, then columns,
|
||||
/// the column pass and reconstruction across all columns at once (they vectorise).
|
||||
pub fn idct8_add(d: &[i32; 64], dst: &mut [u8], stride: usize) {
|
||||
let mut t = [0i32; 64];
|
||||
for i in 0..8 {
|
||||
let row: [i32; 8] = d[i * 8..i * 8 + 8].try_into().unwrap();
|
||||
t[i * 8..i * 8 + 8].copy_from_slice(&idct8_1d(row));
|
||||
let mut t = [[0i32; 8]; 8];
|
||||
for (i, row) in t.iter_mut().enumerate() {
|
||||
*row = idct8_1d(d[i * 8..i * 8 + 8].try_into().unwrap());
|
||||
}
|
||||
let mut o = [[0i32; 8]; 8];
|
||||
for j in 0..8 {
|
||||
let col = idct8_1d(std::array::from_fn(|i| t[i * 8 + j]));
|
||||
let col = idct8_1d(std::array::from_fn(|i| t[i][j]));
|
||||
for i in 0..8 {
|
||||
let p = &mut dst[i * stride + j];
|
||||
*p = clip_u8(*p as i32 + ((col[i] + 32) >> 6));
|
||||
o[i][j] = col[i];
|
||||
}
|
||||
}
|
||||
for (i, row) in o.iter().enumerate() {
|
||||
let line: &mut [u8; 8] = (&mut dst[i * stride..i * stride + 8]).try_into().unwrap();
|
||||
for (p, &v) in line.iter_mut().zip(row) {
|
||||
*p = clip_u8(*p as i32 + ((v + 32) >> 6));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -166,6 +178,52 @@ pub fn idct8_add(d: &[i32; 64], dst: &mut [u8], stride: usize) {
|
|||
mod tests {
|
||||
use super::*;
|
||||
|
||||
/// The straightforward per-column formulation (8.5.12.2 / 8.5.13.2) the vectorised
|
||||
/// transforms must equal.
|
||||
fn reference_add(d: &[i32], n: usize, dst: &mut [u8], stride: usize) {
|
||||
let one = |v: &[i32]| -> Vec<i32> {
|
||||
if n == 4 {
|
||||
let (e, f, g, h) = (v[0] + v[2], v[0] - v[2], (v[1] >> 1) - v[3], v[1] + (v[3] >> 1));
|
||||
vec![e + h, f + g, f - g, e - h]
|
||||
} else {
|
||||
idct8_1d(v.try_into().unwrap()).to_vec()
|
||||
}
|
||||
};
|
||||
let rows: Vec<Vec<i32>> = (0..n).map(|i| one(&d[i * n..i * n + n])).collect();
|
||||
for j in 0..n {
|
||||
let col = one(&(0..n).map(|i| rows[i][j]).collect::<Vec<_>>());
|
||||
for i in 0..n {
|
||||
let p = &mut dst[i * stride + j];
|
||||
*p = (*p as i32 + ((col[i] + 32) >> 6)).clamp(0, 255) as u8;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn vectorised_transforms_match_reference() {
|
||||
let mut rng = 0x1234_5678_9abc_def1u64;
|
||||
let mut next = move || {
|
||||
rng ^= rng << 13;
|
||||
rng ^= rng >> 7;
|
||||
rng ^= rng << 17;
|
||||
rng
|
||||
};
|
||||
for iter in 0..5000 {
|
||||
let range = [64u64, 1024, 8192, 1 << 15][iter % 4];
|
||||
let d4: [i32; 16] = std::array::from_fn(|_| (next() % (2 * range + 1)) as i32 - range as i32);
|
||||
let d8: [i32; 64] = std::array::from_fn(|_| (next() % (2 * range + 1)) as i32 - range as i32);
|
||||
let base: Vec<u8> = (0..24 * 8).map(|_| next() as u8).collect();
|
||||
let (mut a, mut b) = (base.clone(), base.clone());
|
||||
idct4_add(&d4, &mut a[3..], 24);
|
||||
reference_add(&d4, 4, &mut b[3..], 24);
|
||||
assert_eq!(a, b, "4x4 iteration {iter}");
|
||||
let (mut a, mut b) = (base.clone(), base);
|
||||
idct8_add(&d8, &mut a[5..], 24);
|
||||
reference_add(&d8, 8, &mut b[5..], 24);
|
||||
assert_eq!(a, b, "8x8 iteration {iter}");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn idct4_dc_only_matches_shortcut() {
|
||||
let mut d = [0i32; 16];
|
||||
|
|
|
|||
|
|
@ -422,8 +422,15 @@ pub fn decode_file(path: &Path) -> Result<Vec<Picture>, DecodeFailure> {
|
|||
|
||||
/// Decode with `threads` worker threads (0 = decoder default).
|
||||
pub fn decode_file_threads(path: &Path, threads: usize) -> Result<Vec<Picture>, DecodeFailure> {
|
||||
decode_file_opts(path, threads, false)
|
||||
}
|
||||
|
||||
/// [`decode_file_threads`] with draft mode ([`Decoder::set_draft`]) on or off.
|
||||
pub fn decode_file_opts(path: &Path, threads: usize, draft: bool) -> Result<Vec<Picture>, DecodeFailure> {
|
||||
let data = std::fs::read(path).unwrap();
|
||||
let mut dec = if threads == 0 { Decoder::new() } else { Decoder::with_threads(threads) };
|
||||
assert!(!dec.draft(), "draft mode is off by default");
|
||||
dec.set_draft(draft);
|
||||
let mut out = Vec::new();
|
||||
for (i, au) in split_access_units(&data).into_iter().enumerate() {
|
||||
match dec.decode(au, i as i64) {
|
||||
|
|
@ -494,8 +501,9 @@ pub fn check_fixture(name: &str) -> Result<bool, String> {
|
|||
let f = fixture(name);
|
||||
let Some((h264, yuv)) = ensure(f) else { return Ok(false) };
|
||||
let reference = std::fs::read(&yuv).unwrap();
|
||||
// single-threaded and frame-threaded decoding must both be bit-exact
|
||||
for threads in [1, 0] {
|
||||
// single-threaded and frame-threaded decoding (an odd thread count and every core) must all
|
||||
// be bit-exact
|
||||
for threads in [1, 3, 0] {
|
||||
let pics = match decode_file_threads(&h264, threads) {
|
||||
Ok(p) => p,
|
||||
Err((au, e, partial)) => {
|
||||
|
|
@ -505,6 +513,9 @@ pub fn check_fixture(name: &str) -> Result<bool, String> {
|
|||
}
|
||||
};
|
||||
compare(&pics, &reference, f.width as usize, f.height as usize).map_err(|e| format!("{name} (threads {threads}): {e}"))?;
|
||||
if pics.iter().any(|p| p.draft) {
|
||||
return Err(format!("{name} (threads {threads}): draft picture without draft mode"));
|
||||
}
|
||||
check_pts(&pics).map_err(|e| format!("{name} (threads {threads}): {e}"))?;
|
||||
}
|
||||
Ok(true)
|
||||
|
|
|
|||
|
|
@ -56,6 +56,48 @@ fixture_tests!(
|
|||
vt_high_1080p,
|
||||
);
|
||||
|
||||
/// Draft mode (reduced-resolution playback) skips deblocking of non-reference pictures only:
|
||||
/// every other picture stays bit-exact with ffmpeg, single- and frame-threaded, the skipped ones
|
||||
/// are flagged, and they are the only ones that differ.
|
||||
#[test]
|
||||
fn draft_mode_changes_only_flagged_non_reference_pictures() {
|
||||
let mut checked = 0;
|
||||
for name in ["main_cabac_b", "bpyramid", "cavlc_b", "direct_temporal", "weightb"] {
|
||||
let f = common::fixture(name);
|
||||
let Some((h264, yuv)) = common::ensure(f) else {
|
||||
eprintln!("skipped {name}");
|
||||
continue;
|
||||
};
|
||||
let reference = std::fs::read(&yuv).unwrap();
|
||||
let (w, h) = (f.width as usize, f.height as usize);
|
||||
let fsize = w * h + 2 * w.div_ceil(2) * h.div_ceil(2);
|
||||
for threads in [1, 0] {
|
||||
let pics = common::decode_file_opts(&h264, threads, true).unwrap_or_else(|(au, e, _)| panic!("{name}: error at {au}: {e}"));
|
||||
common::check_pts(&pics).unwrap_or_else(|e| panic!("{name}: {e}"));
|
||||
assert_eq!(pics.len() * fsize, reference.len(), "{name}: picture count");
|
||||
let (mut draft, mut changed) = (0, 0);
|
||||
for (i, p) in pics.iter().enumerate() {
|
||||
let r = &reference[i * fsize..(i + 1) * fsize];
|
||||
let same = p.y == r[..w * h] && [&p.u[..], &p.v[..]].concat() == r[w * h..];
|
||||
if p.draft {
|
||||
draft += 1;
|
||||
changed += !same as usize;
|
||||
} else {
|
||||
assert!(same, "{name} (threads {threads}): picture {i} (not draft) differs from the reference");
|
||||
}
|
||||
}
|
||||
assert!(draft > 0, "{name}: has non-reference pictures");
|
||||
if name != "no_deblock" {
|
||||
assert!(changed > 0, "{name}: skipping deblocking changes some draft picture");
|
||||
}
|
||||
checked += 1;
|
||||
}
|
||||
}
|
||||
if checked == 0 {
|
||||
eprintln!("skipped: no fixtures");
|
||||
}
|
||||
}
|
||||
|
||||
/// Print which coding tools each fixture exercises (run with `-- --ignored --nocapture`).
|
||||
#[test]
|
||||
#[ignore]
|
||||
|
|
|
|||
|
|
@ -9,6 +9,10 @@
|
|||
//! The same way, [`with_catch_up`] tells sources which frames before the requested one are late
|
||||
//! (playback has passed them): a decoder that has to decode forward to the frame may skip
|
||||
//! pictures that only produce late frames and that nothing else references.
|
||||
//!
|
||||
//! [`with_draft`] marks reduced-resolution playback (opt-in): decoders may take spec-safe
|
||||
//! shortcuts that only change pictures nothing references (H.264: no deblocking of non-reference
|
||||
//! pictures). Never set for exports, renders or a frame shown while paused.
|
||||
|
||||
use std::cell::{Cell, RefCell};
|
||||
use std::sync::Arc;
|
||||
|
|
@ -19,6 +23,22 @@ use filmcraft_time::Tick;
|
|||
thread_local! {
|
||||
static CURRENT: RefCell<Option<Arc<AtomicBool>>> = const { RefCell::new(None) };
|
||||
static CATCH_UP: Cell<Option<Tick>> = const { Cell::new(None) };
|
||||
static DRAFT: Cell<bool> = const { Cell::new(false) };
|
||||
}
|
||||
|
||||
/// Run `f` with the draft-decoding hint `on` (restoring the previous one after). Frames decoded
|
||||
/// in draft mode are only handed to requests made in draft mode.
|
||||
pub fn with_draft<R>(on: bool, f: impl FnOnce() -> R) -> R {
|
||||
let prev = DRAFT.with(|c| c.replace(on));
|
||||
let r = f();
|
||||
DRAFT.with(|c| c.set(prev));
|
||||
r
|
||||
}
|
||||
|
||||
/// Whether the work running on this thread accepts draft-quality decoding (off unless inside
|
||||
/// [`with_draft`]`(true, …)`).
|
||||
pub fn draft() -> bool {
|
||||
DRAFT.with(Cell::get)
|
||||
}
|
||||
|
||||
/// Run `f` with the catch-up hint `margin` (restoring the previous one after): frames shown more
|
||||
|
|
@ -66,6 +86,17 @@ mod tests {
|
|||
assert!(!cancelled());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn draft_is_off_by_default_and_scoped() {
|
||||
assert!(!draft());
|
||||
with_draft(true, || {
|
||||
assert!(draft());
|
||||
with_draft(false, || assert!(!draft()));
|
||||
assert!(draft());
|
||||
});
|
||||
assert!(!draft());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn catch_up_is_scoped_to_the_closure() {
|
||||
assert_eq!(catch_up(), None);
|
||||
|
|
|
|||
|
|
@ -442,7 +442,7 @@ pub fn interpolation_blend(item: &TrackItem, t: Tick, src_rate: filmcraft_time::
|
|||
}
|
||||
|
||||
/// Largest power-of-two box decimation that keeps at least `target_w` pixels of width.
|
||||
fn decimation(have_w: f32, target_w: f32) -> usize {
|
||||
pub(crate) fn decimation(have_w: f32, target_w: f32) -> usize {
|
||||
let mut n = 1usize;
|
||||
while n < 16 && have_w / (n as f32 * 2.0) >= target_w.max(1.0) {
|
||||
n *= 2;
|
||||
|
|
|
|||
|
|
@ -187,6 +187,15 @@ fn push_item(
|
|||
// log / HDR / wide-gamut media is converted on the CPU (below), and so are blended
|
||||
// in-between frames (Frame Blending / Optical Flow on speed-changed clips)
|
||||
if !crate::colorman::needs_management(&seq.settings.color, cs, &frame) && crate::interpolation_blend(item, t, src.info().frame_rate()).is_none() {
|
||||
// Draft playback at reduced resolution: hand over planes box-filtered to the size
|
||||
// drawn instead of the full picture (a quarter at 1/2, a sixteenth at 1/4 of the
|
||||
// upload and sampling). The mean is taken over Y'CbCr codes, not linear light as
|
||||
// the shader's supersampling does, so this stays limited to the opt-in draft mode.
|
||||
let n = if filmcraft_media::cancel::draft() { crate::decimation(frame.width as f32, size.0 as f32 * want) } else { 1 };
|
||||
let frame = match frame.box_decimated(n) {
|
||||
Some(small) => Arc::new(small),
|
||||
None => frame,
|
||||
};
|
||||
let px_scale = frame.width as f64 / size.0.max(1) as f64;
|
||||
let m = Affine::scale(opts.scale as f64, opts.scale as f64).then_apply(&motion).then_apply(&Affine::scale(1.0 / px_scale, 1.0 / px_scale));
|
||||
out.push(PlanLayer { frame, matrix: m, opacity: op * extra_opacity });
|
||||
|
|
|
|||
|
|
@ -299,3 +299,54 @@ fn offline_slate_is_deterministic_and_marked() {
|
|||
::image::save_buffer(p, &a, 640, 360, ::image::ExtendedColorType::Rgba8).unwrap();
|
||||
}
|
||||
}
|
||||
|
||||
/// A 320×180 4:2:0 source with smooth gradients (what decoders hand the renderer).
|
||||
struct YuvSource(GeneratorSource);
|
||||
|
||||
impl MediaSource for YuvSource {
|
||||
fn info(&self) -> &filmcraft_media::MediaInfo {
|
||||
self.0.info()
|
||||
}
|
||||
fn video_frame(&self, _req: FrameRequest) -> filmcraft_media::Result<Arc<filmcraft_frame::VideoFrame>> {
|
||||
let (w, h) = (320usize, 180usize);
|
||||
let y = (0..w * h).map(|i| (16 + (i % w) * 200 / w + (i / w) * 20 / h) as u8).collect();
|
||||
let u = (0..w * h / 4).map(|i| (90 + (i % (w / 2)) * 60 / (w / 2)) as u8).collect();
|
||||
let v = (0..w * h / 4).map(|i| (100 + (i / (w / 2)) * 50 / (h / 2)) as u8).collect();
|
||||
Ok(Arc::new(filmcraft_frame::VideoFrame {
|
||||
width: w as u32,
|
||||
height: h as u32,
|
||||
data: filmcraft_frame::PixelData::Yuv8 { planes: [Arc::new(y), Arc::new(u), Arc::new(v)], chroma: filmcraft_frame::Chroma::C420, alpha: None },
|
||||
color: filmcraft_color::ColorInfo::REC709,
|
||||
par: (1, 1),
|
||||
pts: Tick::ZERO,
|
||||
}))
|
||||
}
|
||||
fn audio(&self, start: i64, frames: usize, sample_rate: u32) -> filmcraft_media::Result<filmcraft_frame::AudioBuffer> {
|
||||
self.0.audio(start, frames, sample_rate)
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn draft_reduced_resolution_plans_carry_decimated_yuv() {
|
||||
let (mut p, red, _ocean, seq, mut map) = setup();
|
||||
let matte = GeneratorSource::new(Generator::ColorMatte { color: [0.0, 0.0, 0.0, 1.0] }, 320, 180, FrameRate::FPS_24, Tick(10 * TICKS_PER_SECOND));
|
||||
map.0.insert(red, Arc::new(YuvSource(matte)) as SharedSource);
|
||||
place(&mut p, seq, 0, red, 0, 48);
|
||||
let t = FrameRate::FPS_24.tick_of(3);
|
||||
for draft in [false, true] {
|
||||
for (scale, small_w) in [(1.0f32, 320u32), (0.5, 160), (0.25, 80)] {
|
||||
let opts = RenderOptions { scale, ..Default::default() };
|
||||
let plan = filmcraft_media::cancel::with_draft(draft, || plan::plan_frame(&p, seq, t, opts, &map));
|
||||
let plan::FramePlan::Layers { layers, .. } = &plan else { panic!("GPU-drawable") };
|
||||
// exact playback hands over the decoded picture; draft playback planes at the drawn size
|
||||
assert_eq!(layers[0].frame.width, if draft { small_w } else { 320 }, "scale {scale}, draft {draft}");
|
||||
assert!(matches!(layers[0].frame.data, filmcraft_frame::PixelData::Yuv8 { .. }));
|
||||
let got = plan::execute_cpu(&plan);
|
||||
let reference = render_sequence(&p, seq, t, opts, &map);
|
||||
assert_eq!((got.w, got.h), (reference.w, reference.h));
|
||||
let max = got.px.iter().zip(&reference.px).map(|(a, b)| (a - b).abs()).fold(0f32, f32::max);
|
||||
// the Y'CbCr-code mean differs from a linear-light mean by a little on gradients
|
||||
assert!(max < if draft { 0.03 } else { 1e-5 }, "scale {scale}, draft {draft}: max diff {max}");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -258,6 +258,7 @@ pub fn section_markdown(v: &Value) -> String {
|
|||
("ui_p95_ms", "UI p95 ms"),
|
||||
("seeks", "seeks"),
|
||||
("skipped", "skipped"),
|
||||
("draft_frames", "draft"),
|
||||
("load", "load"),
|
||||
],
|
||||
1,
|
||||
|
|
|
|||
|
|
@ -229,6 +229,9 @@ pub struct Bench {
|
|||
pub server: FrameServer,
|
||||
pub refresh_hz: f64,
|
||||
pub seconds: f64,
|
||||
/// Play with draft decoding (Settings ▸ Playback ▸ Draft decoding), as the monitor does at
|
||||
/// 1/2 resolution or lower.
|
||||
pub draft: bool,
|
||||
}
|
||||
|
||||
impl Bench {
|
||||
|
|
@ -253,7 +256,7 @@ impl Bench {
|
|||
let last_frame = rate.frame_at(q.duration()) - 1;
|
||||
let t_pre = Instant::now();
|
||||
loop {
|
||||
let key = FrameKey { target, frame: first, size, revision: s.revision };
|
||||
let key = FrameKey { target, frame: first, size, revision: s.revision, draft: self.draft };
|
||||
self.server.schedule_playback(key, rate, scale, &project, 1.0, true);
|
||||
let exact = display.refresh(&self.server, key);
|
||||
meter.refresh(first, exact);
|
||||
|
|
@ -275,7 +278,7 @@ impl Bench {
|
|||
}
|
||||
frame = rate.frame_at(t);
|
||||
due_at.entry(frame).or_insert(now);
|
||||
let key = FrameKey { target, frame, size, revision: s.revision };
|
||||
let key = FrameKey { target, frame, size, revision: s.revision, draft: self.draft };
|
||||
self.server.schedule_playback(key, rate, scale, &project, 1.0, false);
|
||||
let exact = display.refresh(&self.server, key);
|
||||
meter.refresh(frame, exact);
|
||||
|
|
@ -368,7 +371,7 @@ impl Bench {
|
|||
rng ^= rng >> 7;
|
||||
rng ^= rng << 17;
|
||||
let frame = (rng % frames as u64) as i64;
|
||||
let key = FrameKey { target, frame, size, revision: s.revision };
|
||||
let key = FrameKey { target, frame, size, revision: s.revision, draft: false };
|
||||
let start = Instant::now();
|
||||
rep.due += 1;
|
||||
loop {
|
||||
|
|
@ -454,7 +457,7 @@ impl Bench {
|
|||
};
|
||||
for _ in 0..seeks {
|
||||
let frame = (next() % frames as u64) as i64;
|
||||
let key = FrameKey { target, frame, size, revision: s.revision };
|
||||
let key = FrameKey { target, frame, size, revision: s.revision, draft: false };
|
||||
match wait(display, key, Instant::now()) {
|
||||
Some(l) => rep.seek_ms.push(l),
|
||||
None => rep.timeouts += 1,
|
||||
|
|
@ -463,7 +466,7 @@ impl Bench {
|
|||
for _ in 0..drags {
|
||||
let from = (next() % (frames - drag_frames).max(1) as u64) as i64;
|
||||
for f in from..from + drag_frames {
|
||||
let key = FrameKey { target, frame: f.min(frames - 1), size, revision: s.revision };
|
||||
let key = FrameKey { target, frame: f.min(frames - 1), size, revision: s.revision, draft: false };
|
||||
self.server.request(key, rate.tick_of(key.frame), scale, &project, 0);
|
||||
if display.refresh(&self.server, key) {
|
||||
rep.drag_shown += 1;
|
||||
|
|
@ -471,7 +474,7 @@ impl Bench {
|
|||
rep.drag_steps += 1;
|
||||
std::thread::sleep(period);
|
||||
}
|
||||
let key = FrameKey { target, frame: (from + drag_frames).min(frames - 1), size, revision: s.revision };
|
||||
let key = FrameKey { target, frame: (from + drag_frames).min(frames - 1), size, revision: s.revision, draft: false };
|
||||
match wait(display, key, Instant::now()) {
|
||||
Some(l) => rep.settle_ms.push(l),
|
||||
None => rep.timeouts += 1,
|
||||
|
|
@ -720,7 +723,7 @@ pub fn cli_main() {
|
|||
// A fresh frame server per run so caches start cold, as after opening a project.
|
||||
let server = FrameServer::new(s.media.clone(), s.services.clone(), s.previews.clone(), args.workers);
|
||||
server.set_profiling(true);
|
||||
let bench = Bench { server, refresh_hz: args.refresh, seconds: args.seconds };
|
||||
let bench = Bench { server, refresh_hz: args.refresh, seconds: args.seconds, draft: false };
|
||||
let mut display = Display::new(args.gpu);
|
||||
if args.gpu && display.gpu.is_none() {
|
||||
eprintln!("no GPU adapter; using the CPU path");
|
||||
|
|
|
|||
|
|
@ -149,23 +149,27 @@ pub fn seek(o: &Opts) -> Vec<Value> {
|
|||
fn new_bench(s: &Session, seconds: f64) -> Bench {
|
||||
let server = FrameServer::new(s.media.clone(), s.services.clone(), s.previews.clone(), FrameServer::default_workers());
|
||||
server.set_profiling(true);
|
||||
Bench { server, refresh_hz: 60.0, seconds }
|
||||
Bench { server, refresh_hz: 60.0, seconds, draft: false }
|
||||
}
|
||||
|
||||
/// Program-monitor playback through the real scheduler (see `playback.rs`), frames shown vs
|
||||
/// dropped over N seconds.
|
||||
pub fn playback(o: &Opts) -> Vec<Value> {
|
||||
let seconds = if o.quick { 4.0 } else { 8.0 };
|
||||
let cases: &[(&str, &str, f32)] = &[
|
||||
("h264-1080", "full", 1.0),
|
||||
("stack3", "full", 1.0),
|
||||
("h264-2160", "full", 1.0),
|
||||
("h264-2160", "half", 0.5),
|
||||
("hevc-2160", "full", 1.0),
|
||||
("hevc-2160", "half", 0.5),
|
||||
// (scenario, resolution, scale, draft decoding)
|
||||
let cases: &[(&str, &str, f32, bool)] = &[
|
||||
("h264-1080", "full", 1.0, false),
|
||||
("stack3", "full", 1.0, false),
|
||||
("h264-2160", "full", 1.0, false),
|
||||
("h264-2160", "half", 0.5, false),
|
||||
("h264-2160", "half draft", 0.5, true),
|
||||
("h264-2160", "quarter", 0.25, false),
|
||||
("h264-2160", "quarter draft", 0.25, true),
|
||||
("hevc-2160", "full", 1.0, false),
|
||||
("hevc-2160", "half", 0.5, false),
|
||||
];
|
||||
let mut rows = Vec::new();
|
||||
for &(scenario, res, scale) in cases.iter().filter(|c| o.wants(&format!("{} {}", c.0, c.1))) {
|
||||
for &(scenario, res, scale, draft) in cases.iter().filter(|c| o.wants(&format!("{} {}", c.0, c.1))) {
|
||||
let Some((s, seq)) = playback::scenario_session(scenario) else {
|
||||
eprintln!("playback {scenario}: skipped (fixtures unavailable)");
|
||||
continue;
|
||||
|
|
@ -173,7 +177,8 @@ pub fn playback(o: &Opts) -> Vec<Value> {
|
|||
let fps = s.project.sequence(seq).map(|q| q.settings.frame_rate.as_f64()).unwrap_or(24.0);
|
||||
for rep in 0..o.repeat {
|
||||
// A fresh frame server per run: caches start cold, as after opening a project.
|
||||
let bench = new_bench(&s, seconds);
|
||||
let mut bench = new_bench(&s, seconds);
|
||||
bench.draft = draft;
|
||||
let mut display = Display::new(o.gpu);
|
||||
let path = if display.gpu_available() { "gpu" } else { "cpu" };
|
||||
let r = bench.play(&s, seq, &mut display, scale, Tick::ZERO, &format!("{scenario} {res} #{}", rep + 1));
|
||||
|
|
@ -184,6 +189,7 @@ pub fn playback(o: &Opts) -> Vec<Value> {
|
|||
v["ui_p95_ms"] = json!(pct(&r.present, 0.95));
|
||||
v["seeks"] = json!(r.gop.seeks);
|
||||
v["skipped"] = json!(r.gop.skipped);
|
||||
v["draft_frames"] = json!(r.gop.draft);
|
||||
v["load"] = json!(r.load);
|
||||
v["cores_needed"] = json!(r.process_cpu / frames * fps / 1000.0);
|
||||
eprintln!("playback {scenario} {res}: {}/{} shown/dropped (load {})", r.shown, r.dropped, r.load);
|
||||
|
|
|
|||
|
|
@ -40,6 +40,17 @@ pub struct FrameKey {
|
|||
/// Output width in pixels (thumbnails) or scale ×1000 (monitors).
|
||||
pub size: u32,
|
||||
pub revision: u64,
|
||||
/// Draft decoding (reduced-resolution playback with Settings ▸ Playback ▸ Draft decoding):
|
||||
/// sources may decode approximate non-reference pictures
|
||||
/// ([`filmcraft_media::cancel::with_draft`]). Draft frames have their own keys, so a paused
|
||||
/// monitor, an export or a render never shows one.
|
||||
pub draft: bool,
|
||||
}
|
||||
|
||||
/// Whether a monitor frame is requested with draft decoding ([`FrameKey::draft`]): only while
|
||||
/// playing, at 1/2 resolution or lower, with the preference on.
|
||||
pub fn draft_playback(playing: bool, scale: f32, enabled: bool) -> bool {
|
||||
enabled && playing && scale <= 0.5
|
||||
}
|
||||
|
||||
pub struct Rgba {
|
||||
|
|
@ -597,7 +608,7 @@ impl FrameServer {
|
|||
/// The most recent cached frame for `target` at or before `frame` within `max_back` frames.
|
||||
pub fn nearest(&self, target: Target, frame: i64, size: u32, revision: u64, max_back: i64) -> Option<Arc<Rgba>> {
|
||||
for d in 0..=max_back {
|
||||
if let Some(v) = self.get(&FrameKey { target, frame: frame - d, size, revision }) {
|
||||
if let Some(v) = self.get(&FrameKey { target, frame: frame - d, size, revision, draft: false }) {
|
||||
return Some(v);
|
||||
}
|
||||
}
|
||||
|
|
@ -756,23 +767,25 @@ fn run_job(sh: &Shared, job: &Job, pool: &Arc<MediaPool>, services: &Arc<dyn Ser
|
|||
let mut loading = false;
|
||||
let preview = filmcraft_media::cancel::with_cancel(&job.cancel, || {
|
||||
filmcraft_media::cancel::with_catch_up(job.catch_up, || {
|
||||
if let Target::SequencePlan(seq) = job.key.target {
|
||||
let (plan, pv) = plan_job(job, seq, pool, services, previews);
|
||||
loading = filmcraft_media::pending::take();
|
||||
if !job.cancel.load(Ordering::Relaxed) && !loading {
|
||||
// Convert texels for upload here, not on the UI thread when the frame is shown.
|
||||
let prepared = filmcraft_gpu::prepare(&plan);
|
||||
sh.plans.lock().unwrap_or_else(|e| e.into_inner()).insert(job.key, GpuPlan { plan, prepared });
|
||||
filmcraft_media::cancel::with_draft(job.key.draft, || {
|
||||
if let Target::SequencePlan(seq) = job.key.target {
|
||||
let (plan, pv) = plan_job(job, seq, pool, services, previews);
|
||||
loading = filmcraft_media::pending::take();
|
||||
if !job.cancel.load(Ordering::Relaxed) && !loading {
|
||||
// Convert texels for upload here, not on the UI thread when the frame is shown.
|
||||
let prepared = filmcraft_gpu::prepare(&plan);
|
||||
sh.plans.lock().unwrap_or_else(|e| e.into_inner()).insert(job.key, GpuPlan { plan, prepared });
|
||||
}
|
||||
pv
|
||||
} else {
|
||||
let (img, pv) = render_job(job, pool, services, previews);
|
||||
loading = filmcraft_media::pending::take();
|
||||
if !job.cancel.load(Ordering::Relaxed) && !loading {
|
||||
sh.done.lock().unwrap_or_else(|e| e.into_inner()).insert(job.key, Arc::new(img));
|
||||
}
|
||||
pv
|
||||
}
|
||||
pv
|
||||
} else {
|
||||
let (img, pv) = render_job(job, pool, services, previews);
|
||||
loading = filmcraft_media::pending::take();
|
||||
if !job.cancel.load(Ordering::Relaxed) && !loading {
|
||||
sh.done.lock().unwrap_or_else(|e| e.into_inner()).insert(job.key, Arc::new(img));
|
||||
}
|
||||
pv
|
||||
}
|
||||
})
|
||||
})
|
||||
});
|
||||
let cancelled = job.cancel.load(Ordering::Relaxed);
|
||||
|
|
@ -894,6 +907,16 @@ fn render_job(job: &Job, pool: &Arc<MediaPool>, services: &Arc<dyn Services>, pr
|
|||
mod tests {
|
||||
use super::{PlaybackMeter, playback_plan};
|
||||
|
||||
#[test]
|
||||
fn draft_decoding_only_for_reduced_resolution_playback() {
|
||||
use super::draft_playback;
|
||||
assert!(draft_playback(true, 0.5, true) && draft_playback(true, 0.25, true));
|
||||
assert!(!draft_playback(true, 1.0, true), "full resolution is never draft");
|
||||
assert!(!draft_playback(false, 0.5, true), "a paused frame is never draft");
|
||||
assert!(!draft_playback(true, 0.5, false), "off unless enabled");
|
||||
assert!(!filmcraft_engine::autosave::Preferences::default().playback.draft_decode, "off by default");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn playback_plan_skips_frames_only_when_workers_cannot_keep_up() {
|
||||
// decode-only / light frames: every frame from the playhead on
|
||||
|
|
|
|||
|
|
@ -644,7 +644,7 @@ impl FilmcraftApp {
|
|||
let rate = pi.frame_rate();
|
||||
let frame = rate.frame_at(t);
|
||||
let rev = self.item_revision(item);
|
||||
let key = FrameKey { target: Target::Item(item), frame, size: width, revision: rev };
|
||||
let key = FrameKey { target: Target::Item(item), frame, size: width, revision: rev, draft: false };
|
||||
let name = format!("thumb-{}-{}-{}", item.0, frame, width);
|
||||
if let Some(img) = self.frames.get(&key) {
|
||||
let id = self.texture_for(ctx, &name, key, &img);
|
||||
|
|
|
|||
|
|
@ -824,7 +824,7 @@ fn preview_column(app: &mut FilmcraftApp, ui: &mut egui::Ui, rect: Rect, seq_id:
|
|||
ui.painter().rect_filled(pic, 0.0, Color32::BLACK);
|
||||
if app.ui.export.settings.has_video() {
|
||||
let frame = rate.frame_at(app.session.playhead());
|
||||
let key = FrameKey { target: Target::Sequence(seq_id), frame, size: 500, revision: app.session.revision };
|
||||
let key = FrameKey { target: Target::Sequence(seq_id), frame, size: 500, revision: app.session.revision, draft: false };
|
||||
let project = app.session.project.clone();
|
||||
app.frames.request(key, rate.tick_of(frame), 0.5, &project, 0);
|
||||
if let Some(img) = app.frames.get(&key) {
|
||||
|
|
|
|||
|
|
@ -154,7 +154,9 @@ pub fn show(app: &mut FilmcraftApp, ui: &mut egui::Ui, rect: Rect, which: Which)
|
|||
(true, Target::Sequence(s)) => Target::SequencePlan(s),
|
||||
(_, t) => t,
|
||||
};
|
||||
let key = FrameKey { target, frame, size: size_key, revision: rev };
|
||||
// Draft decoding (opt-in): only while playing at 1/2 resolution or lower.
|
||||
let draft = crate::frames::draft_playback(playing, scale, app.session.prefs.playback.draft_decode);
|
||||
let key = FrameKey { target, frame, size: size_key, revision: rev, draft };
|
||||
let project = app.session.project.clone();
|
||||
if playing {
|
||||
let preroll = app.playback.preroll.is_some();
|
||||
|
|
@ -190,7 +192,7 @@ pub fn show(app: &mut FilmcraftApp, ui: &mut egui::Ui, rect: Rect, which: Which)
|
|||
if let Some(ra) = ref_area {
|
||||
// the reference frame (CPU path)
|
||||
let rf = rate.frame_at(Tick(mv.compare_ref.unwrap_or(time.0)));
|
||||
let rkey = FrameKey { target: cpu_target, frame: rf, size: size_key, revision: rev };
|
||||
let rkey = FrameKey { target: cpu_target, frame: rf, size: size_key, revision: rev, draft: false };
|
||||
app.frames.request(rkey, rate.tick_of(rf), scale, &project, 1);
|
||||
let rpic = fit(ra, frame_size.0 as f32, frame_size.1 as f32);
|
||||
ui.painter().rect_filled(rpic, 0.0, t.monitor_bg);
|
||||
|
|
|
|||
|
|
@ -285,7 +285,7 @@ pub fn grid(app: &mut FilmcraftApp, ui: &mut egui::Ui, area: Rect) {
|
|||
let scale = crate::panels::monitor::quantize_scale(app.ui.program.res.scale().min((cell_px / sw).max(1.0 / 32.0)));
|
||||
let mt = Tick(info["sourceTime"].as_i64().unwrap_or(0));
|
||||
let frame = rate.frame_at(mt);
|
||||
let key = FrameKey { target: Target::MulticamGrid(src), frame, size: (scale * 1000.0) as u32, revision: grid_revision(app, src) };
|
||||
let key = FrameKey { target: Target::MulticamGrid(src), frame, size: (scale * 1000.0) as u32, revision: grid_revision(app, src), draft: false };
|
||||
let project = app.session.project.clone();
|
||||
if app.playback.playing {
|
||||
app.frames.schedule_playback(key, rate, scale, &project, app.playback.speed, app.playback.preroll.is_some());
|
||||
|
|
|
|||
|
|
@ -48,7 +48,7 @@ pub fn show(app: &mut FilmcraftApp, ui: &mut egui::Ui, rect: Rect) {
|
|||
ui.painter().rect_filled(pic, 0.0, t.monitor_bg);
|
||||
let ppp = ctx.pixels_per_point();
|
||||
let scale = crate::panels::monitor::quantize_scale((pic.width() * ppp / w).clamp(1.0 / 32.0, 1.0));
|
||||
let key = FrameKey { target: Target::Sequence(seq_id), frame, size: (scale * 1000.0) as u32, revision: app.session.revision };
|
||||
let key = FrameKey { target: Target::Sequence(seq_id), frame, size: (scale * 1000.0) as u32, revision: app.session.revision, draft: false };
|
||||
let project = app.session.project.clone();
|
||||
app.frames.request(key, rate.tick_of(frame), scale, &project, 3);
|
||||
let tex = match app.frames.get(&key) {
|
||||
|
|
|
|||
|
|
@ -68,7 +68,7 @@ pub fn frame_signal(app: &mut FilmcraftApp, ctx: &egui::Context, slot: &str, fra
|
|||
let fresh = if pq {
|
||||
filmcraft_engine::scopes::scope_signal(&app.session, rate.tick_of(frame), 0.125, true)
|
||||
} else {
|
||||
let fkey = FrameKey { target: Target::Sequence(seq_id), frame, size: 250, revision: rev };
|
||||
let fkey = FrameKey { target: Target::Sequence(seq_id), frame, size: 250, revision: rev, draft: false };
|
||||
let project = app.session.project.clone();
|
||||
app.frames.request(fkey, rate.tick_of(frame), 0.25, &project, prio);
|
||||
app.frames.get(&fkey).map(|img| Signal::from_rgba8(img.w, img.h, &img.px))
|
||||
|
|
|
|||
|
|
@ -126,7 +126,7 @@ fn picture(app: &mut FilmcraftApp, ui: &mut egui::Ui, area: Rect, side: &Value,
|
|||
let scale = scale.min(app.ui.program.res.scale());
|
||||
let size_key = (scale * 1000.0) as u32;
|
||||
let rev = app.item_revision(item);
|
||||
let key = FrameKey { target: Target::Item(item), frame, size: size_key, revision: rev };
|
||||
let key = FrameKey { target: Target::Item(item), frame, size: size_key, revision: rev, draft: false };
|
||||
let project = app.session.project.clone();
|
||||
app.frames.request(key, rate.tick_of(frame), scale, &project, 0);
|
||||
let ctx = ui.ctx().clone();
|
||||
|
|
|
|||
|
|
@ -18,6 +18,7 @@ pub fn stats(app: &FilmcraftApp) -> Value {
|
|||
"dropped": dropped,
|
||||
"dropRate": if shown + dropped == 0 { 0.0 } else { dropped as f64 / (shown + dropped) as f64 },
|
||||
"resolution": app.ui.program.res.label(),
|
||||
"draftDecode": app.session.prefs.playback.draft_decode,
|
||||
});
|
||||
let mut frames = app.frames.stats().to_json();
|
||||
frames["workers"] = json!(app.frames.workers());
|
||||
|
|
|
|||
|
|
@ -242,8 +242,14 @@ file ──► codecs (MP4/MOV, MKV, audio) demux + decode, GOP-aware see
|
|||
playhead's distance, two seconds for a scrubbed-to frame), and the GOP cache skips
|
||||
non-reference pictures of late frames on the way (`VideoDecoder::is_disposable`, H.264
|
||||
`nal_ref_idc` 0 and HEVC sub-layer non-reference pictures; the wanted frame decodes exactly as
|
||||
before). `perf.stats` reports the counters. GPU plans carry their texels already
|
||||
converted for upload (`filmcraft_gpu::prepare`), so the UI thread only copies them.
|
||||
before). With Settings ▸ Playback ▸ Draft decoding (off by default), frames played at 1/2
|
||||
resolution or lower are requested as draft frames (`FrameKey::draft`,
|
||||
`filmcraft_media::cancel::with_draft`): the H.264 decoder skips deblocking of non-reference
|
||||
pictures (`VideoDecoder::set_draft`), the GOP cache serves those draft frames to draft
|
||||
requests only (a paused frame, render or export decodes them again exactly), and draft plans
|
||||
hand the GPU box-decimated Y'CbCr planes at the drawn size. `perf.stats` reports the counters.
|
||||
GPU plans carry their texels already converted for upload (`filmcraft_gpu::prepare`), so the UI
|
||||
thread only copies them.
|
||||
- **Audio clock.** The desktop app passes a cpal output (`apps/filmcraft/src/audio.rs`) to the UI as
|
||||
`AudioOut`. While playing, the samples played by the sound card drive the playhead and video follows.
|
||||
Without an audio device, playback falls back to the wall clock. `PlaybackMeter` counts timeline
|
||||
|
|
|
|||
|
|
@ -29,7 +29,7 @@ Methods (handlers in `crates/ui-egui/src/control.rs`):
|
|||
| `ui.playback` | `{action: play|stop|toggle, speed?}` | |
|
||||
| `ui.screenshot` | `{path?, panel?}` | PNG of the window or one panel; fails after 10 s when no frame is presented (window hidden, display asleep) |
|
||||
| `ui.resize`, `ui.focus`, `app.quit` | | `ui.focus` is the only request that activates the app and takes keyboard focus |
|
||||
| `perf.stats` | – | performance counters (also the command `perf.stats`, so `engine.execute` and MCP `command_run` reach it; headless sessions return the engine part). `decode`: GOP-cache requests / hits / `cacheHitRate`, decoder seeks, samples decoded and skipped while catching up, decoder ms (total and per sample); `playback`: playing, shown / dropped frames and `dropRate` of the current or last play, resolution; `frames`: frame-worker jobs, cancelled, `requestHitRate`, `decodeMs` / `renderMs` (mean, p50, p95 of the last 256 jobs), queue, render-cost estimate, cache use; `ui`: fps and frame ms; `process`: CPU seconds; `media`, `jobs`. Counters are cumulative: diff two readings to measure an interval ([performance.md](performance.md)) |
|
||||
| `perf.stats` | – | performance counters (also the command `perf.stats`, so `engine.execute` and MCP `command_run` reach it; headless sessions return the engine part). `decode`: GOP-cache requests / hits / `cacheHitRate`, decoder seeks, samples decoded and skipped while catching up, `draftFrames` (decoded in draft mode), `h264Threads` (frame-threading workers per H.264 decoder), decoder ms (total and per sample); `playback`: playing, shown / dropped frames and `dropRate` of the current or last play, resolution, `draftDecode` (Settings ▸ Playback ▸ Draft decoding); `frames`: frame-worker jobs, cancelled, `requestHitRate`, `decodeMs` / `renderMs` (mean, p50, p95 of the last 256 jobs), queue, render-cost estimate, cache use; `ui`: fps and frame ms; `process`: CPU seconds; `media`, `jobs`. Counters are cumulative: diff two readings to measure an interval ([performance.md](performance.md)) |
|
||||
|
||||
**Focus.** Driving the app never steals the user's keyboard. Started with `--control`, the app opens
|
||||
without activating itself. UI requests and screenshots that need a rendered frame bring the window
|
||||
|
|
|
|||
|
|
@ -9,11 +9,104 @@ and `.md` (options and sections: [testing.md](testing.md) §5). Agents read live
|
|||
**75–200** during every run below. Wall-clock columns (fps, shown/dropped frames, seek latency)
|
||||
swing by 2–5× between back-to-back runs at that load and are only comparable within one run. CPU
|
||||
time per frame, samples decoded and skipped, and peak RSS are much less sensitive, and are what
|
||||
the before/after comparison relies on. Baseline = commit `457aca1` (the harness on unchanged
|
||||
code); after = M4.8. Runs alternated base / after (2 rounds each, 2026-10-02).
|
||||
the before/after comparison relies on. Each milestone below alternated base / after runs (2 rounds
|
||||
each); the most load-independent figure is the decoder's own cycle count (`proc_pid_rusage`
|
||||
instructions and cycles of a process, all threads summed).
|
||||
|
||||
## Results (M4.9: 4K H.264, before → after)
|
||||
|
||||
Baseline = commit `3d14099` (M3.12, unchanged decoder); after = M4.9. Interleaved runs on
|
||||
2026-10-02/03, load average 92–178 (the playback rows say which load each run saw).
|
||||
|
||||
### Decoder work per frame (load-independent)
|
||||
|
||||
`h264dec` example, one decoder, CPU cycles and instructions counted by the kernel (all threads),
|
||||
output bit-exact with ffmpeg:
|
||||
|
||||
| stream | Mcycles / frame before | after | after, draft mode |
|
||||
|---|---|---|---|
|
||||
| 2160p (`a2160.mp4`, 170 Mbit/s, first 48 frames) | 505 | **390** (−23 %) | **356** (−30 %) |
|
||||
| 1080p (`dec_h264_1080.mp4`, 120 frames) | 132 | **100** (−24 %) | **90** (−32 %) |
|
||||
|
||||
The cycle count is the same with 1, 4 or 14 frame threads (no spinning or contention overhead):
|
||||
at ~4.5 GHz a 4K frame is ~85 ms of one core, so 23.976 fps needs about two cores.
|
||||
|
||||
### Decode (every frame through the media stack)
|
||||
|
||||
| codec | size | CPU ms/frame before (2 rounds) | after (2 rounds) |
|
||||
|---|---|---|---|
|
||||
| H.264 | 1080p | 35.8, 39.5 | **33.0, 29.4** |
|
||||
| H.264 | 2160p | 138.9, 152.1 | **125.7, 112.3** |
|
||||
|
||||
(A run at load ~190 before any change gave 41.6 / 160.2 ms; an intermediate build 32.2 / 118.3.)
|
||||
|
||||
### Cold seeks on the 4K fixture (12 random targets, GOP 250, fresh source per seek)
|
||||
|
||||
| mode | CPU ms per seek before | after | wall p50 ms before → after |
|
||||
|---|---|---|---|
|
||||
| full decode from the keyframe | 7529, 7418 | **5531, 5268** (−27 %) | 2757, 1848 → 924, 739 |
|
||||
| keep 2 s (scrub) | 7488, 7373 | **5442, 5397** | 2408, 1995 → 872, 795 |
|
||||
| late (playback catch-up) | 5597, 5487 | **4079, 3949** | 2014, 1248 → 735, 526 |
|
||||
|
||||
Samples decoded per seek are identical (~48, ~34 in catch-up): the same pictures, decoded with
|
||||
less work. The cold seek is a chain of reference pictures, so it shows the per-picture cost
|
||||
directly.
|
||||
|
||||
### Program-monitor playback, 2160p (8 s, GPU path)
|
||||
|
||||
| case | before: shown/dropped (load) | after: shown/dropped (load) | CPU ms/frame before → after |
|
||||
|---|---|---|---|
|
||||
| Full | 192/0 (92), 9/183 (167) | 113/79 (171), **192/0** (139) | 159 (no skips), 133 (17 skipped) → 131, 130 |
|
||||
| 1/2 | 12/180 (95), 5/187 (169) | 173/19 (172), **192/0** (124) | 143, 127 → 133, 132 |
|
||||
| 1/2 draft | – | 183/9 (175), **192/0** (119) | – → 144, 134 |
|
||||
| 1/4 | – | 186/6 (177), **192/0** (111) | – → 134, 125 |
|
||||
| 1/4 draft | – | **192/0** (178), 190/2 (101) | – → 132, 131 |
|
||||
| 1080p Full | 192/0, 192/0 | 192/0, 192/0 | 41, 48 → 39, 36 |
|
||||
|
||||
Before, the 4K clip kept up in one run (Full at load 92; 1/2 dropped 94 % at load 95) and dropped
|
||||
95–97 % at load ~168. After, every 4K case plays 192/0 at load 100–140; at load 171–178 Full drops
|
||||
41 %, 1/2 10 % (5 % with draft decoding), 1/4 3 % (none with draft decoding). CPU per
|
||||
displayed frame (whole process: decode, frame workers, compositing) drops ~15 % where no frames
|
||||
were skipped; draft decoding at 1/2 and 1/4 is within run-to-run noise in this column (its saving
|
||||
is the ~9 % of decoder cycles above, plus a quarter / sixteenth of the texture upload), while the
|
||||
frame jobs wait for the decoder less (decode ms/job 212 → 67 at 1/2, 98 → 44 at 1/4 in the busy
|
||||
round). On an idle machine the 4K clip needs about two cores.
|
||||
|
||||
## What M4.9 changed
|
||||
|
||||
1. **Deblocking** (25–30 % of 4K decode time before): the edge filter runs on all 16 luma / 8
|
||||
chroma lines of an edge at once with per-lane masks and min/max clipping (no `clamp`, whose
|
||||
bounds assert blocked vectorisation), so it compiles to NEON; edges whose alpha or beta index
|
||||
is 0 are skipped. Bit-exact with the per-line filter (random-edge unit test) and ffmpeg.
|
||||
2. **CABAC**: `decode_decision` / `decode_bypass` select instead of branching on the bin value
|
||||
(the MPS/LPS outcome is close to random at high bit rates), renormalisation has no branch on
|
||||
the range, and context / state indexing has no bounds checks.
|
||||
3. **Inverse transforms**: the 4x4 / 8x8 column pass and the add/clip run across all columns.
|
||||
4. **Frame threading** was already in place (M2): pictures decode concurrently and wait per
|
||||
macroblock row for the reference rows their motion vectors reach, with deblocking pipelined row
|
||||
by row inside each picture's job. The conformance suite now decodes every fixture with 1, 3 and
|
||||
all threads (bit-exact each time). A separate deblocking thread was not added: a picture's rows
|
||||
are published about two rows after they are reconstructed, so a reference chain advances a few
|
||||
rows behind its predecessor and the pool already has more pictures in flight than cores
|
||||
(crates/h264/README.md, "Threading model").
|
||||
5. **Draft decoding** (Settings ▸ Playback ▸ Draft decoding, off by default; `perf.stats`
|
||||
`playback.draftDecode`, `decode.draftFrames`, `decode.h264Threads`): while the Program monitor
|
||||
plays at 1/2 or 1/4, H.264 non-reference pictures skip deblocking (no other picture can change),
|
||||
and draft plans hand the GPU box-decimated Y'CbCr planes at the drawn size. Draft frames carry
|
||||
their own frame-cache keys and the GOP cache serves them to draft requests only, so pausing,
|
||||
rendering and exporting always decode exact pictures (tested through the media stack against
|
||||
ffmpeg after draft playback). The CPU renderer already converted reduced-resolution frames
|
||||
straight to the decimated size (`to_linear_f32_decimated`).
|
||||
|
||||
Not done: CABAC residual decoding (now 41 % of the work, ~19 cycles per bin) is inherently serial;
|
||||
the remaining deblocking cost is mostly the vertical-edge transposes; motion compensation and the
|
||||
per-row / per-picture copies (4 %) are unchanged.
|
||||
|
||||
## Results (M4.8, before → after)
|
||||
|
||||
Baseline = commit `457aca1` (the harness on unchanged code); after = M4.8. Runs alternated base /
|
||||
after (2 rounds each, 2026-10-02).
|
||||
|
||||
### Decode (every frame through the media stack: container, GOP cache, decoder, conversion)
|
||||
|
||||
CPU ms per frame (both rounds; lower is better). fps is the best of the run at that moment's load.
|
||||
|
|
|
|||
|
|
@ -237,7 +237,10 @@ wgpu render ms per frame), `export` (H.264 + AAC and ProRes) and `project` (save
|
|||
Options: `--sections decode,scrub`, `--only <substring>` (fixture or scenario), `--repeat N`,
|
||||
`--quick`, `--cpu`, `--label NAME` (output `target/bench/bench-<label>.{json,md}`),
|
||||
`--section NAME --json FILE` (one section in-process, e.g. under a profiler). Fixtures are made
|
||||
with ffmpeg in `target/fixtures/playback/` (or `$FILMCRAFT_FIXTURES/playback`).
|
||||
with ffmpeg in `target/fixtures/playback/` (or `$FILMCRAFT_FIXTURES/playback`). The suite's
|
||||
`playback` section plays `h264-1080` and `stack3` at Full, `h264-2160` at Full, 1/2 and 1/4 and with
|
||||
draft decoding (Settings ▸ Playback ▸ Draft decoding) at 1/2 and 1/4, and `hevc-2160` at Full and
|
||||
1/2; its `draft` column counts the frames decoded in draft mode.
|
||||
|
||||
The perf tests check bit-exactness before they time anything. Results go in the crate README's
|
||||
performance table, with machine and thread count. Headline numbers go in
|
||||
|
|
|
|||
Loading…
Reference in New Issue