HW1: platform crate + VideoToolbox H.264/HEVC hardware decode (#30)
- New crates/platform (filmcraft-platform, L5): OS media FFI only. The one crate allowed `unsafe` (own [lints] with unsafe_code = "deny", #[allow(unsafe_code)] only on the videotoolbox module, SAFETY comment on every block, safe Result-returning public API). AGENTS.md §0.3 names the exception; docs/adr/0001-platform-ffi.md records why and the rules. - VideoToolbox decoder for avcC / hvcC, 8/10-bit 4:2:0 and 4:2:2: session from the sample entry's parameter sets with hardware required, asynchronous decode (2 in flight), biplanar CVPixelBuffer copied to planar Yuv8/Yuv16, reorder buffer of the stream's own depth, RASL skipped after a starting CRA. Output callback under catch_unwind. - HybridDecoder: transparent mid-stream fallback to the software decoder (replays samples since the last restart point, drops already-returned pictures). New codecs::software_video_decoder (built-in factories only) and codecs::hw (switch, counters, NalStreamInfo shared with software decoders; vui_color / sar_par helpers now used by both). - Settings ▸ Playback ▸ Hardware decoding: Auto / Off (engine applies it to the codecs switch); desktop app, CLI/MCP and bench call register(). - perf.stats decode.framesDecoded and decode.hardware counters. - Tests: bit-exact parity with the software decoders (H.264 High, HEVC Main open GOP, Main 10; resets, reseeks, flushes), forced-failure fallback, mutation fuzz of samples and parameter sets, cross-platform fallback with a stand-in decoder, setting Off, HEVC 4:2:2 vs ffmpeg. - Bench --hw auto|off; results in docs/performance.md (4K CPU/frame ~86-119 ms -> ~4 ms). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Brandon Thomas
parent
6102a3794a
commit
c3ec0e85d3
|
|
@ -19,7 +19,12 @@ panic path, and fix a crash before building on top of it. The cross-app standard
|
|||
fallback that is truly correct (never one that silently corrupts a project). An unfinished feature
|
||||
returns an "unsupported" error. The only exception is a call that is provably infallible from local
|
||||
code alone (a literal that always parses): a single-item `#[allow(clippy::expect_used)]` with the reason.
|
||||
3. **No `unsafe`** (`unsafe_code = "forbid"` for the workspace).
|
||||
3. **No `unsafe`** (`unsafe_code = "forbid"` for the workspace). The single exception is
|
||||
`crates/platform`, which needs `unsafe` to call the operating system's media APIs (hardware
|
||||
video decoding through VideoToolbox; later encoding and other OSes) and does nothing else. It
|
||||
uses `unsafe_code = "deny"` with `#[allow(unsafe_code)]` only on its FFI modules, a
|
||||
`// SAFETY:` comment on every `unsafe` block, a safe `Result`-returning public API, and the
|
||||
pure-Rust decoder as the tested fallback ([ADR 0001](docs/adr/0001-platform-ffi.md)).
|
||||
4. **Every input-derived number is hostile.** Media files, project and interchange files, presets, fonts,
|
||||
CLI / MCP / control-channel parameters and UI state are untrusted. Index and slice with `get()` (or
|
||||
validate bounds once, up front, for a hot loop); slice strings only at char boundaries; use
|
||||
|
|
|
|||
|
|
@ -12,7 +12,7 @@ FilmCraft is a clean-room, open-source, pure-Rust non-linear video editor target
|
|||
**Read [`AGENTS.md`](AGENTS.md) first; its rules override everything here.**
|
||||
- **Never crash** ([`AGENTS.md`](AGENTS.md) §0, standard: [`craftrules/standards/never-crash.md`](https://github.com/storytold/craftrules/blob/main/standards/never-crash.md)). People depend on FilmCraft; a crash loses their work, so this outranks feature work. No `unwrap`/`expect`/`panic!`/`unreachable!`/`todo!`/`unimplemented!` or `unsafe` outside tests: return `Result<T, E>` and propagate with `?` (`ok_or`, `let … else`, `if let`). Treat every number from media, project files, presets or CLI/MCP/control params as hostile: `get()` instead of `[i]`, checked/saturating math, no division by possibly-zero values, capped allocations, bounded recursion. `lock().unwrap_or_else(PoisonError::into_inner)`; background jobs run under `catch_unwind`. The panic hook + UI `catch_unwind` (`crates/ui-egui/src/crash.rs`) are the last-resort guard, not a licence. Every crash fix gets a regression test that panicked before. In particular (§1): never use Adobe iconography, images or any other Adobe asset. Every asset must be openly licensed (OSS / public domain / Creative Commons, or original work by a contributor) and must have a `<file>.attribution` sidecar plus an entry in `ATTRIBUTION.md`. `cargo xtask assets` enforces this.
|
||||
- **Clean-room.** Never read/disassemble anything inside Adobe app bundles (names/listings only). Never copy Adobe icons, presets, LUTs, fonts. Never copy GPL/LGPL/AGPL code (FFmpeg, x264, x265, MLT, Kdenlive, Shotcut, Olive, LAME…). ffmpeg/ffprobe run only as external test oracles / fixture generators — never linked or shipped.
|
||||
- **Pure Rust** in the product. OS media APIs only via Rust bindings in `crates/platform` (planned; not created yet), optional, behind traits.
|
||||
- **Pure Rust** in the product. OS media APIs only via Rust bindings in `crates/platform` (hardware decoding; the one crate allowed `unsafe`, see `AGENTS.md` §0.3 and `docs/adr/0001-platform-ffi.md`), optional, behind traits, with our decoders as the fallback.
|
||||
- **Layering** ([`docs/architecture.md`](docs/architecture.md) §1, enforced by `cargo xtask layers`): nothing below L5 depends on egui/eframe/winit/rfd/cpal. L0 codec/container crates depend only on `filmcraft-bitstream`.
|
||||
- **Exact time:** all time is `filmcraft_time::Tick` (254 016 000 000/s). Never use f64 seconds for edit math.
|
||||
- **Everything is a command** (`crates/engine`): id, label, menu path, shortcut, params, enabled(), run(). UI, CLI, control channel and MCP all dispatch by id.
|
||||
|
|
|
|||
|
|
@ -1662,6 +1662,7 @@ dependencies = [
|
|||
"eframe",
|
||||
"egui",
|
||||
"filmcraft-engine",
|
||||
"filmcraft-platform",
|
||||
"filmcraft-ui-egui",
|
||||
"log",
|
||||
"muda",
|
||||
|
|
@ -1761,6 +1762,7 @@ dependencies = [
|
|||
"filmcraft-mpegts",
|
||||
"filmcraft-mxf",
|
||||
"filmcraft-ogg",
|
||||
"filmcraft-platform",
|
||||
"filmcraft-time",
|
||||
"serde_json",
|
||||
"tokio",
|
||||
|
|
@ -2069,6 +2071,24 @@ dependencies = [
|
|||
"filmcraft-testkit",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "filmcraft-platform"
|
||||
version = "0.2.0"
|
||||
dependencies = [
|
||||
"filmcraft-codecs",
|
||||
"filmcraft-color",
|
||||
"filmcraft-frame",
|
||||
"filmcraft-isobmff",
|
||||
"filmcraft-media",
|
||||
"filmcraft-testkit",
|
||||
"filmcraft-time",
|
||||
"log",
|
||||
"objc2-core-foundation",
|
||||
"objc2-core-media",
|
||||
"objc2-core-video",
|
||||
"objc2-video-toolbox",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "filmcraft-project"
|
||||
version = "0.2.0"
|
||||
|
|
@ -2183,6 +2203,7 @@ dependencies = [
|
|||
"filmcraft-gpu",
|
||||
"filmcraft-interchange",
|
||||
"filmcraft-media",
|
||||
"filmcraft-platform",
|
||||
"filmcraft-project",
|
||||
"filmcraft-render",
|
||||
"filmcraft-scopes",
|
||||
|
|
@ -3779,7 +3800,9 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
|
|||
checksum = "2a180dd8642fa45cdb7dd721cd4c11b1cadd4929ce112ebd8b9f5803cc79d536"
|
||||
dependencies = [
|
||||
"bitflags 2.13.2",
|
||||
"block2 0.6.2",
|
||||
"dispatch2",
|
||||
"libc",
|
||||
"objc2 0.6.4",
|
||||
]
|
||||
|
||||
|
|
@ -3820,6 +3843,37 @@ dependencies = [
|
|||
"objc2-foundation 0.2.2",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "objc2-core-media"
|
||||
version = "0.3.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "05ec576860167a15dd9fce7fbee7512beb4e31f532159d3482d1f9c6caedf31d"
|
||||
dependencies = [
|
||||
"bitflags 2.13.2",
|
||||
"block2 0.6.2",
|
||||
"dispatch2",
|
||||
"objc2 0.6.4",
|
||||
"objc2-core-audio",
|
||||
"objc2-core-audio-types",
|
||||
"objc2-core-foundation",
|
||||
"objc2-core-video",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "objc2-core-video"
|
||||
version = "0.3.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "d425caf1df73233f29fd8a5c3e5edbc30d2d4307870f802d18f00d83dc5141a6"
|
||||
dependencies = [
|
||||
"bitflags 2.13.2",
|
||||
"block2 0.6.2",
|
||||
"objc2 0.6.4",
|
||||
"objc2-core-foundation",
|
||||
"objc2-core-graphics",
|
||||
"objc2-io-surface",
|
||||
"objc2-metal 0.3.2",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "objc2-encode"
|
||||
version = "4.1.0"
|
||||
|
|
@ -3992,6 +4046,23 @@ dependencies = [
|
|||
"objc2-foundation 0.2.2",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "objc2-video-toolbox"
|
||||
version = "0.3.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "05bf9a3c14831a7d9641b0d81d87dd913ee238a012b2fde27db5a84b56f5df3e"
|
||||
dependencies = [
|
||||
"bitflags 2.13.2",
|
||||
"block2 0.6.2",
|
||||
"objc2 0.6.4",
|
||||
"objc2-core-foundation",
|
||||
"objc2-core-graphics",
|
||||
"objc2-core-media",
|
||||
"objc2-core-video",
|
||||
"objc2-foundation 0.3.2",
|
||||
"objc2-metal 0.3.2",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "once_cell"
|
||||
version = "1.21.4"
|
||||
|
|
|
|||
|
|
@ -51,6 +51,7 @@ filmcraft-format = { path = "crates/format" }
|
|||
filmcraft-engine = { path = "crates/engine" }
|
||||
filmcraft-ui-egui = { path = "crates/ui-egui" }
|
||||
filmcraft-automation = { path = "crates/automation" }
|
||||
filmcraft-platform = { path = "crates/platform" }
|
||||
filmcraft-testkit = { path = "crates/testkit" }
|
||||
|
||||
# foundation
|
||||
|
|
|
|||
|
|
@ -21,6 +21,7 @@ whisper = ["filmcraft-engine/whisper", "filmcraft-engine/speech-download"]
|
|||
filmcraft-engine = { workspace = true }
|
||||
filmcraft-media = { workspace = true }
|
||||
filmcraft-codecs = { workspace = true }
|
||||
filmcraft-platform = { workspace = true }
|
||||
filmcraft-mxf = { workspace = true }
|
||||
filmcraft-mpegts = { workspace = true }
|
||||
filmcraft-ogg = { workspace = true }
|
||||
|
|
|
|||
|
|
@ -155,6 +155,8 @@ async fn main() {
|
|||
println!("filmcraft-cli {}", env!("CARGO_PKG_VERSION"));
|
||||
return;
|
||||
}
|
||||
// OS hardware video decoders (VideoToolbox on macOS) in front of our own, as in the desktop app.
|
||||
let _ = filmcraft_platform::register();
|
||||
let a = Args::parse(std::env::args().skip(1));
|
||||
let Some(cmd) = a.pos(0) else { usage("missing subcommand") };
|
||||
match cmd {
|
||||
|
|
|
|||
|
|
@ -18,6 +18,7 @@ whisper = ["filmcraft-engine/whisper", "filmcraft-engine/speech-download"]
|
|||
[dependencies]
|
||||
filmcraft-engine = { workspace = true }
|
||||
filmcraft-ui-egui = { workspace = true }
|
||||
filmcraft-platform = { workspace = true }
|
||||
egui = { workspace = true }
|
||||
eframe = { workspace = true, features = ["default_fonts", "wgpu", "accesskit", "wayland", "x11"] }
|
||||
serde_json = { workspace = true }
|
||||
|
|
|
|||
|
|
@ -73,6 +73,9 @@ fn main() -> eframe::Result {
|
|||
// Panics anywhere go to <data dir>/Logs/crash-<day>.log with a backtrace; the UI pass and
|
||||
// frame workers catch them and keep running (see filmcraft_ui_egui::crash).
|
||||
filmcraft_ui_egui::crash::install(data_dir.clone().or_else(default_data_dir).map(|d| d.join("Logs")));
|
||||
// OS hardware video decoders (VideoToolbox on macOS) in front of our own; Settings ▸ Playback ▸
|
||||
// Hardware decoding switches them off. Unsupported streams and failures use our decoders.
|
||||
log::info!("hardware decoding: {:?}", filmcraft_platform::register());
|
||||
let options = eframe::NativeOptions {
|
||||
viewport: egui::ViewportBuilder::default()
|
||||
.with_title("FilmCraft")
|
||||
|
|
|
|||
|
|
@ -33,6 +33,9 @@ pub struct GopStats {
|
|||
pub skipped: u64,
|
||||
/// Frames decoded in draft mode (reduced-resolution playback, [`VideoDecoder::set_draft`]).
|
||||
pub draft: u64,
|
||||
/// Pictures decoders output (hardware and software; [`crate::hw::hw_stats`] counts the
|
||||
/// hardware ones).
|
||||
pub frames: u64,
|
||||
}
|
||||
|
||||
impl GopStats {
|
||||
|
|
@ -56,6 +59,12 @@ static EVICTED: AtomicU64 = AtomicU64::new(0);
|
|||
static DECODE_NS: AtomicU64 = AtomicU64::new(0);
|
||||
static SKIPPED: AtomicU64 = AtomicU64::new(0);
|
||||
static DRAFT: AtomicU64 = AtomicU64::new(0);
|
||||
static FRAMES: AtomicU64 = AtomicU64::new(0);
|
||||
|
||||
/// Count pictures a decoder call output.
|
||||
fn count_frames<T>(out: &[T]) {
|
||||
FRAMES.fetch_add(out.len() as u64, Ordering::Relaxed);
|
||||
}
|
||||
|
||||
/// Run a decoder call, adding its wall time to the process-wide counter.
|
||||
fn timed<R>(f: impl FnOnce() -> R) -> R {
|
||||
|
|
@ -76,6 +85,7 @@ pub fn gop_stats() -> GopStats {
|
|||
decode_ns: DECODE_NS.load(Ordering::Relaxed),
|
||||
skipped: SKIPPED.load(Ordering::Relaxed),
|
||||
draft: DRAFT.load(Ordering::Relaxed),
|
||||
frames: FRAMES.load(Ordering::Relaxed),
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -91,6 +101,7 @@ impl std::ops::Sub for GopStats {
|
|||
decode_ns: self.decode_ns - o.decode_ns,
|
||||
skipped: self.skipped - o.skipped,
|
||||
draft: self.draft - o.draft,
|
||||
frames: self.frames - o.frames,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -252,6 +263,7 @@ impl GopCache {
|
|||
|
||||
/// Store decoder output (in presentation order) and advance `out_max`.
|
||||
fn store_output(&self, st: &mut State, out: Vec<crate::video::DecodedFrame>) {
|
||||
count_frames(&out);
|
||||
for d in out {
|
||||
st.out_max = st.out_max.max(d.pts);
|
||||
self.store(st, d.pts, d.frame, d.draft);
|
||||
|
|
@ -419,6 +431,7 @@ impl GopCache {
|
|||
let k = *next;
|
||||
let data = s.read(k)?;
|
||||
let pics = timed(|| d.decode(&data, s.pts(k)))?;
|
||||
count_frames(&pics);
|
||||
*next += 1;
|
||||
found |= pics.iter().any(|p| p.pts == want_pts);
|
||||
out.extend(pics);
|
||||
|
|
@ -427,7 +440,9 @@ impl GopCache {
|
|||
}
|
||||
}
|
||||
if !found {
|
||||
out.extend(timed(|| d.flush()));
|
||||
let pics = timed(|| d.flush());
|
||||
count_frames(&pics);
|
||||
out.extend(pics);
|
||||
*next = usize::MAX;
|
||||
}
|
||||
Ok(out)
|
||||
|
|
@ -451,7 +466,9 @@ impl GopCache {
|
|||
if st.spare.len() < 16 {
|
||||
st.spare.push(dec);
|
||||
}
|
||||
for d in res? {
|
||||
let res = res?;
|
||||
count_frames(&res);
|
||||
for d in res {
|
||||
self.store(&mut st, d.pts, d.frame, d.draft);
|
||||
}
|
||||
st.at_or_before(want_pts, true).ok_or_else(|| CodecError::Decode("frame not produced".into()))
|
||||
|
|
|
|||
|
|
@ -0,0 +1,350 @@
|
|||
//! Support for hardware (OS) video decoders registered by `filmcraft-platform`: the
|
||||
//! Settings ▸ Playback ▸ Hardware decoding switch, the hardware counters `perf.stats` reports, and
|
||||
//! what such a decoder needs from an `avcC` / `hvcC` sample entry to behave exactly like our
|
||||
//! software decoder for the same stream (parameter sets, cropping, colour, pixel aspect, reorder
|
||||
//! depth, random-access and disposable samples).
|
||||
//!
|
||||
//! The software decoders stay the reference: a hardware factory declines streams it cannot
|
||||
//! decode, and a hardware decoder that fails mid-stream switches to
|
||||
//! [`crate::software_video_decoder`], so registering one never makes a file undecodable.
|
||||
|
||||
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
|
||||
|
||||
use filmcraft_isobmff::{CodecConfig, SampleEntry};
|
||||
|
||||
use crate::video::{avcc_length_size, h264_disposable, hevc_disposable, hvcc_length_size_and_tid, sar_par, vui_color};
|
||||
use crate::{CodecError, Result};
|
||||
|
||||
static ENABLED: AtomicBool = AtomicBool::new(true);
|
||||
|
||||
/// Settings ▸ Playback ▸ Hardware decoding: `Auto` (true, the default) lets registered hardware
|
||||
/// decoders take streams they support; `Off` (false) makes every new decoder a software one.
|
||||
/// Decoders already created keep running: a source picks the change up when it is reopened.
|
||||
pub fn set_hardware_decoding(on: bool) {
|
||||
ENABLED.store(on, Ordering::Relaxed);
|
||||
}
|
||||
|
||||
/// Whether hardware decoders may be used for new decoders ([`set_hardware_decoding`]).
|
||||
pub fn hardware_decoding() -> bool {
|
||||
ENABLED.load(Ordering::Relaxed)
|
||||
}
|
||||
|
||||
/// Process-wide hardware decoding counters (they only grow; subtract two snapshots to measure an
|
||||
/// interval). Frames decoded in software are `GopStats::frames - frames`.
|
||||
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
|
||||
pub struct HwStats {
|
||||
/// Pictures output by hardware decoders.
|
||||
pub frames: u64,
|
||||
/// Hardware decoders created (one per decoder instance; a seek reuses it).
|
||||
pub sessions: u64,
|
||||
/// Streams a hardware factory handed to the software decoder (unsupported profile, size,
|
||||
/// chroma format or bit depth, no hardware, or the switch is Off).
|
||||
pub declined: u64,
|
||||
/// Hardware decoders that failed mid-stream and continued in software.
|
||||
pub fallbacks: u64,
|
||||
}
|
||||
|
||||
static FRAMES: AtomicU64 = AtomicU64::new(0);
|
||||
static SESSIONS: AtomicU64 = AtomicU64::new(0);
|
||||
static DECLINED: AtomicU64 = AtomicU64::new(0);
|
||||
static FALLBACKS: AtomicU64 = AtomicU64::new(0);
|
||||
|
||||
/// The counters so far.
|
||||
pub fn hw_stats() -> HwStats {
|
||||
HwStats {
|
||||
frames: FRAMES.load(Ordering::Relaxed),
|
||||
sessions: SESSIONS.load(Ordering::Relaxed),
|
||||
declined: DECLINED.load(Ordering::Relaxed),
|
||||
fallbacks: FALLBACKS.load(Ordering::Relaxed),
|
||||
}
|
||||
}
|
||||
|
||||
/// A hardware decoder output `n` pictures.
|
||||
pub fn note_hw_frames(n: usize) {
|
||||
FRAMES.fetch_add(n as u64, Ordering::Relaxed);
|
||||
}
|
||||
|
||||
/// A hardware decoder was created.
|
||||
pub fn note_hw_session() {
|
||||
SESSIONS.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
|
||||
/// A hardware factory declined a stream it is responsible for.
|
||||
pub fn note_hw_declined() {
|
||||
DECLINED.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
|
||||
/// A hardware decoder switched to software mid-stream.
|
||||
pub fn note_hw_fallback() {
|
||||
FALLBACKS.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
|
||||
/// The two codecs whose `avcC` / `hvcC` streams hardware decoders take.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub enum NalCodec {
|
||||
H264,
|
||||
Hevc,
|
||||
}
|
||||
|
||||
/// What a decoder needs to know about a length-prefixed H.264 / HEVC stream, from its sample
|
||||
/// entry's configuration record and active sequence parameter set (parsed with the software
|
||||
/// decoders' own parsers, so both decoders derive the same values).
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct NalStreamInfo {
|
||||
pub codec: NalCodec,
|
||||
/// NAL unit length-prefix size of the samples (1, 2 or 4).
|
||||
pub length_size: usize,
|
||||
/// HEVC: highest TemporalId (numTemporalLayers - 1), when signalled.
|
||||
pub highest_tid: Option<u8>,
|
||||
/// The parameter-set NAL units of the record (header included, emulation prevention kept),
|
||||
/// in record order: H.264 SPS then PPS; HEVC VPS, SPS, PPS. SEI arrays are left out.
|
||||
pub parameter_sets: Vec<Vec<u8>>,
|
||||
/// Coded (decoded) picture size in luma samples.
|
||||
pub coded: (u32, u32),
|
||||
/// Output (cropped) rectangle in luma samples: x, y, width, height.
|
||||
pub crop: (u32, u32, u32, u32),
|
||||
/// 0 monochrome, 1 4:2:0, 2 4:2:2, 3 4:4:4.
|
||||
pub chroma_format_idc: u32,
|
||||
pub bit_depth_luma: u32,
|
||||
pub bit_depth_chroma: u32,
|
||||
/// Field (PAFF / MBAFF) coding (H.264 `frame_mbs_only_flag` 0).
|
||||
pub interlaced: bool,
|
||||
/// H.264 `profile_idc` / HEVC `general_profile_idc`.
|
||||
pub profile_idc: u8,
|
||||
/// Colour of every output picture, as the software decoder reports it.
|
||||
pub color: filmcraft_color::ColorInfo,
|
||||
/// Pixel aspect ratio, as the software decoder reports it.
|
||||
pub par: (u32, u32),
|
||||
/// Pictures that may precede a picture in decoding order and follow it in output order
|
||||
/// (H.264 `max_num_reorder_frames`, HEVC `sps_max_num_reorder_pics`).
|
||||
pub reorder: usize,
|
||||
}
|
||||
|
||||
/// Length-prefixed parameter-set entries (`u16` length + bytes) starting at `pos`.
|
||||
fn read_sets(rec: &[u8], pos: &mut usize, count: usize, out: &mut Vec<Vec<u8>>) -> Result<()> {
|
||||
for _ in 0..count {
|
||||
let len = rec
|
||||
.get(*pos..*pos + 2)
|
||||
.map(|b| u16::from_be_bytes([b[0], b[1]]) as usize)
|
||||
.ok_or_else(|| CodecError::Decode("configuration record truncated".into()))?;
|
||||
*pos += 2;
|
||||
let nal = rec.get(*pos..*pos + len).ok_or_else(|| CodecError::Decode("configuration record truncated".into()))?;
|
||||
*pos += len;
|
||||
if !nal.is_empty() {
|
||||
out.push(nal.to_vec());
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
impl NalStreamInfo {
|
||||
/// The stream info of an `avcC` / `hvcC` sample entry (`None` for other codecs).
|
||||
pub fn from_entry(e: &SampleEntry) -> Option<Result<Self>> {
|
||||
match &e.codec {
|
||||
CodecConfig::Avc(a) => Some(Self::from_avcc(&a.to_bytes())),
|
||||
CodecConfig::Hevc(c) => Some(Self::from_hvcc(&c.to_bytes())),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
/// From an `avcC` (AVCDecoderConfigurationRecord) payload.
|
||||
pub fn from_avcc(avcc: &[u8]) -> Result<Self> {
|
||||
if avcc.len() < 7 || avcc[0] != 1 {
|
||||
return Err(CodecError::Decode("bad avcC record".into()));
|
||||
}
|
||||
let mut sets = Vec::new();
|
||||
let mut pos = 6;
|
||||
read_sets(avcc, &mut pos, (avcc[5] & 0x1f) as usize, &mut sets)?;
|
||||
let nsps = sets.len();
|
||||
let npps = *avcc.get(pos).ok_or_else(|| CodecError::Decode("avcC truncated".into()))? as usize;
|
||||
pos += 1;
|
||||
read_sets(avcc, &mut pos, npps, &mut sets)?;
|
||||
let sps_nal = sets.iter().take(nsps).find(|n| n.first().is_some_and(|h| h & 0x1f == 7)).ok_or_else(|| CodecError::Decode("avcC has no SPS".into()))?;
|
||||
if sets.len() == nsps {
|
||||
return Err(CodecError::Decode("avcC has no PPS".into()));
|
||||
}
|
||||
let sps = filmcraft_h264::params::Sps::parse(&filmcraft_bitstream::unescape_rbsp(sps_nal.get(1..).unwrap_or_default()))
|
||||
.map_err(|e| CodecError::Decode(e.to_string()))?;
|
||||
let crop = sps.crop_rect();
|
||||
let vui = sps.vui.clone().unwrap_or_default();
|
||||
Ok(Self {
|
||||
codec: NalCodec::H264,
|
||||
length_size: avcc_length_size(avcc),
|
||||
highest_tid: None,
|
||||
parameter_sets: sets,
|
||||
coded: (sps.width(), sps.height()),
|
||||
crop,
|
||||
chroma_format_idc: sps.chroma_format_idc,
|
||||
bit_depth_luma: sps.bit_depth_luma,
|
||||
bit_depth_chroma: sps.bit_depth_chroma,
|
||||
interlaced: !sps.frame_mbs_only,
|
||||
profile_idc: sps.profile_idc,
|
||||
color: vui_color(crop.2, crop.3, vui.matrix_coefficients, vui.transfer_characteristics, vui.full_range),
|
||||
par: sar_par(vui.sar),
|
||||
// as the software decoder's DPB (`max_reorder`)
|
||||
reorder: sps.max_num_reorder_frames().min(sps.max_dpb_frames()),
|
||||
})
|
||||
}
|
||||
|
||||
/// From an `hvcC` (HEVCDecoderConfigurationRecord) payload.
|
||||
pub fn from_hvcc(hvcc: &[u8]) -> Result<Self> {
|
||||
if hvcc.len() < 23 || hvcc[0] != 1 {
|
||||
return Err(CodecError::Decode("bad hvcC record".into()));
|
||||
}
|
||||
let (length_size, highest_tid) = hvcc_length_size_and_tid(hvcc);
|
||||
let mut sets = Vec::new();
|
||||
let mut pos = 23;
|
||||
for _ in 0..hvcc[22] {
|
||||
let hdr = hvcc.get(pos..pos + 3).ok_or_else(|| CodecError::Decode("hvcC truncated".into()))?;
|
||||
let (kind, n) = (hdr[0] & 0x3f, u16::from_be_bytes([hdr[1], hdr[2]]) as usize);
|
||||
pos += 3;
|
||||
let mut arr = Vec::new();
|
||||
read_sets(hvcc, &mut pos, n, &mut arr)?;
|
||||
if (32..=34).contains(&kind) {
|
||||
sets.extend(arr);
|
||||
}
|
||||
}
|
||||
let nal_type = |n: &[u8]| n.first().map(|h| (h >> 1) & 0x3f);
|
||||
let sps_nal = sets.iter().find(|n| nal_type(n) == Some(33)).ok_or_else(|| CodecError::Decode("hvcC has no SPS".into()))?;
|
||||
if !sets.iter().any(|n| nal_type(n) == Some(32)) || !sets.iter().any(|n| nal_type(n) == Some(34)) {
|
||||
return Err(CodecError::Decode("hvcC lacks a VPS or PPS".into()));
|
||||
}
|
||||
let sps = filmcraft_hevc::params::Sps::parse(&filmcraft_bitstream::unescape_rbsp(sps_nal.get(2..).unwrap_or_default()))
|
||||
.map_err(|e| CodecError::Decode(e.to_string()))?;
|
||||
let crop = sps.crop_rect();
|
||||
let vui = sps.vui.clone().unwrap_or_default();
|
||||
Ok(Self {
|
||||
codec: NalCodec::Hevc,
|
||||
length_size,
|
||||
highest_tid,
|
||||
parameter_sets: sets,
|
||||
coded: (sps.width, sps.height),
|
||||
crop,
|
||||
chroma_format_idc: sps.chroma_format_idc,
|
||||
bit_depth_luma: sps.bit_depth_luma,
|
||||
bit_depth_chroma: sps.bit_depth_chroma,
|
||||
interlaced: false,
|
||||
profile_idc: sps.ptl.profile_idc,
|
||||
color: vui_color(crop.2, crop.3, vui.matrix_coefficients, vui.transfer_characteristics, vui.full_range),
|
||||
par: sar_par(vui.sar),
|
||||
reorder: sps.max_num_reorder as usize,
|
||||
})
|
||||
}
|
||||
|
||||
/// NAL unit types of a sample (stops at the first malformed length prefix).
|
||||
pub fn nal_types(&self, sample: &[u8]) -> Vec<u8> {
|
||||
self.nals(sample).into_iter().filter_map(|n| n.first().map(|&h| self.nal_type(h))).collect()
|
||||
}
|
||||
|
||||
/// The NAL units of a length-prefixed sample (stops at the first malformed length prefix).
|
||||
pub fn nals<'a>(&self, sample: &'a [u8]) -> Vec<&'a [u8]> {
|
||||
let ls = self.length_size;
|
||||
let mut out = Vec::new();
|
||||
let mut pos = 0usize;
|
||||
while let Some(pre) = sample.get(pos..pos.saturating_add(ls)) {
|
||||
if !(1..=4).contains(&ls) {
|
||||
break;
|
||||
}
|
||||
let len = pre.iter().fold(0usize, |a, &b| (a << 8) | b as usize);
|
||||
pos += ls;
|
||||
let Some(n) = sample.get(pos..pos.saturating_add(len)) else { break };
|
||||
pos += len;
|
||||
out.push(n);
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// The NAL unit type in a NAL header byte.
|
||||
pub fn nal_type(&self, header: u8) -> u8 {
|
||||
match self.codec {
|
||||
NalCodec::H264 => header & 0x1f,
|
||||
NalCodec::Hevc => (header >> 1) & 0x3f,
|
||||
}
|
||||
}
|
||||
|
||||
/// Whether `t` is a parameter-set NAL unit type (H.264 SPS / PPS / SPS extension, HEVC VPS /
|
||||
/// SPS / PPS).
|
||||
pub fn is_parameter_set(&self, t: u8) -> bool {
|
||||
match self.codec {
|
||||
NalCodec::H264 => matches!(t, 7 | 8 | 13 | 15),
|
||||
NalCodec::Hevc => (32..=34).contains(&t),
|
||||
}
|
||||
}
|
||||
|
||||
/// Whether decoding can restart at `sample` with no earlier state: an IDR access unit (H.264)
|
||||
/// or an IRAP one (HEVC IDR / CRA / BLA).
|
||||
pub fn is_irap(&self, sample: &[u8]) -> bool {
|
||||
self.nal_types(sample).iter().any(|&t| match self.codec {
|
||||
NalCodec::H264 => t == 5,
|
||||
NalCodec::Hevc => (16..=23).contains(&t),
|
||||
})
|
||||
}
|
||||
|
||||
/// As the software decoder's [`crate::VideoDecoder::is_random_access`] for this stream
|
||||
/// (length-prefixed samples: unknown, the container's sync flags are trusted).
|
||||
pub fn is_random_access(&self, _sample: &[u8]) -> Option<bool> {
|
||||
None
|
||||
}
|
||||
|
||||
/// As the software decoder's [`crate::VideoDecoder::is_disposable`] for this stream.
|
||||
pub fn is_disposable(&self, sample: &[u8]) -> bool {
|
||||
match self.codec {
|
||||
NalCodec::H264 => h264_disposable(sample, self.length_size),
|
||||
NalCodec::Hevc => hevc_disposable(sample, self.length_size, self.highest_tid),
|
||||
}
|
||||
}
|
||||
|
||||
/// Output picture size (cropped).
|
||||
pub fn size(&self) -> (u32, u32) {
|
||||
(self.crop.2, self.crop.3)
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn switch_defaults_to_auto() {
|
||||
// other tests in this process may flip it; only check the round trip
|
||||
let was = hardware_decoding();
|
||||
set_hardware_decoding(false);
|
||||
assert!(!hardware_decoding());
|
||||
set_hardware_decoding(true);
|
||||
assert!(hardware_decoding());
|
||||
set_hardware_decoding(was);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn hostile_records_are_errors() {
|
||||
for rec in [&[][..], &[1, 2, 3], &[1, 0x64, 0, 0x1f, 0xff, 0xe1, 0, 9, 0x67], &[0; 30]] {
|
||||
assert!(NalStreamInfo::from_avcc(rec).is_err());
|
||||
assert!(NalStreamInfo::from_hvcc(rec).is_err());
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn nal_splitting_is_bounded() {
|
||||
let info = NalStreamInfo {
|
||||
codec: NalCodec::Hevc,
|
||||
length_size: 4,
|
||||
highest_tid: None,
|
||||
parameter_sets: Vec::new(),
|
||||
coded: (16, 16),
|
||||
crop: (0, 0, 16, 16),
|
||||
chroma_format_idc: 1,
|
||||
bit_depth_luma: 8,
|
||||
bit_depth_chroma: 8,
|
||||
interlaced: false,
|
||||
profile_idc: 1,
|
||||
color: filmcraft_color::ColorInfo::REC709,
|
||||
par: (1, 1),
|
||||
reorder: 0,
|
||||
};
|
||||
// CRA (21) then a length that runs past the end
|
||||
let s = [0, 0, 0, 2, 21 << 1, 1, 0xff, 0xff, 0xff, 0xff, 1];
|
||||
assert_eq!(info.nal_types(&s), vec![21]);
|
||||
assert!(info.is_irap(&s));
|
||||
assert!(!info.is_irap(&[0, 0, 0, 2, 1 << 1, 1]));
|
||||
}
|
||||
}
|
||||
|
|
@ -20,6 +20,7 @@
|
|||
|
||||
pub mod audio;
|
||||
pub mod gop;
|
||||
pub mod hw;
|
||||
pub mod mkv;
|
||||
pub mod mp4;
|
||||
pub mod mpeg;
|
||||
|
|
@ -63,20 +64,21 @@ impl From<CodecError> for filmcraft_media::MediaError {
|
|||
}
|
||||
}
|
||||
|
||||
/// Our own (pure-Rust) decoders, in the order they are tried.
|
||||
const BUILTIN_FACTORIES: [VideoDecoderFactory; 8] = [
|
||||
video::h264_factory,
|
||||
video::hevc_factory,
|
||||
video::vp9_factory,
|
||||
video::av1_factory,
|
||||
video::prores_factory,
|
||||
video::dnx_factory,
|
||||
video::mjpeg_factory,
|
||||
video::mpeg2_factory,
|
||||
];
|
||||
|
||||
fn factories() -> &'static RwLock<Vec<VideoDecoderFactory>> {
|
||||
static F: std::sync::OnceLock<RwLock<Vec<VideoDecoderFactory>>> = std::sync::OnceLock::new();
|
||||
F.get_or_init(|| {
|
||||
RwLock::new(vec![
|
||||
video::h264_factory,
|
||||
video::hevc_factory,
|
||||
video::vp9_factory,
|
||||
video::av1_factory,
|
||||
video::prores_factory,
|
||||
video::dnx_factory,
|
||||
video::mjpeg_factory,
|
||||
video::mpeg2_factory,
|
||||
])
|
||||
})
|
||||
F.get_or_init(|| RwLock::new(BUILTIN_FACTORIES.to_vec()))
|
||||
}
|
||||
|
||||
/// Register a video decoder factory (tried before previously registered ones).
|
||||
|
|
@ -98,6 +100,17 @@ pub fn make_video_decoder(entry: &filmcraft_isobmff::SampleEntry) -> Result<Box<
|
|||
Err(CodecError::Unsupported(format!("no decoder for {} video", entry.codec.name())))
|
||||
}
|
||||
|
||||
/// Create one of our own (software) decoders for a sample entry, skipping registered factories:
|
||||
/// what a hardware decoder falls back to when it fails mid-stream.
|
||||
pub fn software_video_decoder(entry: &filmcraft_isobmff::SampleEntry) -> Result<Box<dyn VideoDecoder>> {
|
||||
for f in BUILTIN_FACTORIES {
|
||||
if let Some(r) = f(entry) {
|
||||
return r;
|
||||
}
|
||||
}
|
||||
Err(CodecError::Unsupported(format!("no decoder for {} video", entry.codec.name())))
|
||||
}
|
||||
|
||||
/// Openers for the engine's media pool (MP4/MOV, Matroska/WebM, MXF, Ogg Opus/Vorbis, MPEG TS/PS and
|
||||
/// MPEG-1/2 video elementary streams, standalone audio).
|
||||
pub fn openers() -> Vec<filmcraft_media::Opener> {
|
||||
|
|
|
|||
|
|
@ -128,6 +128,42 @@ pub(crate) fn tight_plane<T: Copy>(src: Vec<T>, stride: usize, w: usize, h: usiz
|
|||
out
|
||||
}
|
||||
|
||||
/// Colour of an H.264 / HEVC picture from its VUI code points (ITU-T H.273), as the decoders
|
||||
/// report it: BT.709 by default with the matrix guessed from the size, the signalled matrix and
|
||||
/// transfer when known, and full range when flagged. Shared by the software and hardware
|
||||
/// decoders so the two are interchangeable.
|
||||
pub fn vui_color(width: u32, height: u32, matrix: u8, transfer: u8, full_range: bool) -> filmcraft_color::ColorInfo {
|
||||
let mut color = filmcraft_color::ColorInfo { matrix: filmcraft_frame::default_matrix(width, height), ..filmcraft_color::ColorInfo::REC709 };
|
||||
if let Some(m) = filmcraft_color::Matrix::from_code(matrix) {
|
||||
color.matrix = m;
|
||||
}
|
||||
if let Some(t) = filmcraft_color::Transfer::from_code(transfer) {
|
||||
color.transfer = t;
|
||||
}
|
||||
if full_range {
|
||||
color.range = filmcraft_color::Range::Full;
|
||||
}
|
||||
color
|
||||
}
|
||||
|
||||
/// Pixel aspect ratio from a VUI sample aspect ratio ((0, 0) = unspecified = square).
|
||||
pub fn sar_par(sar: (u16, u16)) -> (u32, u32) {
|
||||
if sar.0 > 0 && sar.1 > 0 { (sar.0 as u32, sar.1 as u32) } else { (1, 1) }
|
||||
}
|
||||
|
||||
/// NAL length-prefix size of an `avcC` record (byte 4: lengthSizeMinusOne).
|
||||
pub fn avcc_length_size(avcc: &[u8]) -> usize {
|
||||
avcc.get(4).map_or(4, |b| (b & 3) as usize + 1)
|
||||
}
|
||||
|
||||
/// NAL length-prefix size and highest TemporalId of an `hvcC` record.
|
||||
pub fn hvcc_length_size_and_tid(hvcc: &[u8]) -> (usize, Option<u8>) {
|
||||
// hvcC byte 21: constantFrameRate(2) numTemporalLayers(3) temporalIdNested(1) lengthSizeMinusOne(2)
|
||||
let length_size = hvcc.get(21).map_or(4, |b| (b & 3) as usize + 1);
|
||||
let highest_tid = hvcc.get(21).map(|b| (b >> 3) & 7).filter(|&n| n > 0).map(|n| n - 1);
|
||||
(length_size, highest_tid)
|
||||
}
|
||||
|
||||
/// A factory returns `None` when it does not handle the entry.
|
||||
pub type VideoDecoderFactory = fn(&SampleEntry) -> Option<Result<Box<dyn VideoDecoder>>>;
|
||||
|
||||
|
|
@ -169,7 +205,7 @@ pub struct H264Decoder {
|
|||
impl H264Decoder {
|
||||
pub fn new(avcc: Vec<u8>) -> Result<Self> {
|
||||
let dec = filmcraft_h264::Decoder::from_avcc(&avcc).map_err(|e| CodecError::Decode(e.to_string()))?;
|
||||
let length_size = avcc.get(4).map_or(4, |b| (b & 3) as usize + 1);
|
||||
let length_size = avcc_length_size(&avcc);
|
||||
Ok(Self { avcc, dec, length_size, draft: false })
|
||||
}
|
||||
/// A decoder for Annex B byte-stream samples (start codes, in-band parameter sets: MXF, TS).
|
||||
|
|
@ -184,17 +220,8 @@ impl H264Decoder {
|
|||
let y = tight_plane(p.y, p.y_stride, w, h);
|
||||
let u = tight_plane(p.u, p.uv_stride, cw, ch);
|
||||
let v = tight_plane(p.v, p.uv_stride, cw, ch);
|
||||
let mut color = filmcraft_color::ColorInfo { matrix: filmcraft_frame::default_matrix(p.width, p.height), ..filmcraft_color::ColorInfo::REC709 };
|
||||
if let Some(m) = filmcraft_color::Matrix::from_code(p.color.matrix) {
|
||||
color.matrix = m;
|
||||
}
|
||||
if let Some(t) = filmcraft_color::Transfer::from_code(p.color.transfer) {
|
||||
color.transfer = t;
|
||||
}
|
||||
if p.color.full_range {
|
||||
color.range = filmcraft_color::Range::Full;
|
||||
}
|
||||
let par = if p.sar.0 > 0 && p.sar.1 > 0 { (p.sar.0 as u32, p.sar.1 as u32) } else { (1, 1) };
|
||||
let color = vui_color(p.width, p.height, p.color.matrix, p.color.transfer, p.color.full_range);
|
||||
let par = sar_par(p.sar);
|
||||
let frame = VideoFrame {
|
||||
width: p.width,
|
||||
height: p.height,
|
||||
|
|
@ -265,9 +292,7 @@ pub struct HevcDecoder {
|
|||
impl HevcDecoder {
|
||||
pub fn new(hvcc: Vec<u8>) -> Result<Self> {
|
||||
let dec = filmcraft_hevc::Decoder::from_hvcc(&hvcc).map_err(|e| CodecError::Decode(e.to_string()))?;
|
||||
// hvcC byte 21: constantFrameRate(2) numTemporalLayers(3) temporalIdNested(1) lengthSizeMinusOne(2)
|
||||
let length_size = hvcc.get(21).map_or(4, |b| (b & 3) as usize + 1);
|
||||
let highest_tid = hvcc.get(21).map(|b| (b >> 3) & 7).filter(|&n| n > 0).map(|n| n - 1);
|
||||
let (length_size, highest_tid) = hvcc_length_size_and_tid(&hvcc);
|
||||
Ok(Self { hvcc, dec, length_size, highest_tid })
|
||||
}
|
||||
/// A decoder for Annex B byte-stream samples (start codes, in-band parameter sets: TS).
|
||||
|
|
@ -299,17 +324,8 @@ impl HevcDecoder {
|
|||
alpha: None,
|
||||
},
|
||||
};
|
||||
let mut color = filmcraft_color::ColorInfo { matrix: filmcraft_frame::default_matrix(p.width, p.height), ..filmcraft_color::ColorInfo::REC709 };
|
||||
if let Some(m) = filmcraft_color::Matrix::from_code(p.color.matrix) {
|
||||
color.matrix = m;
|
||||
}
|
||||
if let Some(t) = filmcraft_color::Transfer::from_code(p.color.transfer) {
|
||||
color.transfer = t;
|
||||
}
|
||||
if p.color.full_range {
|
||||
color.range = filmcraft_color::Range::Full;
|
||||
}
|
||||
let par = if p.sar.0 > 0 && p.sar.1 > 0 { (p.sar.0 as u32, p.sar.1 as u32) } else { (1, 1) };
|
||||
let color = vui_color(p.width, p.height, p.color.matrix, p.color.transfer, p.color.full_range);
|
||||
let par = sar_par(p.sar);
|
||||
let frame = VideoFrame { width: p.width, height: p.height, data, color, par, pts: filmcraft_time::Tick::ZERO };
|
||||
DecodedFrame { pts: p.pts, frame, draft: p.draft }
|
||||
}
|
||||
|
|
|
|||
|
|
@ -82,6 +82,11 @@ pub struct PlaybackPrefs {
|
|||
/// skip deblocking (faster; those frames are approximate). Never used for exports, renders or
|
||||
/// a paused frame. Off by default.
|
||||
pub draft_decode: bool,
|
||||
/// "Hardware decoding": `auto` (default: the system's hardware decoder for streams it
|
||||
/// supports, e.g. VideoToolbox H.264 / HEVC on macOS, falling back to FilmCraft's own decoder)
|
||||
/// or `off` (always FilmCraft's own decoders). Media already open keeps its decoder until it
|
||||
/// is reopened.
|
||||
pub hardware_decoding: String,
|
||||
}
|
||||
|
||||
impl Default for PlaybackPrefs {
|
||||
|
|
@ -94,6 +99,7 @@ impl Default for PlaybackPrefs {
|
|||
enable_transmit: false,
|
||||
disable_video_in_background: true,
|
||||
draft_decode: false,
|
||||
hardware_decoding: "auto".into(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -477,6 +477,7 @@ impl Session {
|
|||
pub fn start_autosave(&mut self, cfg: autosave::AutosaveConfig) -> std::io::Result<()> {
|
||||
let prefs_path = cfg.data_dir.join("preferences.json");
|
||||
self.prefs = autosave::Preferences::load(&prefs_path);
|
||||
self.prefs.apply_hardware_decoding();
|
||||
self.media.set_use_proxies(self.prefs.media.enable_proxies);
|
||||
self.shortcuts.set_dir(&cfg.data_dir);
|
||||
self.presets.set_dir(&cfg.data_dir);
|
||||
|
|
@ -548,6 +549,7 @@ impl Session {
|
|||
pub fn set_prefs(&mut self, p: autosave::Preferences) -> std::io::Result<()> {
|
||||
let cache_changed = self.prefs.media_cache != p.media_cache;
|
||||
self.prefs = p;
|
||||
self.prefs.apply_hardware_decoding();
|
||||
if cache_changed {
|
||||
self.apply_media_cache();
|
||||
}
|
||||
|
|
|
|||
|
|
@ -9,6 +9,7 @@ use crate::Session;
|
|||
/// measure an interval).
|
||||
pub fn decode_json() -> Value {
|
||||
let g = filmcraft_codecs::gop_stats();
|
||||
let hw = filmcraft_codecs::hw::hw_stats();
|
||||
json!({
|
||||
"requests": g.hits + g.misses,
|
||||
"cacheHits": g.hits,
|
||||
|
|
@ -22,6 +23,17 @@ pub fn decode_json() -> Value {
|
|||
"evicted": g.evicted,
|
||||
"decodeMs": g.decode_ns as f64 / 1e6,
|
||||
"decodeMsPerSample": g.decode_ms_per_sample(),
|
||||
"framesDecoded": g.frames,
|
||||
// Settings ▸ Playback ▸ Hardware decoding: pictures from OS hardware decoders vs ours,
|
||||
// hardware decoders created, streams handed to software up front, mid-stream fallbacks.
|
||||
"hardware": {
|
||||
"enabled": filmcraft_codecs::hw::hardware_decoding(),
|
||||
"frames": hw.frames,
|
||||
"softwareFrames": g.frames.saturating_sub(hw.frames),
|
||||
"sessions": hw.sessions,
|
||||
"declined": hw.declined,
|
||||
"fallbacks": hw.fallbacks,
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
|
|
@ -48,9 +60,24 @@ mod tests {
|
|||
s.execute("file.openDemoProject", json!({})).unwrap();
|
||||
let undo = s.history.undo.len();
|
||||
let v = s.execute("perf.stats", json!({})).unwrap();
|
||||
for k in ["requests", "cacheHitRate", "seeks", "samplesDecoded", "samplesSkipped", "draftFrames", "h264Threads", "decodeMs", "decodeMsPerSample"] {
|
||||
for k in [
|
||||
"requests",
|
||||
"cacheHitRate",
|
||||
"seeks",
|
||||
"samplesDecoded",
|
||||
"samplesSkipped",
|
||||
"draftFrames",
|
||||
"h264Threads",
|
||||
"decodeMs",
|
||||
"decodeMsPerSample",
|
||||
"framesDecoded",
|
||||
] {
|
||||
assert!(v["decode"][k].is_number(), "decode.{k} in {v}");
|
||||
}
|
||||
for k in ["frames", "softwareFrames", "sessions", "declined", "fallbacks"] {
|
||||
assert!(v["decode"]["hardware"][k].is_number(), "decode.hardware.{k} in {v}");
|
||||
}
|
||||
assert!(v["decode"]["hardware"]["enabled"].is_boolean());
|
||||
assert!(v["media"]["openSources"].is_number());
|
||||
assert_eq!(v["jobs"]["running"], json!(0));
|
||||
assert_eq!(s.history.undo.len(), undo, "a query adds no undo step");
|
||||
|
|
|
|||
|
|
@ -653,6 +653,7 @@ const CACHE_MGMT: &[(&str, &str)] = &[
|
|||
];
|
||||
const AUTO_SCROLL: &[(&str, &str)] = &[("noScroll", "No Scroll"), ("pageScroll", "Page Scroll"), ("smoothScroll", "Smooth Scroll")];
|
||||
const MOUSE_SCROLL: &[(&str, &str)] = &[("vertical", "Vertical"), ("horizontal", "Horizontal")];
|
||||
const HW_DECODE: &[(&str, &str)] = &[("auto", "Auto"), ("off", "Off")];
|
||||
const TRACKS: &[(&str, &str)] = &[("useFile", "Use File"), ("mono", "Mono"), ("stereo", "Stereo"), ("5.1", "5.1"), ("adaptive", "Adaptive")];
|
||||
pub const DURATION_UNITS: &[(&str, &str)] = &[("frames", "Frames"), ("seconds", "Seconds")];
|
||||
|
||||
|
|
@ -952,6 +953,10 @@ static CATEGORIES: &[Category] = &[
|
|||
b("playback.enableTransmit", "Enable Mercury Transmit", false),
|
||||
b("playback.disableVideoInBackground", "Disable video output when in the background", false),
|
||||
b("playback.draftDecode", "Draft decoding at reduced playback resolution (H.264: faster, some frames less filtered)", true),
|
||||
f("playback.hardwareDecoding", "Hardware decoding", Kind::Choice(HW_DECODE), true),
|
||||
Row::Note(
|
||||
"Hardware decoding: Auto uses the system's video decoder (VideoToolbox on macOS) for the H.264 and HEVC streams it supports, and FilmCraft's own decoder for everything else or if the hardware fails. Media that is already open keeps its decoder until it is reopened.",
|
||||
),
|
||||
],
|
||||
},
|
||||
Category {
|
||||
|
|
@ -1122,6 +1127,12 @@ pub fn migrate(v: &mut Value) {
|
|||
}
|
||||
|
||||
impl crate::autosave::Preferences {
|
||||
/// Settings ▸ Playback ▸ Hardware decoding as the process-wide switch the hardware decoder
|
||||
/// factories consult (new decoders only).
|
||||
pub fn apply_hardware_decoding(&self) {
|
||||
filmcraft_codecs::hw::set_hardware_decoding(self.playback.hardware_decoding != "off");
|
||||
}
|
||||
|
||||
/// Values of one category as a JSON object.
|
||||
pub fn category_value(&self, id: &str) -> Option<Value> {
|
||||
self.to_value().get(id).cloned()
|
||||
|
|
|
|||
|
|
@ -480,3 +480,15 @@ fn auto_transcribe_on_import_and_transcription_defaults() {
|
|||
set(&mut s, "mediaAnalysis.whisperModel", json!("whisper-tiny"));
|
||||
let _ = std::fs::remove_dir_all(&dir);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn hardware_decoding_setting_drives_the_decoder_switch() {
|
||||
let mut s = Session::default();
|
||||
assert_eq!(s.prefs.playback.hardware_decoding, "auto", "Auto by default");
|
||||
set(&mut s, "playback.hardwareDecoding", json!("off"));
|
||||
assert!(!filmcraft_codecs::hw::hardware_decoding(), "Off reaches the decoder registry");
|
||||
assert!(s.execute("prefs.set", json!({"key": "playback.hardwareDecoding", "value": "gpu"})).is_err());
|
||||
set(&mut s, "playback.hardwareDecoding", json!("auto"));
|
||||
assert!(filmcraft_codecs::hw::hardware_decoding());
|
||||
assert!(settings::field("playback.hardwareDecoding").is_some_and(|f| f.wired && matches!(f.kind, Kind::Choice(_))));
|
||||
}
|
||||
|
|
|
|||
|
|
@ -0,0 +1,40 @@
|
|||
[package]
|
||||
name = "filmcraft-platform"
|
||||
description = "OS media integration: hardware video decoding through the operating system's codecs (VideoToolbox on macOS), with transparent fallback to the pure-Rust decoders"
|
||||
version.workspace = true
|
||||
edition.workspace = true
|
||||
license.workspace = true
|
||||
rust-version.workspace = true
|
||||
|
||||
# The one crate allowed to contain `unsafe` (OS media FFI; AGENTS.md §0.3, docs/adr/0001-platform-ffi.md).
|
||||
# The workspace lints, except that `unsafe_code` is `deny` instead of `forbid`: only the FFI modules
|
||||
# opt in with `#[allow(unsafe_code)]`, and every `unsafe` block carries a `// SAFETY:` comment.
|
||||
[lints.rust]
|
||||
unsafe_code = "deny"
|
||||
|
||||
[lints.clippy]
|
||||
dbg_macro = "warn"
|
||||
todo = "warn"
|
||||
too_many_arguments = "allow"
|
||||
needless_range_loop = "allow"
|
||||
type_complexity = "allow"
|
||||
undocumented_unsafe_blocks = "deny"
|
||||
|
||||
[dependencies]
|
||||
filmcraft-codecs = { workspace = true }
|
||||
filmcraft-isobmff = { workspace = true }
|
||||
log = { workspace = true }
|
||||
|
||||
[target.'cfg(target_os = "macos")'.dependencies]
|
||||
filmcraft-color = { workspace = true }
|
||||
filmcraft-frame = { workspace = true }
|
||||
filmcraft-time = { workspace = true }
|
||||
objc2-core-foundation = "0.3.2"
|
||||
objc2-core-media = "0.3.2"
|
||||
objc2-core-video = "0.3.2"
|
||||
objc2-video-toolbox = "0.3.2"
|
||||
|
||||
[dev-dependencies]
|
||||
filmcraft-testkit = { workspace = true }
|
||||
filmcraft-media = { workspace = true }
|
||||
filmcraft-frame = { workspace = true }
|
||||
|
|
@ -0,0 +1,69 @@
|
|||
# filmcraft-platform
|
||||
|
||||
OS media integration for FilmCraft (layer L5): hardware video decoding through the operating
|
||||
system's codecs, behind `filmcraft_codecs::VideoDecoder`. It holds OS media FFI and nothing else.
|
||||
It is the one crate of the workspace allowed to contain `unsafe`, under the rules of
|
||||
[ADR 0001](../../docs/adr/0001-platform-ffi.md) and [AGENTS.md](../../AGENTS.md) §0.3.
|
||||
|
||||
```rust
|
||||
// at startup (the desktop app, filmcraft-cli, the bench)
|
||||
let availability = filmcraft_platform::register(); // Available("VideoToolbox") on macOS
|
||||
```
|
||||
|
||||
## What it does
|
||||
|
||||
- **macOS: VideoToolbox H.264 (`avcC`) and HEVC (`hvcC`)**, 8- and 10-bit, 4:2:0 and 4:2:2
|
||||
(`videotoolbox.rs`). The session is created from the sample entry's parameter sets with a
|
||||
hardware decoder *required*; samples go in as `CMSampleBuffer`s with asynchronous decompression
|
||||
(two access units in flight); the output callback copies each NV12 / P010-style biplanar
|
||||
`CVPixelBuffer` into planar `Yuv8` / `Yuv16` (chroma deinterleaved, 10-bit samples shifted down
|
||||
from the high bits, cropped to the conformance window when the buffer is the coded size). A
|
||||
reorder buffer of the stream's own depth (`max_num_reorder_frames` /
|
||||
`sps_max_num_reorder_pics`) restores presentation order; a run starting at an HEVC CRA leaves
|
||||
out its RASL pictures, as our decoder does. Each seek (`reset`) starts a fresh session.
|
||||
- **Other systems:** `register()` does nothing and returns `Availability::Unavailable`.
|
||||
- **`HybridDecoder`** (`hybrid.rs`, safe code): the hardware decoder plus the means to build our
|
||||
software decoder for the same `SampleEntry` (`filmcraft_codecs::software_video_decoder`). On a
|
||||
mid-stream failure (decode error, invalidated session, changed in-band parameter sets) it replays
|
||||
the samples since the last restart point (IDR / IRAP; for an HEVC CRA the one before, so its RASL
|
||||
pictures decode) through the software decoder, drops pictures already returned, keeps the ones
|
||||
the hardware had decoded but not returned, and stays in software for that instance. The replay
|
||||
log is bounded (600 samples / 256 MB); beyond it one error is returned and the next seek restarts
|
||||
in software. Streams our decoders cannot decode (HEVC 4:2:2) have no fallback: the error stands.
|
||||
|
||||
## Guarantees
|
||||
|
||||
- **Never undecodable:** the factory declines (returns `None`, so the software decoder is used)
|
||||
when Settings ▸ Playback ▸ Hardware decoding is Off, for formats it does not take (field-coded
|
||||
H.264, bit depths other than 8 / 10, 4:4:4 or monochrome, luma / chroma depth mismatch, larger
|
||||
than 8192×8192) and when VideoToolbox cannot create a hardware session.
|
||||
- **Interchangeable:** colour, pixel aspect, pts, presentation order, `is_random_access` and
|
||||
`is_disposable` come from the software decoders' own helpers (`filmcraft_codecs::hw`,
|
||||
`video::vui_color`, `sar_par`).
|
||||
- **Never crash:** no `unwrap` / `expect` / `panic!` outside tests; the output callback runs under
|
||||
`catch_unwind`; every `unsafe` block has a `// SAFETY:` comment; the public API is safe.
|
||||
- **Counted:** `perf.stats` `decode.hardware` (frames, software frames, sessions, declined,
|
||||
fallbacks; `filmcraft_codecs::hw::hw_stats`).
|
||||
|
||||
## Tests
|
||||
|
||||
| test | what |
|
||||
|---|---|
|
||||
| `tests/videotoolbox.rs` (macOS) | H.264 High, HEVC Main (open GOP: CRA + RASL) and HEVC Main 10, 640×360 (coded 368: cropping) with B-frames: every picture **bit-exact** with our software decoder, same pts order, count, colour and aspect, also after `reset` + reseek to every later sync sample, mid-stream `flush`, and a full pass after resets; forced mid-stream failures (`VtDecoder::fail_after`) at five points continue with the software decoder's exact output; seeded mutation of samples and parameter sets (bit flips, truncation, corrupt length prefixes) never panics or hangs; HEVC 4:2:2 10-bit is bit-exact with ffmpeg's decode |
|
||||
| `tests/fallback.rs` (every OS) | `HybridDecoder` with a stand-in hardware decoder failing after N samples (every sync sample ± a few, first / last sample, after a seek): output identical to the software decoder; in-band parameter sets identical to the sample entry's stay in hardware, different ones switch to software |
|
||||
| `tests/setting.rs` | Hardware decoding Off gives the software decoder through `make_video_decoder` and the media stack (no hardware frames); Auto gives VideoToolbox where available |
|
||||
|
||||
Fixtures are made with ffmpeg into `target/fixtures/platform/` (generator only, never linked);
|
||||
tests skip without ffmpeg or without a hardware decoder.
|
||||
|
||||
## Performance
|
||||
|
||||
M4 Pro, load 150–190 (`cargo xtask bench --hw off|auto`): CPU per decoded frame H.264 2160p
|
||||
119 → 4.2 ms, HEVC 2160p 86 → 3.7 ms; decode 35 → 107 fps and 49 → 217 fps; 4K H.264 and HEVC
|
||||
playback with no dropped frames at Full, 1/2 and 1/4. Details in
|
||||
[docs/performance.md](../../docs/performance.md).
|
||||
|
||||
## Not yet
|
||||
|
||||
Zero-copy upload of `CVPixelBuffer`s into wgpu textures; hardware encoding; Media Foundation /
|
||||
D3D11 (Windows) and VA-API (Linux) decoders; field-coded H.264.
|
||||
|
|
@ -0,0 +1,235 @@
|
|||
//! [`HybridDecoder`]: a hardware decoder that turns into the software decoder for the same stream
|
||||
//! when it fails mid-stream (decode error, session lost to a GPU change or sleep, an in-band
|
||||
//! parameter-set change it was not set up for), without the caller noticing.
|
||||
//!
|
||||
//! The hybrid keeps the samples fed since the last point decoding can restart from (the run's
|
||||
//! first sample, then every IDR / IRAP access unit). On a failure it builds the software decoder
|
||||
//! ([`filmcraft_codecs::software_video_decoder`], skipping registered factories), replays those
|
||||
//! samples through it and carries on, dropping pictures the hardware already returned and keeping
|
||||
//! the ones it had decoded but not yet returned, so the output is the software decoder's own
|
||||
//! sequence. That replay log is bounded; past the bound a failure is reported as an error once
|
||||
//! and the instance continues in software from the next seek ([`VideoDecoder::reset`]).
|
||||
|
||||
use std::collections::{BTreeMap, BTreeSet};
|
||||
|
||||
use filmcraft_codecs::hw::NalStreamInfo;
|
||||
use filmcraft_codecs::{CodecError, DecodedFrame, Result, VideoDecoder};
|
||||
use filmcraft_isobmff::SampleEntry;
|
||||
|
||||
/// Replay log bounds: samples and bytes since the last restart point.
|
||||
const LOG_MAX_SAMPLES: usize = 600;
|
||||
const LOG_MAX_BYTES: usize = 256 << 20;
|
||||
/// Presentation times remembered as returned (older ones are forgotten first).
|
||||
const EMITTED_MAX: usize = 4096;
|
||||
|
||||
/// A hardware decoder with a transparent software fallback.
|
||||
pub struct HybridDecoder {
|
||||
hw: Option<Box<dyn VideoDecoder>>,
|
||||
sw: Option<Box<dyn VideoDecoder>>,
|
||||
entry: SampleEntry,
|
||||
info: NalStreamInfo,
|
||||
draft: bool,
|
||||
/// Samples (and pts) since the last restart point; `None` once the bound was passed.
|
||||
log: Option<Vec<(Vec<u8>, i64)>>,
|
||||
log_bytes: usize,
|
||||
/// Index in the log of the most recent restart point.
|
||||
last_irap: usize,
|
||||
/// Pictures returned since the last reset (by pts).
|
||||
emitted: BTreeSet<i64>,
|
||||
/// Pictures the hardware had decoded but not returned when it failed.
|
||||
carry: BTreeMap<i64, DecodedFrame>,
|
||||
}
|
||||
|
||||
impl HybridDecoder {
|
||||
/// Wrap `hw`, a decoder for `entry` (whose stream is described by `info`).
|
||||
pub fn new(hw: Box<dyn VideoDecoder>, entry: SampleEntry, info: NalStreamInfo) -> Self {
|
||||
filmcraft_codecs::hw::note_hw_session();
|
||||
Self {
|
||||
hw: Some(hw),
|
||||
sw: None,
|
||||
entry,
|
||||
info,
|
||||
draft: false,
|
||||
log: Some(Vec::new()),
|
||||
log_bytes: 0,
|
||||
last_irap: 0,
|
||||
emitted: BTreeSet::new(),
|
||||
carry: BTreeMap::new(),
|
||||
}
|
||||
}
|
||||
|
||||
/// Whether the hardware decoder is still in use (false after a fallback).
|
||||
pub fn is_hardware(&self) -> bool {
|
||||
self.hw.is_some()
|
||||
}
|
||||
|
||||
fn remember(&mut self, sample: &[u8], pts: i64) {
|
||||
if self.info.is_irap(sample) {
|
||||
// An HEVC CRA keeps the previous restart point: replaying from the CRA itself would
|
||||
// treat it as a first picture and drop its RASL pictures, which the hardware decodes.
|
||||
let cra = self.info.codec == filmcraft_codecs::hw::NalCodec::Hevc && self.info.nal_types(sample).contains(&21);
|
||||
match self.log.as_mut() {
|
||||
Some(log) if cra => {
|
||||
log.drain(..self.last_irap.min(log.len()));
|
||||
self.log_bytes = log.iter().map(|s| s.0.len()).sum();
|
||||
}
|
||||
_ => {
|
||||
self.log = Some(Vec::new());
|
||||
self.log_bytes = 0;
|
||||
}
|
||||
}
|
||||
self.last_irap = self.log.as_ref().map_or(0, Vec::len);
|
||||
}
|
||||
let Some(log) = self.log.as_mut() else { return };
|
||||
self.log_bytes = self.log_bytes.saturating_add(sample.len());
|
||||
if log.len() >= LOG_MAX_SAMPLES || self.log_bytes > LOG_MAX_BYTES {
|
||||
self.log = None;
|
||||
return;
|
||||
}
|
||||
log.push((sample.to_vec(), pts));
|
||||
}
|
||||
|
||||
fn note_emitted(&mut self, out: &[DecodedFrame]) {
|
||||
for f in out {
|
||||
self.emitted.insert(f.pts);
|
||||
}
|
||||
while self.emitted.len() > EMITTED_MAX {
|
||||
self.emitted.pop_first();
|
||||
}
|
||||
}
|
||||
|
||||
/// In-band parameter sets that differ from the sample entry's (the hardware session was set
|
||||
/// up from those).
|
||||
fn parameter_sets_changed(&self, sample: &[u8]) -> bool {
|
||||
self.info
|
||||
.nals(sample)
|
||||
.into_iter()
|
||||
.any(|n| n.first().is_some_and(|&h| self.info.is_parameter_set(self.info.nal_type(h))) && !self.info.parameter_sets.iter().any(|p| p == n))
|
||||
}
|
||||
|
||||
/// Software output after a fallback: pictures already returned are dropped, carried hardware
|
||||
/// pictures are merged in presentation order (a software picture replaces the carried one with
|
||||
/// the same pts).
|
||||
fn merge(&mut self, frames: Vec<DecodedFrame>) -> Vec<DecodedFrame> {
|
||||
let mut out = Vec::with_capacity(frames.len());
|
||||
for f in frames {
|
||||
if self.emitted.contains(&f.pts) {
|
||||
self.carry.remove(&f.pts);
|
||||
continue;
|
||||
}
|
||||
while let Some(e) = self.carry.first_entry() {
|
||||
if *e.key() >= f.pts {
|
||||
break;
|
||||
}
|
||||
out.push(e.remove());
|
||||
}
|
||||
self.carry.remove(&f.pts);
|
||||
out.push(f);
|
||||
}
|
||||
self.note_emitted(&out);
|
||||
out
|
||||
}
|
||||
|
||||
/// Switch to the software decoder after `err`, replaying the log (which ends with the sample
|
||||
/// that failed).
|
||||
fn fall_back(&mut self, err: CodecError) -> Result<Vec<DecodedFrame>> {
|
||||
let Some(mut hw) = self.hw.take() else { return Err(err) };
|
||||
log::warn!("{} failed ({err}); continuing with the software decoder", hw.name());
|
||||
filmcraft_codecs::hw::note_hw_fallback();
|
||||
for f in hw.flush() {
|
||||
if !self.emitted.contains(&f.pts) {
|
||||
self.carry.insert(f.pts, f);
|
||||
}
|
||||
}
|
||||
drop(hw);
|
||||
let mut sw = filmcraft_codecs::software_video_decoder(&self.entry).map_err(|e| CodecError::Decode(format!("{err}; no software decoder: {e}")))?;
|
||||
sw.set_draft(self.draft);
|
||||
self.sw = Some(sw);
|
||||
let Some(log) = self.log.take() else {
|
||||
// Too far from a restart point to replay: the next seek restarts in software.
|
||||
self.carry.clear();
|
||||
return Err(CodecError::Decode(format!("{err} (continuing in software after the next seek)")));
|
||||
};
|
||||
let mut frames = Vec::new();
|
||||
for (s, p) in &log {
|
||||
let Some(sw) = self.sw.as_mut() else { break };
|
||||
frames.extend(sw.decode(s, *p)?);
|
||||
}
|
||||
self.log = Some(log);
|
||||
Ok(self.merge(frames))
|
||||
}
|
||||
}
|
||||
|
||||
impl VideoDecoder for HybridDecoder {
|
||||
fn decode(&mut self, sample: &[u8], pts: i64) -> Result<Vec<DecodedFrame>> {
|
||||
if let Some(sw) = self.sw.as_mut() {
|
||||
let frames = sw.decode(sample, pts)?;
|
||||
return Ok(self.merge(frames));
|
||||
}
|
||||
self.remember(sample, pts);
|
||||
if self.parameter_sets_changed(sample) {
|
||||
return self.fall_back(CodecError::Decode("in-band parameter sets differ from the sample entry".into()));
|
||||
}
|
||||
let Some(hw) = self.hw.as_mut() else {
|
||||
return Err(CodecError::Decode("no decoder".into()));
|
||||
};
|
||||
match hw.decode(sample, pts) {
|
||||
Ok(out) => {
|
||||
filmcraft_codecs::hw::note_hw_frames(out.len());
|
||||
self.note_emitted(&out);
|
||||
Ok(out)
|
||||
}
|
||||
Err(e) => self.fall_back(e),
|
||||
}
|
||||
}
|
||||
|
||||
fn flush(&mut self) -> Vec<DecodedFrame> {
|
||||
if let Some(hw) = self.hw.as_mut() {
|
||||
let out = hw.flush();
|
||||
filmcraft_codecs::hw::note_hw_frames(out.len());
|
||||
self.note_emitted(&out);
|
||||
return out;
|
||||
}
|
||||
let frames = self.sw.as_mut().map(|d| d.flush()).unwrap_or_default();
|
||||
let mut out = self.merge(frames);
|
||||
out.extend(std::mem::take(&mut self.carry).into_values());
|
||||
out
|
||||
}
|
||||
|
||||
fn reset(&mut self) {
|
||||
if let Some(hw) = self.hw.as_mut() {
|
||||
hw.reset();
|
||||
}
|
||||
if let Some(sw) = self.sw.as_mut() {
|
||||
sw.reset();
|
||||
}
|
||||
self.log = Some(Vec::new());
|
||||
self.log_bytes = 0;
|
||||
self.last_irap = 0;
|
||||
self.emitted.clear();
|
||||
self.carry.clear();
|
||||
}
|
||||
|
||||
fn name(&self) -> &str {
|
||||
match (&self.hw, &self.sw) {
|
||||
(Some(hw), _) => hw.name(),
|
||||
(None, Some(sw)) => sw.name(),
|
||||
(None, None) => "hardware decoder",
|
||||
}
|
||||
}
|
||||
|
||||
fn is_random_access(&self, sample: &[u8]) -> Option<bool> {
|
||||
self.info.is_random_access(sample)
|
||||
}
|
||||
|
||||
fn is_disposable(&self, sample: &[u8]) -> bool {
|
||||
self.info.is_disposable(sample)
|
||||
}
|
||||
|
||||
fn set_draft(&mut self, on: bool) {
|
||||
self.draft = on;
|
||||
if let Some(sw) = self.sw.as_mut() {
|
||||
sw.set_draft(on);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,86 @@
|
|||
//! OS media integration (layer L5): hardware video decoding through the operating system's codecs.
|
||||
//!
|
||||
//! [`register`] puts the platform's hardware decoder factory in front of FilmCraft's own decoders
|
||||
//! (`filmcraft_codecs::register_video_decoder`). Today that is VideoToolbox on macOS for H.264
|
||||
//! (`avcC`) and HEVC (`hvcC`) streams, 8- and 10-bit, 4:2:0 and 4:2:2; on other systems
|
||||
//! registration does nothing and reports [`Availability::Unavailable`].
|
||||
//!
|
||||
//! Hardware decoding never makes a file undecodable:
|
||||
//!
|
||||
//! - the factory declines (falls through to the software decoder) when Settings ▸ Playback ▸
|
||||
//! Hardware decoding is Off (`filmcraft_codecs::hw::set_hardware_decoding`), when the stream's
|
||||
//! format is one the hardware path does not take, or when the OS cannot create a hardware
|
||||
//! session for it (profile, size, no hardware decoder);
|
||||
//! - a decoder that fails mid-stream switches to the software decoder transparently
|
||||
//! ([`HybridDecoder`]) and logs it.
|
||||
//!
|
||||
//! Pictures are the software decoder's: same planes (bit-exact on the parity fixtures), colour,
|
||||
//! pixel aspect, pts and presentation order, so the two are interchangeable.
|
||||
//!
|
||||
//! This is the one crate allowed to use `unsafe` (OS FFI), and only in its FFI modules
|
||||
//! (docs/adr/0001-platform-ffi.md).
|
||||
|
||||
#![cfg_attr(not(test), deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable))]
|
||||
|
||||
pub mod hybrid;
|
||||
#[cfg(target_os = "macos")]
|
||||
#[allow(unsafe_code)]
|
||||
pub mod videotoolbox;
|
||||
|
||||
pub use hybrid::HybridDecoder;
|
||||
|
||||
/// What [`register`] made available.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub enum Availability {
|
||||
/// A hardware decoder factory was registered (its name).
|
||||
Available(&'static str),
|
||||
/// Nothing to register on this system (why).
|
||||
Unavailable(&'static str),
|
||||
}
|
||||
|
||||
/// Register the platform's hardware video decoders (call once at startup; repeated calls are
|
||||
/// harmless). Streams they do not take, and every stream while hardware decoding is Off, keep
|
||||
/// using FilmCraft's own decoders.
|
||||
pub fn register() -> Availability {
|
||||
#[cfg(target_os = "macos")]
|
||||
{
|
||||
filmcraft_codecs::register_video_decoder(videotoolbox_factory);
|
||||
Availability::Available("VideoToolbox")
|
||||
}
|
||||
#[cfg(not(target_os = "macos"))]
|
||||
{
|
||||
Availability::Unavailable("no hardware video decoder for this system yet")
|
||||
}
|
||||
}
|
||||
|
||||
/// Whether this system's hardware decoder takes the stream of `entry` (whatever the Hardware
|
||||
/// decoding setting says): diagnostics and tests.
|
||||
pub fn hardware_decoder_for(entry: &filmcraft_isobmff::SampleEntry) -> bool {
|
||||
#[cfg(target_os = "macos")]
|
||||
{
|
||||
filmcraft_codecs::hw::NalStreamInfo::from_entry(entry).and_then(|r| r.ok()).is_some_and(|info| videotoolbox::VtDecoder::new(info).is_ok())
|
||||
}
|
||||
#[cfg(not(target_os = "macos"))]
|
||||
{
|
||||
let _ = entry;
|
||||
false
|
||||
}
|
||||
}
|
||||
|
||||
/// The VideoToolbox factory: a [`HybridDecoder`] around [`videotoolbox::VtDecoder`] for `avcC` /
|
||||
/// `hvcC` streams VideoToolbox can decode in hardware, `None` otherwise.
|
||||
#[cfg(target_os = "macos")]
|
||||
pub fn videotoolbox_factory(entry: &filmcraft_isobmff::SampleEntry) -> Option<filmcraft_codecs::Result<Box<dyn filmcraft_codecs::VideoDecoder>>> {
|
||||
if !filmcraft_codecs::hw::hardware_decoding() {
|
||||
return None;
|
||||
}
|
||||
let info = filmcraft_codecs::hw::NalStreamInfo::from_entry(entry)?.ok()?;
|
||||
match videotoolbox::VtDecoder::new(info.clone()) {
|
||||
Ok(vt) => Some(Ok(Box::new(HybridDecoder::new(Box::new(vt), entry.clone(), info)))),
|
||||
Err(why) => {
|
||||
log::info!("hardware decoding declined for {} video: {why}", entry.codec.name());
|
||||
filmcraft_codecs::hw::note_hw_declined();
|
||||
None
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,569 @@
|
|||
//! VideoToolbox (macOS) hardware H.264 / HEVC decoding.
|
||||
//!
|
||||
//! The only module of the workspace with `unsafe` code besides the other FFI modules of this crate
|
||||
//! (docs/adr/0001-platform-ffi.md). Rules: every `unsafe` block has a `// SAFETY:` comment; no
|
||||
//! panic may unwind into VideoToolbox (the output callback runs under `catch_unwind`); every
|
||||
//! failure is a `Result` the caller ([`crate::hybrid::HybridDecoder`]) answers by switching to the
|
||||
//! software decoder.
|
||||
//!
|
||||
//! A session is created from the parameter sets of the sample entry (`avcC` / `hvcC`) with
|
||||
//! hardware decoding required, so streams the hardware cannot take are declined up front. Samples
|
||||
//! are fed as `CMSampleBuffer`s with asynchronous decompression (a couple in flight); the output
|
||||
//! callback (on a VideoToolbox thread) copies each decoded
|
||||
//! `CVPixelBuffer` (NV12 / P010-style biplanar) into planar [`PixelData::Yuv8`] /
|
||||
//! [`PixelData::Yuv16`]. VideoToolbox emits pictures in decoding order, so a reorder buffer of the
|
||||
//! stream's own depth (`max_num_reorder_frames` / `sps_max_num_reorder_pics`) puts them in
|
||||
//! presentation order, as the software decoders do.
|
||||
|
||||
use std::ffi::c_void;
|
||||
use std::panic::AssertUnwindSafe;
|
||||
use std::ptr::NonNull;
|
||||
use std::sync::{Arc, Mutex, PoisonError};
|
||||
|
||||
use filmcraft_codecs::hw::{NalCodec, NalStreamInfo};
|
||||
use filmcraft_codecs::{CodecError, DecodedFrame, Result, VideoDecoder};
|
||||
use filmcraft_frame::{Chroma, PixelData, VideoFrame};
|
||||
use objc2_core_foundation::{CFBoolean, CFDictionary, CFNumber, CFRetained, CFString, CFType};
|
||||
use objc2_core_media::{
|
||||
CMBlockBuffer, CMFormatDescription, CMSampleBuffer, CMSampleTimingInfo, CMTime, CMTimeFlags, CMVideoFormatDescriptionCreateFromH264ParameterSets,
|
||||
CMVideoFormatDescriptionCreateFromHEVCParameterSets, kCMBlockBufferAssureMemoryNowFlag,
|
||||
};
|
||||
use objc2_core_video::{
|
||||
CVImageBuffer, CVPixelBufferGetBaseAddressOfPlane, CVPixelBufferGetBytesPerRowOfPlane, CVPixelBufferGetHeight, CVPixelBufferGetHeightOfPlane,
|
||||
CVPixelBufferGetPixelFormatType, CVPixelBufferGetPlaneCount, CVPixelBufferGetWidth, CVPixelBufferGetWidthOfPlane, CVPixelBufferLockBaseAddress,
|
||||
CVPixelBufferLockFlags, CVPixelBufferUnlockBaseAddress, kCVPixelBufferPixelFormatTypeKey,
|
||||
};
|
||||
use objc2_video_toolbox::{
|
||||
VTDecodeFrameFlags, VTDecodeInfoFlags, VTDecompressionOutputCallbackRecord, VTDecompressionSession,
|
||||
kVTVideoDecoderSpecification_RequireHardwareAcceleratedVideoDecoder,
|
||||
};
|
||||
|
||||
/// `CVPixelBuffer` formats we ask VideoToolbox for: biplanar Y + interleaved CbCr, 8-bit or
|
||||
/// 10-bit in the high bits of 16-bit words.
|
||||
const NV12_VIDEO: u32 = u32::from_be_bytes(*b"420v");
|
||||
const NV12_FULL: u32 = u32::from_be_bytes(*b"420f");
|
||||
const P010_VIDEO: u32 = u32::from_be_bytes(*b"x420");
|
||||
const P010_FULL: u32 = u32::from_be_bytes(*b"xf20");
|
||||
const NV16_VIDEO: u32 = u32::from_be_bytes(*b"422v");
|
||||
const NV16_FULL: u32 = u32::from_be_bytes(*b"422f");
|
||||
const P210_VIDEO: u32 = u32::from_be_bytes(*b"x422");
|
||||
const P210_FULL: u32 = u32::from_be_bytes(*b"xf22");
|
||||
|
||||
/// How decoded buffers are laid out and what each output picture carries.
|
||||
#[derive(Clone, Debug)]
|
||||
struct Layout {
|
||||
pixel_format: u32,
|
||||
/// Coded size and output rectangle (luma samples).
|
||||
coded: (u32, u32),
|
||||
crop: (u32, u32, u32, u32),
|
||||
chroma: Chroma,
|
||||
/// Significant bits per sample (8 = `Yuv8`).
|
||||
bits: u32,
|
||||
color: filmcraft_color::ColorInfo,
|
||||
par: (u32, u32),
|
||||
}
|
||||
|
||||
/// One output-callback result.
|
||||
enum Output {
|
||||
Frame(DecodedFrame),
|
||||
Failed(String),
|
||||
}
|
||||
|
||||
/// State shared with the output callback (its `decompressionOutputRefCon`). Boxed by the decoder
|
||||
/// and kept alive until the session is invalidated.
|
||||
struct Shared {
|
||||
layout: Layout,
|
||||
out: Mutex<Vec<Output>>,
|
||||
}
|
||||
|
||||
impl Shared {
|
||||
fn push(&self, o: Output) {
|
||||
self.out.lock().unwrap_or_else(PoisonError::into_inner).push(o);
|
||||
}
|
||||
}
|
||||
|
||||
/// A VideoToolbox session with its callback state. Dropping it invalidates the session before the
|
||||
/// callback state is freed.
|
||||
struct Session {
|
||||
session: CFRetained<VTDecompressionSession>,
|
||||
format: CFRetained<CMFormatDescription>,
|
||||
/// Pointed to by the session's callback record: must outlive `session`.
|
||||
shared: Box<Shared>,
|
||||
}
|
||||
|
||||
// SAFETY: a VTDecompressionSession and the CoreMedia objects it uses may be used from any thread
|
||||
// (Apple: "thread safe"); the decoder owning a `Session` is used by one thread at a time (`&mut
|
||||
// self`), and the callback state is behind a `Mutex`.
|
||||
unsafe impl Send for Session {}
|
||||
|
||||
impl Drop for Session {
|
||||
fn drop(&mut self) {
|
||||
// SAFETY: the session is valid until released; invalidating it guarantees no callback runs
|
||||
// afterwards, so `shared` can be freed when this function returns.
|
||||
unsafe {
|
||||
self.session.wait_for_asynchronous_frames();
|
||||
self.session.invalidate();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The output callback. Never unwinds: panics are caught and reported as a failed frame.
|
||||
unsafe extern "C-unwind" fn output_callback(
|
||||
refcon: *mut c_void,
|
||||
frame_refcon: *mut c_void,
|
||||
status: i32,
|
||||
_flags: VTDecodeInfoFlags,
|
||||
image: *mut CVImageBuffer,
|
||||
_pts: CMTime,
|
||||
_duration: CMTime,
|
||||
) {
|
||||
if refcon.is_null() {
|
||||
return;
|
||||
}
|
||||
// SAFETY: `refcon` is the `Shared` the session was created with; it lives in a `Box` owned by
|
||||
// the `Session`, which invalidates the session (no more callbacks) before freeing it.
|
||||
let shared = unsafe { &*(refcon as *const Shared) };
|
||||
let pts = frame_refcon as isize as i64;
|
||||
let r = std::panic::catch_unwind(AssertUnwindSafe(|| {
|
||||
if status != 0 {
|
||||
return Output::Failed(format!("VideoToolbox decode error {status}"));
|
||||
}
|
||||
let Some(image) = NonNull::new(image) else {
|
||||
return Output::Failed("VideoToolbox dropped a picture".into());
|
||||
};
|
||||
// SAFETY: VideoToolbox passes a valid image buffer that stays alive for the duration of
|
||||
// the callback; we only borrow it here.
|
||||
let pb = unsafe { image.as_ref() };
|
||||
match copy_out(pb, &shared.layout) {
|
||||
Ok(frame) => Output::Frame(DecodedFrame { pts, frame, draft: false }),
|
||||
Err(e) => Output::Failed(e),
|
||||
}
|
||||
}));
|
||||
shared.push(r.unwrap_or_else(|_| Output::Failed("panic while reading a decoded picture".into())));
|
||||
}
|
||||
|
||||
/// Unlocks a pixel buffer's base address when dropped.
|
||||
struct Locked<'a>(&'a CVImageBuffer);
|
||||
|
||||
impl Drop for Locked<'_> {
|
||||
fn drop(&mut self) {
|
||||
// SAFETY: `copy_out` locked the buffer with the same flags, and it is still alive (borrowed).
|
||||
unsafe { CVPixelBufferUnlockBaseAddress(self.0, CVPixelBufferLockFlags::ReadOnly) };
|
||||
}
|
||||
}
|
||||
|
||||
/// One plane of a locked pixel buffer.
|
||||
struct PlaneView<'a> {
|
||||
data: &'a [u8],
|
||||
stride: usize,
|
||||
width: usize,
|
||||
height: usize,
|
||||
}
|
||||
|
||||
impl<'a> PlaneView<'a> {
|
||||
/// `len` bytes of row `y` from byte `x`.
|
||||
fn row(&self, y: usize, x: usize, len: usize) -> std::result::Result<&'a [u8], String> {
|
||||
let start = y * self.stride + x;
|
||||
self.data.get(start..start + len).ok_or_else(|| "plane row out of range".to_string())
|
||||
}
|
||||
}
|
||||
|
||||
fn plane<'a>(pb: &'a CVImageBuffer, _lock: &Locked<'a>, i: usize) -> std::result::Result<PlaneView<'a>, String> {
|
||||
let base = CVPixelBufferGetBaseAddressOfPlane(pb, i) as *const u8;
|
||||
let stride = CVPixelBufferGetBytesPerRowOfPlane(pb, i);
|
||||
let (width, height) = (CVPixelBufferGetWidthOfPlane(pb, i), CVPixelBufferGetHeightOfPlane(pb, i));
|
||||
if base.is_null() || stride == 0 || height == 0 {
|
||||
return Err(format!("decoded picture plane {i} is not mapped"));
|
||||
}
|
||||
let len = stride.checked_mul(height).ok_or("plane size overflows")?;
|
||||
// SAFETY: the buffer's base address is locked (`_lock` borrows the same buffer for 'a), and
|
||||
// CoreVideo maps `bytes_per_row * height` bytes from each plane's base address.
|
||||
let data = unsafe { std::slice::from_raw_parts(base, len) };
|
||||
Ok(PlaneView { data, stride, width, height })
|
||||
}
|
||||
|
||||
/// Copy a decoded biplanar buffer into a planar frame (deinterleaving chroma, cropping to the
|
||||
/// output rectangle, shifting 10-bit samples down from the high bits).
|
||||
fn copy_out(pb: &CVImageBuffer, l: &Layout) -> std::result::Result<VideoFrame, String> {
|
||||
let fmt = CVPixelBufferGetPixelFormatType(pb);
|
||||
if fmt != l.pixel_format || CVPixelBufferGetPlaneCount(pb) != 2 {
|
||||
return Err(format!("unexpected decoded pixel format {fmt:#x}"));
|
||||
}
|
||||
let (bw, bh) = (CVPixelBufferGetWidth(pb) as u32, CVPixelBufferGetHeight(pb) as u32);
|
||||
let (cx, cy, w, h) = l.crop;
|
||||
// VideoToolbox outputs the cropped picture; a buffer of the coded size is cropped here.
|
||||
let (ox, oy) = if (bw, bh) == (w, h) {
|
||||
(0, 0)
|
||||
} else if (bw, bh) == l.coded && cx + w <= bw && cy + h <= bh {
|
||||
(cx, cy)
|
||||
} else {
|
||||
return Err(format!("decoded picture is {bw}x{bh}, expected {w}x{h}"));
|
||||
};
|
||||
// SAFETY: `pb` is a valid pixel buffer for the duration of this call (see the callback).
|
||||
if unsafe { CVPixelBufferLockBaseAddress(pb, CVPixelBufferLockFlags::ReadOnly) } != 0 {
|
||||
return Err("cannot lock the decoded picture".into());
|
||||
}
|
||||
let lock = Locked(pb);
|
||||
let (ys, cs) = (plane(pb, &lock, 0)?, plane(pb, &lock, 1)?);
|
||||
let (w, h, ox, oy) = (w as usize, h as usize, ox as usize, oy as usize);
|
||||
let sub_y = if l.chroma == Chroma::C420 { 2 } else { 1 };
|
||||
let (cw, ch) = (w.div_ceil(2), h.div_ceil(sub_y));
|
||||
let (cox, coy) = (ox / 2, oy / sub_y);
|
||||
let bps = if l.bits > 8 { 2 } else { 1 };
|
||||
if ys.width < ox + w || ys.height < oy + h || ys.stride < ys.width * bps || cs.width < cox + cw || cs.height < coy + ch || cs.stride < cs.width * 2 * bps {
|
||||
return Err("decoded picture planes are smaller than the picture".into());
|
||||
}
|
||||
let data = if bps == 1 {
|
||||
let mut yp = Vec::with_capacity(w * h);
|
||||
for y in 0..h {
|
||||
yp.extend_from_slice(ys.row(oy + y, ox * bps, w * bps)?);
|
||||
}
|
||||
let (mut u, mut v) = (Vec::with_capacity(cw * ch), Vec::with_capacity(cw * ch));
|
||||
for y in 0..ch {
|
||||
for pair in cs.row(coy + y, cox * 2 * bps, cw * 2 * bps)?.as_chunks::<2>().0 {
|
||||
u.push(pair[0]);
|
||||
v.push(pair[1]);
|
||||
}
|
||||
}
|
||||
PixelData::Yuv8 { planes: [Arc::new(yp), Arc::new(u), Arc::new(v)], chroma: l.chroma, alpha: None }
|
||||
} else {
|
||||
let shift = 16 - l.bits;
|
||||
let mut yp = Vec::with_capacity(w * h);
|
||||
for y in 0..h {
|
||||
yp.extend(ys.row(oy + y, ox * bps, w * bps)?.as_chunks::<2>().0.iter().map(|b| u16::from_ne_bytes([b[0], b[1]]) >> shift));
|
||||
}
|
||||
let (mut u, mut v) = (Vec::with_capacity(cw * ch), Vec::with_capacity(cw * ch));
|
||||
for y in 0..ch {
|
||||
for q in cs.row(coy + y, cox * 2 * bps, cw * 2 * bps)?.as_chunks::<4>().0 {
|
||||
u.push(u16::from_ne_bytes([q[0], q[1]]) >> shift);
|
||||
v.push(u16::from_ne_bytes([q[2], q[3]]) >> shift);
|
||||
}
|
||||
}
|
||||
PixelData::Yuv16 { planes: [Arc::new(yp), Arc::new(u), Arc::new(v)], chroma: l.chroma, bits: l.bits, alpha: None }
|
||||
};
|
||||
drop(lock);
|
||||
Ok(VideoFrame { width: w as u32, height: h as u32, data, color: l.color, par: l.par, pts: filmcraft_time::Tick::ZERO })
|
||||
}
|
||||
|
||||
/// The layout for a stream, or why VideoToolbox is not used for it.
|
||||
fn layout(info: &NalStreamInfo) -> std::result::Result<Layout, String> {
|
||||
if info.interlaced {
|
||||
return Err("field-coded H.264".into());
|
||||
}
|
||||
if info.bit_depth_luma != info.bit_depth_chroma || !matches!(info.bit_depth_luma, 8 | 10) {
|
||||
return Err(format!("{}-bit luma / {}-bit chroma", info.bit_depth_luma, info.bit_depth_chroma));
|
||||
}
|
||||
let full = info.color.range == filmcraft_color::Range::Full;
|
||||
let ten = info.bit_depth_luma == 10;
|
||||
let (chroma, pixel_format) = match (info.chroma_format_idc, ten, full) {
|
||||
(1, false, false) => (Chroma::C420, NV12_VIDEO),
|
||||
(1, false, true) => (Chroma::C420, NV12_FULL),
|
||||
(1, true, false) => (Chroma::C420, P010_VIDEO),
|
||||
(1, true, true) => (Chroma::C420, P010_FULL),
|
||||
(2, false, false) => (Chroma::C422, NV16_VIDEO),
|
||||
(2, false, true) => (Chroma::C422, NV16_FULL),
|
||||
(2, true, false) => (Chroma::C422, P210_VIDEO),
|
||||
(2, true, true) => (Chroma::C422, P210_FULL),
|
||||
(c, ..) => return Err(format!("chroma_format_idc {c}")),
|
||||
};
|
||||
let (cx, cy, w, h) = info.crop;
|
||||
if w == 0 || h == 0 || w > 8192 || h > 8192 || cx.saturating_add(w) > info.coded.0 || cy.saturating_add(h) > info.coded.1 {
|
||||
return Err(format!("picture size {w}x{h}"));
|
||||
}
|
||||
Ok(Layout { pixel_format, coded: info.coded, crop: info.crop, chroma, bits: info.bit_depth_luma, color: info.color, par: info.par })
|
||||
}
|
||||
|
||||
impl Session {
|
||||
fn new(info: &NalStreamInfo, layout: Layout) -> std::result::Result<Session, String> {
|
||||
let format = format_description(info)?;
|
||||
// SAFETY: reading an immutable framework constant.
|
||||
let key: &CFString = unsafe { kVTVideoDecoderSpecification_RequireHardwareAcceleratedVideoDecoder };
|
||||
let spec = CFDictionary::<CFString, CFType>::from_slices(&[key], &[CFBoolean::new(true).as_ref()]);
|
||||
let pf = CFNumber::new_i32(layout.pixel_format as i32);
|
||||
// SAFETY: reading an immutable framework constant.
|
||||
let pf_key: &CFString = unsafe { kCVPixelBufferPixelFormatTypeKey };
|
||||
let attrs = CFDictionary::<CFString, CFType>::from_slices(&[pf_key], &[pf.as_ref()]);
|
||||
let shared = Box::new(Shared { layout, out: Mutex::new(Vec::new()) });
|
||||
let record = VTDecompressionOutputCallbackRecord {
|
||||
decompressionOutputCallback: Some(output_callback),
|
||||
decompressionOutputRefCon: &*shared as *const Shared as *mut c_void,
|
||||
};
|
||||
let mut out: *mut VTDecompressionSession = std::ptr::null_mut();
|
||||
// SAFETY: every pointer is valid for the call (VideoToolbox copies the callback record);
|
||||
// the refcon points into `shared`, which the returned `Session` keeps alive until the
|
||||
// session is invalidated.
|
||||
let status = unsafe { VTDecompressionSession::create(None, &format, Some(spec.as_ref()), Some(attrs.as_ref()), &record, NonNull::from(&mut out)) };
|
||||
let session = NonNull::new(out).filter(|_| status == 0).ok_or_else(|| format!("VTDecompressionSessionCreate failed ({status})"))?;
|
||||
// SAFETY: a created session is returned retained (+1); `CFRetained` takes that reference over.
|
||||
let session = unsafe { CFRetained::from_raw(session) };
|
||||
Ok(Session { session, format, shared })
|
||||
}
|
||||
|
||||
/// Submit one length-prefixed access unit (asynchronous decompression: its picture arrives
|
||||
/// through the callback, possibly after this returns).
|
||||
fn submit(&self, sample: &[u8], pts: i64) -> std::result::Result<(), String> {
|
||||
let buffer = sample_buffer(sample, pts, &self.format)?;
|
||||
let mut info = VTDecodeInfoFlags(0);
|
||||
// SAFETY: the session and sample buffer are valid (VideoToolbox retains the sample buffer
|
||||
// for as long as it needs it); the frame refcon is not a pointer, it carries the pts.
|
||||
let status =
|
||||
unsafe { self.session.decode_frame(&buffer, VTDecodeFrameFlags::Frame_EnableAsynchronousDecompression, pts as isize as *mut c_void, &mut info) };
|
||||
if status != 0 {
|
||||
return Err(format!("VTDecompressionSessionDecodeFrame failed ({status})"));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Wait until every submitted picture has been emitted.
|
||||
fn wait(&self) {
|
||||
// SAFETY: the session is valid.
|
||||
unsafe { self.session.wait_for_asynchronous_frames() };
|
||||
}
|
||||
|
||||
/// What the callback produced so far.
|
||||
fn take(&self) -> Vec<Output> {
|
||||
std::mem::take(&mut *self.shared.out.lock().unwrap_or_else(PoisonError::into_inner))
|
||||
}
|
||||
}
|
||||
|
||||
/// Access units decoding at once (asynchronous decompression overlaps decoding a picture with
|
||||
/// copying out the previous one).
|
||||
const MAX_IN_FLIGHT: usize = 2;
|
||||
|
||||
/// A `CMVideoFormatDescription` from the record's parameter sets.
|
||||
fn format_description(info: &NalStreamInfo) -> std::result::Result<CFRetained<CMFormatDescription>, String> {
|
||||
let mut ptrs: Vec<NonNull<u8>> = Vec::with_capacity(info.parameter_sets.len());
|
||||
let mut sizes: Vec<usize> = Vec::with_capacity(info.parameter_sets.len());
|
||||
for p in &info.parameter_sets {
|
||||
ptrs.push(NonNull::new(p.as_ptr() as *mut u8).ok_or("empty parameter set")?);
|
||||
sizes.push(p.len());
|
||||
}
|
||||
let ptr_array = NonNull::new(ptrs.as_mut_ptr()).ok_or("no parameter sets")?;
|
||||
let size_array = NonNull::new(sizes.as_mut_ptr()).ok_or("no parameter sets")?;
|
||||
let mut out: *const CMFormatDescription = std::ptr::null();
|
||||
let ls = info.length_size as i32;
|
||||
// SAFETY: `ptrs` / `sizes` hold `len` valid (pointer, size) pairs into `info.parameter_sets`,
|
||||
// which outlive the call; CoreMedia copies the parameter sets.
|
||||
let status = unsafe {
|
||||
match info.codec {
|
||||
NalCodec::H264 => CMVideoFormatDescriptionCreateFromH264ParameterSets(None, ptrs.len(), ptr_array, size_array, ls, NonNull::from(&mut out)),
|
||||
NalCodec::Hevc => CMVideoFormatDescriptionCreateFromHEVCParameterSets(None, ptrs.len(), ptr_array, size_array, ls, None, NonNull::from(&mut out)),
|
||||
}
|
||||
};
|
||||
let fd = NonNull::new(out as *mut CMFormatDescription).filter(|_| status == 0).ok_or_else(|| format!("bad parameter sets for VideoToolbox ({status})"))?;
|
||||
// SAFETY: a "Create" function returns the description retained (+1); `CFRetained` takes it over.
|
||||
Ok(unsafe { CFRetained::from_raw(fd) })
|
||||
}
|
||||
|
||||
/// A ready `CMSampleBuffer` holding a copy of `sample`.
|
||||
fn sample_buffer(sample: &[u8], pts: i64, format: &CMFormatDescription) -> std::result::Result<CFRetained<CMSampleBuffer>, String> {
|
||||
if sample.is_empty() {
|
||||
return Err("empty sample".into());
|
||||
}
|
||||
let mut block: *mut CMBlockBuffer = std::ptr::null_mut();
|
||||
// SAFETY: CoreMedia allocates (AssureMemoryNow) a block of `len` bytes with the default
|
||||
// allocator; no external memory is referenced.
|
||||
let status = unsafe {
|
||||
CMBlockBuffer::create_with_memory_block(
|
||||
None,
|
||||
std::ptr::null_mut(),
|
||||
sample.len(),
|
||||
None,
|
||||
std::ptr::null(),
|
||||
0,
|
||||
sample.len(),
|
||||
kCMBlockBufferAssureMemoryNowFlag,
|
||||
NonNull::from(&mut block),
|
||||
)
|
||||
};
|
||||
let block = NonNull::new(block).filter(|_| status == 0).ok_or_else(|| format!("CMBlockBufferCreateWithMemoryBlock failed ({status})"))?;
|
||||
// SAFETY: created retained (+1).
|
||||
let block = unsafe { CFRetained::from_raw(block) };
|
||||
let src = NonNull::new(sample.as_ptr() as *mut c_void).ok_or("empty sample")?;
|
||||
// SAFETY: copies `len` bytes from `sample` into the block, which holds exactly `len` bytes.
|
||||
let status = unsafe { CMBlockBuffer::replace_data_bytes(src, &block, 0, sample.len()) };
|
||||
if status != 0 {
|
||||
return Err(format!("CMBlockBufferReplaceDataBytes failed ({status})"));
|
||||
}
|
||||
let invalid = CMTime { value: 0, timescale: 0, flags: CMTimeFlags(0), epoch: 0 };
|
||||
let timing = CMSampleTimingInfo {
|
||||
duration: invalid,
|
||||
presentationTimeStamp: CMTime { value: pts, timescale: 1, flags: CMTimeFlags::Valid, epoch: 0 },
|
||||
decodeTimeStamp: invalid,
|
||||
};
|
||||
let size = sample.len();
|
||||
let mut out: *mut CMSampleBuffer = std::ptr::null_mut();
|
||||
// SAFETY: one sample with one timing entry and one size entry, all pointing at locals that
|
||||
// outlive the call; the sample buffer retains the block buffer and format description.
|
||||
let status = unsafe { CMSampleBuffer::create_ready(None, Some(&block), Some(format), 1, 1, &timing, 1, &size, NonNull::from(&mut out)) };
|
||||
let out = NonNull::new(out).filter(|_| status == 0).ok_or_else(|| format!("CMSampleBufferCreateReady failed ({status})"))?;
|
||||
// SAFETY: created retained (+1).
|
||||
Ok(unsafe { CFRetained::from_raw(out) })
|
||||
}
|
||||
|
||||
/// Whether VideoToolbox reports a hardware decoder for the codec at all.
|
||||
pub fn hardware_available(codec: NalCodec) -> bool {
|
||||
let t = match codec {
|
||||
NalCodec::H264 => u32::from_be_bytes(*b"avc1"),
|
||||
NalCodec::Hevc => u32::from_be_bytes(*b"hvc1"),
|
||||
};
|
||||
// SAFETY: a pure query with no pointer arguments.
|
||||
unsafe { objc2_video_toolbox::VTIsHardwareDecodeSupported(t) }
|
||||
}
|
||||
|
||||
/// The VideoToolbox decoder for one `avcC` / `hvcC` stream.
|
||||
pub struct VtDecoder {
|
||||
info: NalStreamInfo,
|
||||
layout: Layout,
|
||||
/// `None` after a reset until the next sample (a fresh session per decoding run).
|
||||
session: Option<Session>,
|
||||
/// Decoded pictures waiting for presentation order (sorted by pts).
|
||||
pending: Vec<DecodedFrame>,
|
||||
/// Presentation times of submitted access units whose picture has not arrived yet.
|
||||
in_flight: Vec<i64>,
|
||||
/// The run started at a CRA / BLA picture: its RASL pictures are not decodable and are left
|
||||
/// out (as the software decoder does).
|
||||
skip_rasl: bool,
|
||||
first: bool,
|
||||
/// Test hook: fail every decode once this many samples were fed.
|
||||
fail_after: Option<u64>,
|
||||
fed: u64,
|
||||
name: &'static str,
|
||||
}
|
||||
|
||||
impl VtDecoder {
|
||||
/// A decoder for the stream, or why VideoToolbox does not take it (unsupported format, no
|
||||
/// hardware decoder, session creation failed).
|
||||
pub fn new(info: NalStreamInfo) -> std::result::Result<Self, String> {
|
||||
let layout = layout(&info)?;
|
||||
if !hardware_available(info.codec) {
|
||||
return Err("no hardware decoder".into());
|
||||
}
|
||||
// Create the session now so unsupported streams are declined up front.
|
||||
let session = Session::new(&info, layout.clone())?;
|
||||
let name = match info.codec {
|
||||
NalCodec::H264 => "VideoToolbox H.264",
|
||||
NalCodec::Hevc => "VideoToolbox HEVC",
|
||||
};
|
||||
Ok(Self {
|
||||
info,
|
||||
layout,
|
||||
session: Some(session),
|
||||
pending: Vec::new(),
|
||||
in_flight: Vec::new(),
|
||||
skip_rasl: false,
|
||||
first: true,
|
||||
fail_after: None,
|
||||
fed: 0,
|
||||
name,
|
||||
})
|
||||
}
|
||||
|
||||
/// Test hook: from the `n`-th sample on, every decode fails as a lost session would.
|
||||
pub fn fail_after(&mut self, n: u64) {
|
||||
self.fail_after = Some(n);
|
||||
}
|
||||
|
||||
fn insert(&mut self, f: DecodedFrame) {
|
||||
let at = self.pending.partition_point(|p| p.pts <= f.pts);
|
||||
self.pending.insert(at, f);
|
||||
}
|
||||
|
||||
/// Pictures that are due: everything beyond the reorder depth, smallest pts first.
|
||||
fn bump(&mut self) -> Vec<DecodedFrame> {
|
||||
let mut out = Vec::new();
|
||||
// A picture still decoding counts towards the depth and may come before the waiting ones.
|
||||
while self.pending.len() + self.in_flight.len() > self.info.reorder {
|
||||
let first = self.pending.first().map(|f| f.pts);
|
||||
match (first, self.in_flight.iter().min()) {
|
||||
(Some(p), Some(&q)) if q < p => break,
|
||||
(Some(_), _) => out.push(self.pending.remove(0)),
|
||||
(None, _) => break,
|
||||
}
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// Move the callback's pictures into the reorder buffer.
|
||||
fn collect(&mut self) -> std::result::Result<(), String> {
|
||||
let Some(session) = self.session.as_ref() else { return Ok(()) };
|
||||
for o in session.take() {
|
||||
match o {
|
||||
Output::Frame(f) => {
|
||||
if let Some(i) = self.in_flight.iter().position(|&p| p == f.pts) {
|
||||
self.in_flight.swap_remove(i);
|
||||
}
|
||||
self.insert(f);
|
||||
}
|
||||
Output::Failed(e) => return Err(e),
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
impl VideoDecoder for VtDecoder {
|
||||
fn decode(&mut self, sample: &[u8], pts: i64) -> Result<Vec<DecodedFrame>> {
|
||||
self.fed += 1;
|
||||
if self.fail_after.is_some_and(|n| self.fed > n) {
|
||||
return Err(CodecError::Decode("VideoToolbox session failed (test hook)".into()));
|
||||
}
|
||||
if self.info.codec == NalCodec::Hevc {
|
||||
let types = self.info.nal_types(sample);
|
||||
let irap = types.iter().any(|t| (16..=23).contains(t));
|
||||
if irap {
|
||||
// CRA / BLA starting a run: its RASL pictures reference pictures we never decoded.
|
||||
self.skip_rasl = self.first && types.iter().any(|t| (16..=21).contains(t) && !(19..=20).contains(t));
|
||||
} else if self.skip_rasl && types.iter().any(|t| matches!(t, 8 | 9)) {
|
||||
self.first = false;
|
||||
return Ok(Vec::new());
|
||||
}
|
||||
}
|
||||
self.first = false;
|
||||
if self.session.is_none() {
|
||||
self.session = Some(Session::new(&self.info, self.layout.clone()).map_err(CodecError::Decode)?);
|
||||
}
|
||||
let Some(session) = self.session.as_ref() else {
|
||||
return Err(CodecError::Decode("no VideoToolbox session".into()));
|
||||
};
|
||||
session.submit(sample, pts).map_err(CodecError::Decode)?;
|
||||
self.in_flight.push(pts);
|
||||
if self.in_flight.len() > MAX_IN_FLIGHT {
|
||||
session.wait();
|
||||
}
|
||||
self.collect().map_err(CodecError::Decode)?;
|
||||
Ok(self.bump())
|
||||
}
|
||||
|
||||
fn flush(&mut self) -> Vec<DecodedFrame> {
|
||||
if let Some(s) = self.session.as_ref() {
|
||||
s.wait();
|
||||
}
|
||||
if let Err(e) = self.collect() {
|
||||
log::warn!("{}: {e} while flushing", self.name);
|
||||
}
|
||||
self.in_flight.clear();
|
||||
std::mem::take(&mut self.pending)
|
||||
}
|
||||
|
||||
fn reset(&mut self) {
|
||||
// A fresh session per run (made on the next sample): no reference pictures survive a seek.
|
||||
self.session = None;
|
||||
self.pending.clear();
|
||||
self.in_flight.clear();
|
||||
self.skip_rasl = false;
|
||||
self.first = true;
|
||||
}
|
||||
|
||||
fn name(&self) -> &str {
|
||||
self.name
|
||||
}
|
||||
|
||||
fn is_random_access(&self, sample: &[u8]) -> Option<bool> {
|
||||
self.info.is_random_access(sample)
|
||||
}
|
||||
|
||||
fn is_disposable(&self, sample: &[u8]) -> bool {
|
||||
self.info.is_disposable(sample)
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,174 @@
|
|||
//! Shared helpers for the platform tests: ffmpeg-made fixtures (external generator only, never
|
||||
//! linked), MP4 sample reading, and decoding a run of samples to compare decoders.
|
||||
#![allow(dead_code)]
|
||||
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::process::{Command, Stdio};
|
||||
|
||||
use filmcraft_codecs::{DecodedFrame, VideoDecoder};
|
||||
use filmcraft_frame::PixelData;
|
||||
use filmcraft_isobmff::SampleEntry;
|
||||
|
||||
/// The parity fixtures: (file, ffmpeg arguments). B-frames everywhere (reordering), a 360-line
|
||||
/// picture (coded 368: cropping), two GOPs or more (seeks), and an open-GOP HEVC stream (CRA with
|
||||
/// RASL pictures).
|
||||
pub const FIXTURES: &[(&str, &[&str])] = &[
|
||||
(
|
||||
"h264_high.mp4",
|
||||
&[
|
||||
"-f",
|
||||
"lavfi",
|
||||
"-i",
|
||||
"testsrc2=s=640x360:r=24:d=3,noise=alls=12:allf=t",
|
||||
"-c:v",
|
||||
"libx264",
|
||||
"-profile:v",
|
||||
"high",
|
||||
"-preset",
|
||||
"fast",
|
||||
"-x264-params",
|
||||
"bframes=3:b-pyramid=normal:keyint=24:min-keyint=24:scenecut=0",
|
||||
"-pix_fmt",
|
||||
"yuv420p",
|
||||
],
|
||||
),
|
||||
(
|
||||
"hevc_main.mp4",
|
||||
&[
|
||||
"-f",
|
||||
"lavfi",
|
||||
"-i",
|
||||
"testsrc2=s=640x360:r=24:d=3,noise=alls=12:allf=t",
|
||||
"-c:v",
|
||||
"libx265",
|
||||
"-preset",
|
||||
"fast",
|
||||
"-tag:v",
|
||||
"hvc1",
|
||||
"-x265-params",
|
||||
"log-level=error:bframes=4:keyint=24:min-keyint=24:scenecut=0:open-gop=1",
|
||||
"-pix_fmt",
|
||||
"yuv420p",
|
||||
],
|
||||
),
|
||||
(
|
||||
"hevc_main10.mp4",
|
||||
&[
|
||||
"-f",
|
||||
"lavfi",
|
||||
"-i",
|
||||
"testsrc2=s=640x360:r=24:d=3,noise=alls=12:allf=t",
|
||||
"-c:v",
|
||||
"libx265",
|
||||
"-preset",
|
||||
"fast",
|
||||
"-profile:v",
|
||||
"main10",
|
||||
"-tag:v",
|
||||
"hvc1",
|
||||
"-x265-params",
|
||||
"log-level=error:bframes=4:keyint=24:min-keyint=24:scenecut=0",
|
||||
"-pix_fmt",
|
||||
"yuv420p10le",
|
||||
],
|
||||
),
|
||||
];
|
||||
|
||||
pub fn dir() -> PathBuf {
|
||||
filmcraft_testkit::fixtures_dir("platform")
|
||||
}
|
||||
|
||||
/// Generate `name` with `ffmpeg -y -v error <args> <out>` (cached).
|
||||
pub fn fixture(ff: &Path, name: &str, args: &[&str]) -> Option<PathBuf> {
|
||||
let out = dir().join(name);
|
||||
filmcraft_testkit::fixtures::generate(&out, |tmp| {
|
||||
let st = Command::new(ff).args(["-y", "-v", "error"]).args(args).arg(tmp).stdin(Stdio::null()).status();
|
||||
match st {
|
||||
Ok(s) if s.success() => true,
|
||||
other => {
|
||||
eprintln!("fixture {name}: ffmpeg failed: {other:?}");
|
||||
false
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
/// The fixture by name.
|
||||
pub fn named(ff: &Path, name: &str) -> Option<PathBuf> {
|
||||
let (_, args) = FIXTURES.iter().find(|f| f.0 == name)?;
|
||||
fixture(ff, name, args)
|
||||
}
|
||||
|
||||
/// A video track's sample entry, samples (decode order) with pts, and sync flags.
|
||||
pub struct Stream {
|
||||
pub entry: SampleEntry,
|
||||
pub samples: Vec<(Vec<u8>, i64)>,
|
||||
pub sync: Vec<bool>,
|
||||
}
|
||||
|
||||
pub fn read_stream(path: &Path) -> Stream {
|
||||
let bytes = std::fs::read(path).unwrap();
|
||||
let file = filmcraft_isobmff::open(bytes.clone()).unwrap();
|
||||
let t = file.track_of_kind(filmcraft_isobmff::TrackKind::Video).unwrap();
|
||||
let track = &file.tracks[t];
|
||||
let samples = (0..track.samples.len()).map(|i| (file.read_sample(&bytes, t, i).unwrap(), track.samples[i].pts)).collect();
|
||||
Stream { entry: track.entries[0].clone(), samples, sync: track.samples.iter().map(|s| s.is_sync).collect() }
|
||||
}
|
||||
|
||||
/// Feed `samples` then flush.
|
||||
pub fn decode_all(d: &mut dyn VideoDecoder, samples: &[(Vec<u8>, i64)]) -> Vec<DecodedFrame> {
|
||||
let mut out = Vec::new();
|
||||
for (s, p) in samples {
|
||||
out.extend(d.decode(s, *p).unwrap_or_else(|e| panic!("{}: {e}", d.name())));
|
||||
}
|
||||
out.extend(d.flush());
|
||||
out
|
||||
}
|
||||
|
||||
/// Assert two decoders' outputs are the same pictures (pts order, size, format, colour, aspect
|
||||
/// and every sample).
|
||||
pub fn assert_same(what: &str, a: &[DecodedFrame], b: &[DecodedFrame]) {
|
||||
let pa: Vec<i64> = a.iter().map(|f| f.pts).collect();
|
||||
let pb: Vec<i64> = b.iter().map(|f| f.pts).collect();
|
||||
assert_eq!(pa, pb, "{what}: pts sequence");
|
||||
for (x, y) in a.iter().zip(b) {
|
||||
let (fx, fy) = (&x.frame, &y.frame);
|
||||
assert_eq!((fx.width, fx.height, fx.par, fx.color), (fy.width, fy.height, fy.par, fy.color), "{what}: pts {} geometry / colour", x.pts);
|
||||
match (&fx.data, &fy.data) {
|
||||
(PixelData::Yuv8 { planes: p, chroma: c, .. }, PixelData::Yuv8 { planes: q, chroma: d, .. }) => {
|
||||
assert_eq!(c, d, "{what}: chroma");
|
||||
for i in 0..3 {
|
||||
assert!(p[i] == q[i], "{what}: pts {} plane {i} differs (max diff {})", x.pts, max_diff8(&p[i], &q[i]));
|
||||
}
|
||||
}
|
||||
(PixelData::Yuv16 { planes: p, chroma: c, bits: bx, .. }, PixelData::Yuv16 { planes: q, chroma: d, bits: by, .. }) => {
|
||||
assert_eq!((c, bx), (d, by), "{what}: chroma / bits");
|
||||
for i in 0..3 {
|
||||
assert!(p[i] == q[i], "{what}: pts {} plane {i} differs (max diff {})", x.pts, max_diff16(&p[i], &q[i]));
|
||||
}
|
||||
}
|
||||
_ => panic!("{what}: pts {}: pixel formats differ", x.pts),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn max_diff8(a: &[u8], b: &[u8]) -> String {
|
||||
if a.len() != b.len() {
|
||||
return format!("sizes {} vs {}", a.len(), b.len());
|
||||
}
|
||||
a.iter().zip(b).map(|(x, y)| x.abs_diff(*y)).max().unwrap_or(0).to_string()
|
||||
}
|
||||
|
||||
fn max_diff16(a: &[u16], b: &[u16]) -> String {
|
||||
if a.len() != b.len() {
|
||||
return format!("sizes {} vs {}", a.len(), b.len());
|
||||
}
|
||||
a.iter().zip(b).map(|(x, y)| x.abs_diff(*y)).max().unwrap_or(0).to_string()
|
||||
}
|
||||
|
||||
pub fn xorshift(seed: &mut u64) -> u64 {
|
||||
*seed ^= *seed << 13;
|
||||
*seed ^= *seed >> 7;
|
||||
*seed ^= *seed << 17;
|
||||
*seed
|
||||
}
|
||||
|
|
@ -0,0 +1,107 @@
|
|||
//! [`HybridDecoder`] fallback on every platform, with a stand-in "hardware" decoder (our software
|
||||
//! decoder that starts failing after N samples, as a lost hardware session does): the output must
|
||||
//! be exactly the software decoder's, whatever sample the failure hits. Also: in-band parameter
|
||||
//! sets that differ from the sample entry's switch to software; identical ones do not.
|
||||
|
||||
mod common;
|
||||
|
||||
use common::*;
|
||||
use filmcraft_codecs::hw::NalStreamInfo;
|
||||
use filmcraft_codecs::{DecodedFrame, Result, VideoDecoder};
|
||||
use filmcraft_platform::HybridDecoder;
|
||||
|
||||
/// Decodes like the software decoder until `fail_after` samples were fed, then fails every call
|
||||
/// (its pending pictures stay retrievable through `flush`, as with a real session).
|
||||
struct Failing {
|
||||
inner: Box<dyn VideoDecoder>,
|
||||
fail_after: usize,
|
||||
fed: usize,
|
||||
}
|
||||
|
||||
impl VideoDecoder for Failing {
|
||||
fn decode(&mut self, sample: &[u8], pts: i64) -> Result<Vec<DecodedFrame>> {
|
||||
self.fed += 1;
|
||||
if self.fed > self.fail_after {
|
||||
return Err(filmcraft_codecs::CodecError::Decode("session invalidated (test)".into()));
|
||||
}
|
||||
self.inner.decode(sample, pts)
|
||||
}
|
||||
fn flush(&mut self) -> Vec<DecodedFrame> {
|
||||
self.inner.flush()
|
||||
}
|
||||
fn reset(&mut self) {
|
||||
self.inner.reset();
|
||||
}
|
||||
fn name(&self) -> &str {
|
||||
"stand-in hardware"
|
||||
}
|
||||
}
|
||||
|
||||
fn hybrid(s: &Stream, fail_after: usize) -> HybridDecoder {
|
||||
let info = NalStreamInfo::from_entry(&s.entry).unwrap().unwrap();
|
||||
let inner = filmcraft_codecs::software_video_decoder(&s.entry).unwrap();
|
||||
HybridDecoder::new(Box::new(Failing { inner, fail_after, fed: 0 }), s.entry.clone(), info)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn failures_anywhere_continue_with_the_software_output() {
|
||||
let ff = filmcraft_testkit::require_ffmpeg!();
|
||||
for (name, _) in FIXTURES {
|
||||
let Some(path) = named(&ff, name) else { continue };
|
||||
let s = read_stream(&path);
|
||||
let reference = decode_all(filmcraft_codecs::software_video_decoder(&s.entry).unwrap().as_mut(), &s.samples);
|
||||
let syncs: Vec<usize> = (0..s.samples.len()).filter(|&i| s.sync[i]).collect();
|
||||
let mut points: Vec<usize> = vec![0, 1, 3, 7, s.samples.len() - 1, s.samples.len()];
|
||||
for &k in &syncs {
|
||||
points.extend([k, k + 1, k + 2, k + 5]);
|
||||
}
|
||||
for fail_after in points {
|
||||
let mut d = hybrid(&s, fail_after);
|
||||
let out = decode_all(&mut d, &s.samples);
|
||||
assert_eq!(d.is_hardware(), fail_after >= s.samples.len(), "{name}: fell back at {fail_after}");
|
||||
assert_same(&format!("{name} failing after {fail_after} samples"), &out, &reference);
|
||||
}
|
||||
// a failure after a seek (reset) into the second GOP
|
||||
if let Some(&k) = syncs.get(1) {
|
||||
let tail = &s.samples[k..];
|
||||
let expect = {
|
||||
let mut sw = filmcraft_codecs::software_video_decoder(&s.entry).unwrap();
|
||||
decode_all(sw.as_mut(), tail)
|
||||
};
|
||||
let mut d = hybrid(&s, k + 4);
|
||||
decode_all(&mut d, &s.samples[..k]);
|
||||
d.reset();
|
||||
assert_same(&format!("{name} failing after a seek"), &decode_all(&mut d, tail), &expect);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn changed_in_band_parameter_sets_switch_to_software() {
|
||||
let ff = filmcraft_testkit::require_ffmpeg!();
|
||||
let Some(path) = named(&ff, "h264_high.mp4") else { return };
|
||||
let s = read_stream(&path);
|
||||
let info = NalStreamInfo::from_entry(&s.entry).unwrap().unwrap();
|
||||
let sps = info.parameter_sets[0].clone();
|
||||
let with_sps = |sps: &[u8], sample: &[u8]| {
|
||||
let mut v = (sps.len() as u32).to_be_bytes().to_vec();
|
||||
v.extend_from_slice(sps);
|
||||
v.extend_from_slice(sample);
|
||||
v
|
||||
};
|
||||
// the same SPS repeated in band at every sync sample: stays in "hardware"
|
||||
let same: Vec<_> = s.samples.iter().zip(&s.sync).map(|((x, p), &k)| (if k { with_sps(&sps, x) } else { x.clone() }, *p)).collect();
|
||||
let mut d = hybrid(&s, usize::MAX);
|
||||
let out = decode_all(&mut d, &same);
|
||||
assert!(d.is_hardware());
|
||||
assert_same("repeated SPS", &out, &decode_all(filmcraft_codecs::software_video_decoder(&s.entry).unwrap().as_mut(), &same));
|
||||
// a different SPS (same values, trailing zero byte) at the second sync sample: software
|
||||
let mut changed_sps = sps.clone();
|
||||
changed_sps.push(0);
|
||||
let k = (1..s.samples.len()).find(|&i| s.sync[i]).unwrap();
|
||||
let changed: Vec<_> = s.samples.iter().enumerate().map(|(i, (x, p))| (if i == k { with_sps(&changed_sps, x) } else { x.clone() }, *p)).collect();
|
||||
let mut d = hybrid(&s, usize::MAX);
|
||||
let out = decode_all(&mut d, &changed);
|
||||
assert!(!d.is_hardware(), "changed SPS: software");
|
||||
assert_same("changed SPS", &out, &decode_all(filmcraft_codecs::software_video_decoder(&s.entry).unwrap().as_mut(), &changed));
|
||||
}
|
||||
|
|
@ -0,0 +1,47 @@
|
|||
//! Settings ▸ Playback ▸ Hardware decoding through the decoder registry and the media stack:
|
||||
//! Off gives the software decoder (and no hardware frames), Auto the hardware one where the
|
||||
//! system has it. Its own test binary: the switch is process-wide.
|
||||
|
||||
mod common;
|
||||
|
||||
use common::*;
|
||||
use filmcraft_media::FrameRequest;
|
||||
|
||||
#[test]
|
||||
fn off_uses_the_software_decoder() {
|
||||
let ff = filmcraft_testkit::require_ffmpeg!();
|
||||
let Some(path) = named(&ff, "hevc_main.mp4") else { return };
|
||||
let s = read_stream(&path);
|
||||
let available = filmcraft_platform::register();
|
||||
assert_eq!(filmcraft_platform::register(), available, "registering twice is harmless");
|
||||
|
||||
filmcraft_codecs::hw::set_hardware_decoding(false);
|
||||
let d = filmcraft_codecs::make_video_decoder(&s.entry).unwrap();
|
||||
assert_eq!(d.name(), "FilmCraft HEVC");
|
||||
let bytes: std::sync::Arc<[u8]> = std::fs::read(&path).unwrap().into();
|
||||
let (g0, h0) = (filmcraft_codecs::gop_stats(), filmcraft_codecs::hw::hw_stats());
|
||||
let src = filmcraft_codecs::open_bytes("hevc_main.mp4", bytes.clone()).unwrap();
|
||||
let rate = src.info().frame_rate();
|
||||
for i in 0..30 {
|
||||
src.video_frame(FrameRequest::full(rate.tick_of(i))).unwrap();
|
||||
}
|
||||
assert!(filmcraft_codecs::gop_stats().frames > g0.frames, "frames counted");
|
||||
assert_eq!(filmcraft_codecs::hw::hw_stats().frames, h0.frames, "Off: no hardware frames");
|
||||
|
||||
filmcraft_codecs::hw::set_hardware_decoding(true);
|
||||
let d = filmcraft_codecs::make_video_decoder(&s.entry).unwrap();
|
||||
let hw = filmcraft_platform::hardware_decoder_for(&s.entry);
|
||||
if !hw {
|
||||
eprintln!("SKIPPED (Auto half): no hardware decoder ({available:?})");
|
||||
assert_eq!(d.name(), "FilmCraft HEVC");
|
||||
return;
|
||||
}
|
||||
assert_eq!(d.name(), "VideoToolbox HEVC");
|
||||
let h1 = filmcraft_codecs::hw::hw_stats();
|
||||
let src = filmcraft_codecs::open_bytes("hevc_main.mp4", bytes).unwrap();
|
||||
for i in 0..30 {
|
||||
src.video_frame(FrameRequest::full(rate.tick_of(i))).unwrap();
|
||||
}
|
||||
let h2 = filmcraft_codecs::hw::hw_stats();
|
||||
assert!(h2.frames > h1.frames && h2.sessions > h1.sessions, "Auto: hardware frames {h1:?} → {h2:?}");
|
||||
}
|
||||
|
|
@ -0,0 +1,232 @@
|
|||
//! VideoToolbox decoding against our software decoders (macOS): every picture of H.264 High,
|
||||
//! HEVC Main and HEVC Main 10 fixtures with B-frames must be identical (bit-exact planes, colour,
|
||||
//! pixel aspect, pts and presentation order), also after `reset` + reseek and a mid-stream
|
||||
//! `flush`; a forced mid-stream failure must continue in software with the software decoder's
|
||||
//! exact output; damaged samples and parameter sets must give errors or fall back, never crash or
|
||||
//! hang. Skips without ffmpeg (fixture generator) or without a hardware decoder.
|
||||
#![cfg(target_os = "macos")]
|
||||
|
||||
mod common;
|
||||
|
||||
use std::time::Duration;
|
||||
|
||||
use common::*;
|
||||
use filmcraft_codecs::VideoDecoder;
|
||||
use filmcraft_codecs::hw::NalStreamInfo;
|
||||
use filmcraft_platform::HybridDecoder;
|
||||
use filmcraft_platform::videotoolbox::VtDecoder;
|
||||
|
||||
/// The hardware decoder for a stream, or `None` (skip) when this machine has none for it.
|
||||
fn hardware(s: &Stream) -> Option<Box<dyn VideoDecoder>> {
|
||||
match filmcraft_platform::videotoolbox_factory(&s.entry) {
|
||||
Some(Ok(d)) => {
|
||||
assert!(d.name().starts_with("VideoToolbox"), "{}", d.name());
|
||||
Some(d)
|
||||
}
|
||||
Some(Err(e)) => panic!("factory error: {e}"),
|
||||
None => {
|
||||
eprintln!("SKIPPED: no VideoToolbox hardware decoder for {}", s.entry.codec.name());
|
||||
None
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn software(s: &Stream) -> Box<dyn VideoDecoder> {
|
||||
filmcraft_codecs::software_video_decoder(&s.entry).unwrap()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn bit_exact_with_the_software_decoders() {
|
||||
let ff = filmcraft_testkit::require_ffmpeg!();
|
||||
for (name, _) in FIXTURES {
|
||||
let Some(path) = named(&ff, name) else { continue };
|
||||
let s = read_stream(&path);
|
||||
let Some(mut hw) = hardware(&s) else { continue };
|
||||
let mut sw = software(&s);
|
||||
let a = decode_all(hw.as_mut(), &s.samples);
|
||||
let b = decode_all(sw.as_mut(), &s.samples);
|
||||
assert!(a.len() >= 60, "{name}: {} pictures", a.len());
|
||||
assert_same(name, &a, &b);
|
||||
assert!(hw.name().starts_with("VideoToolbox"), "{name}: stayed in hardware ({})", hw.name());
|
||||
|
||||
// reset + reseek to each later sync sample, decode a stretch, flush mid-stream
|
||||
let syncs: Vec<usize> = (1..s.samples.len()).filter(|&i| s.sync[i]).collect();
|
||||
assert!(!syncs.is_empty(), "{name}: more than one GOP");
|
||||
for &k in syncs.iter().rev() {
|
||||
let end = (k + 17).min(s.samples.len());
|
||||
for d in [&mut hw, &mut sw] {
|
||||
d.reset();
|
||||
}
|
||||
let a = decode_all(hw.as_mut(), &s.samples[k..end]);
|
||||
let b = decode_all(sw.as_mut(), &s.samples[k..end]);
|
||||
assert!(!a.is_empty(), "{name}: pictures after seeking to {k}");
|
||||
assert_same(&format!("{name} from sample {k}"), &a, &b);
|
||||
}
|
||||
// back to the start after seeking around: the whole stream again
|
||||
hw.reset();
|
||||
sw.reset();
|
||||
assert_same(&format!("{name} after resets"), &decode_all(hw.as_mut(), &s.samples), &decode_all(sw.as_mut(), &s.samples));
|
||||
// random-access / disposable answers match the software decoder's
|
||||
for (smp, _) in &s.samples {
|
||||
assert_eq!(hw.is_random_access(smp), sw.is_random_access(smp), "{name}");
|
||||
assert_eq!(hw.is_disposable(smp), sw.is_disposable(smp), "{name}");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn mid_stream_failure_continues_in_software() {
|
||||
let ff = filmcraft_testkit::require_ffmpeg!();
|
||||
for (name, _) in FIXTURES {
|
||||
let Some(path) = named(&ff, name) else { continue };
|
||||
let s = read_stream(&path);
|
||||
let info = NalStreamInfo::from_entry(&s.entry).unwrap().unwrap();
|
||||
let Ok(_) = VtDecoder::new(info.clone()) else {
|
||||
eprintln!("SKIPPED: no VideoToolbox hardware decoder for {name}");
|
||||
continue;
|
||||
};
|
||||
let reference = decode_all(software(&s).as_mut(), &s.samples);
|
||||
// fail at the first sample, inside the first GOP, at a sync sample and just after one
|
||||
let k = (1..s.samples.len()).find(|&i| s.sync[i]).unwrap();
|
||||
for fail_at in [0, 5, k, k + 1, k + 9] {
|
||||
let mut vt = VtDecoder::new(info.clone()).unwrap();
|
||||
vt.fail_after(fail_at as u64);
|
||||
let mut d = HybridDecoder::new(Box::new(vt), s.entry.clone(), info.clone());
|
||||
let before = filmcraft_codecs::hw::hw_stats().fallbacks;
|
||||
let out = decode_all(&mut d, &s.samples);
|
||||
assert!(!d.is_hardware(), "{name}: switched to software");
|
||||
assert!(filmcraft_codecs::hw::hw_stats().fallbacks > before, "{name}: fallback counted");
|
||||
assert_same(&format!("{name} failing at sample {fail_at}"), &out, &reference);
|
||||
// the instance stays in software across seeks
|
||||
d.reset();
|
||||
assert_same(&format!("{name} after the fallback and a reset"), &decode_all(&mut d, &s.samples), &reference);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Run `f` on a thread; fail if it panics or takes longer than `limit`.
|
||||
fn bounded(what: &str, limit: Duration, f: impl FnOnce() + Send + 'static) {
|
||||
let (tx, rx) = std::sync::mpsc::channel();
|
||||
std::thread::spawn(move || {
|
||||
let r = std::panic::catch_unwind(std::panic::AssertUnwindSafe(f));
|
||||
let _ = tx.send(r.is_ok());
|
||||
});
|
||||
match rx.recv_timeout(limit) {
|
||||
Ok(true) => {}
|
||||
Ok(false) => panic!("{what}: panicked"),
|
||||
Err(_) => panic!("{what}: hung"),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn damaged_samples_and_parameter_sets_never_crash_or_hang() {
|
||||
let ff = filmcraft_testkit::require_ffmpeg!();
|
||||
for (name, _) in FIXTURES {
|
||||
let Some(path) = named(&ff, name) else { continue };
|
||||
let s = read_stream(&path);
|
||||
if hardware(&s).is_none() {
|
||||
continue;
|
||||
}
|
||||
// damaged samples (bit flips, truncation, corrupt length prefixes) through the hybrid
|
||||
for seed in 1..=6u64 {
|
||||
let samples = s.samples.clone();
|
||||
let entry = s.entry.clone();
|
||||
bounded(&format!("{name} seed {seed}"), Duration::from_secs(60), move || {
|
||||
let mut seed = seed.wrapping_mul(0x9E37_79B9_7F4A_7C15) | 1;
|
||||
let mut d = filmcraft_platform::videotoolbox_factory(&entry).unwrap().unwrap();
|
||||
for (i, (smp, pts)) in samples.iter().enumerate().take(40) {
|
||||
let mut m = smp.clone();
|
||||
match xorshift(&mut seed) % 4 {
|
||||
0 => {
|
||||
for _ in 0..1 + xorshift(&mut seed) % 8 {
|
||||
let at = (xorshift(&mut seed) as usize) % m.len().max(1);
|
||||
if let Some(b) = m.get_mut(at) {
|
||||
*b ^= 1 << (xorshift(&mut seed) % 8);
|
||||
}
|
||||
}
|
||||
}
|
||||
1 => m.truncate((xorshift(&mut seed) as usize) % m.len().max(1)),
|
||||
2 if m.len() > 4 => m[..4].copy_from_slice(&(xorshift(&mut seed) as u32).to_be_bytes()),
|
||||
_ => {}
|
||||
}
|
||||
let _ = d.decode(&m, *pts);
|
||||
if i % 13 == 12 {
|
||||
d.reset();
|
||||
}
|
||||
}
|
||||
let _ = d.flush();
|
||||
});
|
||||
}
|
||||
// damaged parameter sets: the factory declines or the decoder errors / falls back
|
||||
let rec = match &s.entry.codec {
|
||||
filmcraft_isobmff::CodecConfig::Avc(a) => a.to_bytes(),
|
||||
filmcraft_isobmff::CodecConfig::Hevc(c) => c.to_bytes(),
|
||||
_ => continue,
|
||||
};
|
||||
let hevc = matches!(s.entry.codec, filmcraft_isobmff::CodecConfig::Hevc(_));
|
||||
let mut seed = 0x1234_5678u64;
|
||||
for round in 0..24 {
|
||||
let mut r = rec.clone();
|
||||
if round % 3 == 2 {
|
||||
r.truncate(r.len() * (round + 1) / 30);
|
||||
} else {
|
||||
for _ in 0..1 + round % 4 {
|
||||
let at = 6 + (xorshift(&mut seed) as usize) % r.len().saturating_sub(6).max(1);
|
||||
if let Some(b) = r.get_mut(at) {
|
||||
*b ^= 1 << (xorshift(&mut seed) % 8);
|
||||
}
|
||||
}
|
||||
}
|
||||
let samples: Vec<_> = s.samples.iter().take(12).cloned().collect();
|
||||
bounded(&format!("{name} record round {round}"), Duration::from_secs(60), move || {
|
||||
let Some(info) = (if hevc { NalStreamInfo::from_hvcc(&r) } else { NalStreamInfo::from_avcc(&r) }).ok() else { return };
|
||||
let Ok(mut vt) = VtDecoder::new(info) else { return };
|
||||
for (smp, pts) in &samples {
|
||||
if vt.decode(smp, *pts).is_err() {
|
||||
break;
|
||||
}
|
||||
}
|
||||
let _ = vt.flush();
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// 4:2:2 (which our HEVC decoder does not decode): when VideoToolbox takes it, its pictures must
|
||||
/// match ffmpeg's decode exactly; when it does not, the factory declines.
|
||||
#[test]
|
||||
fn hevc_422_10bit_matches_ffmpeg_when_the_hardware_takes_it() {
|
||||
let ff = filmcraft_testkit::require_ffmpeg!();
|
||||
let args = [
|
||||
"-f",
|
||||
"lavfi",
|
||||
"-i",
|
||||
"testsrc2=s=640x360:r=24:d=1,noise=alls=12:allf=t",
|
||||
"-c:v",
|
||||
"libx265",
|
||||
"-preset",
|
||||
"fast",
|
||||
"-tag:v",
|
||||
"hvc1",
|
||||
"-x265-params",
|
||||
"log-level=error:bframes=3:keyint=12:min-keyint=12:scenecut=0",
|
||||
"-pix_fmt",
|
||||
"yuv422p10le",
|
||||
];
|
||||
let Some(path) = fixture(&ff, "hevc_422_10.mp4", &args) else { return };
|
||||
let s = read_stream(&path);
|
||||
let Some(mut hw) = hardware(&s) else { return };
|
||||
let out = decode_all(hw.as_mut(), &s.samples);
|
||||
let o =
|
||||
std::process::Command::new(&ff).args(["-v", "error", "-i", path.to_str().unwrap(), "-f", "rawvideo", "-pix_fmt", "yuv422p10le", "-"]).output().unwrap();
|
||||
let (w, h) = (640usize, 360usize);
|
||||
let fsize = (w * h + 2 * (w / 2) * h) * 2;
|
||||
assert_eq!(out.len(), o.stdout.len() / fsize, "frame count");
|
||||
for (i, f) in out.iter().enumerate() {
|
||||
let raw: Vec<u16> = o.stdout[i * fsize..(i + 1) * fsize].as_chunks::<2>().0.iter().map(|b| u16::from_le_bytes([b[0], b[1]])).collect();
|
||||
let filmcraft_frame::PixelData::Yuv16 { planes, chroma, bits, .. } = &f.frame.data else { panic!("16-bit planes") };
|
||||
assert_eq!((*chroma, *bits), (filmcraft_frame::Chroma::C422, 10));
|
||||
let (y, c) = (w * h, (w / 2) * h);
|
||||
assert!(planes[0][..] == raw[..y] && planes[1][..] == raw[y..y + c] && planes[2][..] == raw[y + c..], "frame {i} differs from ffmpeg");
|
||||
}
|
||||
}
|
||||
|
|
@ -40,5 +40,8 @@ rustix = { version = "1", default-features = false, features = ["std", "time"] }
|
|||
[dev-dependencies]
|
||||
pollster = "0.4"
|
||||
filmcraft-codecs = { workspace = true }
|
||||
# the bench registers OS hardware decoders as the desktop app does (dev-dependency only: ui-egui
|
||||
# itself stays OS-independent and web-portable)
|
||||
filmcraft-platform = { workspace = true }
|
||||
filmcraft-interchange = { workspace = true }
|
||||
egui_kittest = { workspace = true, features = ["eframe", "wgpu"] }
|
||||
|
|
|
|||
|
|
@ -5,6 +5,7 @@
|
|||
//! ```sh
|
||||
//! cargo xtask bench # every section, writes target/bench/bench-<label>.{json,md}
|
||||
//! cargo xtask bench --sections decode,scrub --repeat 3 --label after
|
||||
//! cargo xtask bench --sections decode --hw off --label hw-off # FilmCraft's own decoders only
|
||||
//! cargo run --release -p filmcraft-ui-egui --example bench -- --section timeline # one section, this process
|
||||
//! ```
|
||||
//!
|
||||
|
|
@ -33,6 +34,8 @@ pub struct Opts {
|
|||
pub gpu: bool,
|
||||
/// Only cases whose name contains this (decode fixtures, playback / scrub scenarios).
|
||||
pub only: Option<String>,
|
||||
/// Settings ▸ Playback ▸ Hardware decoding: `auto` (default, as the app) or `off`.
|
||||
pub hw: String,
|
||||
}
|
||||
|
||||
impl Opts {
|
||||
|
|
@ -49,7 +52,7 @@ fn main() {
|
|||
let args: Vec<String> = std::env::args().skip(1).collect();
|
||||
if args.iter().any(|a| a == "--help" || a == "-h") {
|
||||
println!(
|
||||
"bench [--sections {}] [--repeat N] [--quick] [--cpu] [--only SUBSTRING] [--label NAME] [--out DIR]\n bench --section NAME [--json FILE] (one section in this process)",
|
||||
"bench [--sections {}] [--repeat N] [--quick] [--cpu] [--only SUBSTRING] [--hw auto|off] [--label NAME] [--out DIR]\n bench --section NAME [--json FILE] (one section in this process)",
|
||||
SECTIONS.join(",")
|
||||
);
|
||||
return;
|
||||
|
|
@ -59,7 +62,11 @@ fn main() {
|
|||
quick: args.iter().any(|a| a == "--quick"),
|
||||
gpu: !args.iter().any(|a| a == "--cpu"),
|
||||
only: arg_value(&args, "--only"),
|
||||
hw: arg_value(&args, "--hw").unwrap_or_else(|| "auto".into()),
|
||||
};
|
||||
// as the desktop app: OS hardware decoders in front of ours, unless Hardware decoding is Off
|
||||
filmcraft_platform::register();
|
||||
filmcraft_codecs::hw::set_hardware_decoding(opts.hw != "off");
|
||||
if let Some(section) = arg_value(&args, "--section") {
|
||||
let v = run_section(§ion, &opts);
|
||||
let text = serde_json::to_string_pretty(&v).unwrap_or_default();
|
||||
|
|
@ -113,6 +120,7 @@ fn orchestrate(args: &[String], o: &Opts) {
|
|||
if let Some(only) = &o.only {
|
||||
cmd.args(["--only", only]);
|
||||
}
|
||||
cmd.args(["--hw", &o.hw]);
|
||||
eprintln!("== {name} (load {})", playback::load_avg());
|
||||
let output = cmd.stderr(std::process::Stdio::piped()).output().expect("run section");
|
||||
let stderr = String::from_utf8_lossy(&output.stderr).to_string();
|
||||
|
|
@ -131,7 +139,7 @@ fn orchestrate(args: &[String], o: &Opts) {
|
|||
println!("{}", section_markdown(&v));
|
||||
results.push(v);
|
||||
}
|
||||
let doc = json!({"label": label, "machine": machine, "repeat": o.repeat, "quick": o.quick, "gpu": o.gpu, "sections": results});
|
||||
let doc = json!({"label": label, "machine": machine, "repeat": o.repeat, "quick": o.quick, "gpu": o.gpu, "hw": o.hw, "sections": results});
|
||||
let jp = out.join(format!("bench-{label}.json"));
|
||||
let mp = out.join(format!("bench-{label}.md"));
|
||||
std::fs::write(&jp, serde_json::to_string_pretty(&doc).unwrap_or_default()).expect("write json");
|
||||
|
|
|
|||
|
|
@ -87,6 +87,7 @@ pub fn decode(o: &Opts) -> Vec<Value> {
|
|||
"fps": best, "fps_min": fps.iter().cloned().fold(f64::MAX, f64::min), "fps_runs": fps,
|
||||
"cpu_ms_per_frame": median(&cpu), "first_frame_ms": median(&first), "realtime": best / rate.as_f64(),
|
||||
"decode_threads": rayon::current_num_threads(), "load": load_avg(),
|
||||
"hw_frames": filmcraft_codecs::hw::hw_stats().frames,
|
||||
});
|
||||
eprintln!("decode {name}: {best:.1} fps");
|
||||
rows.push(row);
|
||||
|
|
|
|||
|
|
@ -0,0 +1,65 @@
|
|||
# ADR 0001: `unsafe` OS media FFI in `crates/platform`, and nowhere else
|
||||
|
||||
- **Status:** accepted (2026-10-05, approved by the project owner; recorded in
|
||||
[AGENTS.md](../../AGENTS.md) §0 item 3 and the craftrules never-crash standard)
|
||||
- **Issue:** #30 (hardware acceleration)
|
||||
|
||||
## Context
|
||||
|
||||
FilmCraft is pure Rust and the workspace sets `unsafe_code = "forbid"`. Every codec has a
|
||||
clean-room software decoder, and those remain the reference. But decoding 4K H.264 / HEVC in
|
||||
software costs 85–120 ms of CPU per frame on an M4 Pro, so 4K playback drops frames under load,
|
||||
while every current computer has a hardware video decoder the operating system exposes
|
||||
(VideoToolbox on macOS; Media Foundation / D3D11 on Windows and VA-API on Linux later).
|
||||
|
||||
Using it needs FFI into C / Objective-C APIs, which is `unsafe` Rust. The bindings we use (the
|
||||
`objc2-*` crates, MIT / Apache-2.0 / Zlib) are generated declarations; nothing GPL/LGPL is linked.
|
||||
|
||||
## Decision
|
||||
|
||||
Allow `unsafe` in exactly one crate, `crates/platform` (`filmcraft-platform`, layer L5), for one
|
||||
job: OS media FFI (hardware decoding now, hardware encoding later). Audio output, dialogs, menus
|
||||
and other OS integration do not go in it.
|
||||
|
||||
Containment rules:
|
||||
|
||||
1. The crate does not use `lints.workspace = true`. Its own `[lints]` table copies the workspace
|
||||
lints except `unsafe_code = "deny"` (not `forbid`), and adds
|
||||
`clippy::undocumented_unsafe_blocks = "deny"`. Only the FFI modules (`videotoolbox`) carry
|
||||
`#[allow(unsafe_code)]`; the rest of the crate (the fallback logic in `hybrid`) has no `unsafe`.
|
||||
2. Every `unsafe` block has a `// SAFETY:` comment saying why it is sound.
|
||||
3. The public API is safe: no `pub unsafe fn`, no raw pointers or FFI types in public signatures;
|
||||
failures are `Result`s. The crate keeps the never-crash `deny(clippy::unwrap_used, …)`
|
||||
attribute of every clean crate.
|
||||
4. No panic crosses the FFI boundary: the VideoToolbox output callback runs under `catch_unwind`
|
||||
and reports a panic as a failed frame.
|
||||
5. The crate compiles on every target. OS bindings are target-specific dependencies; elsewhere
|
||||
`register()` reports `Unavailable` and does nothing. It is not part of the wasm check (L5).
|
||||
|
||||
## The fallback guarantee
|
||||
|
||||
Registering a hardware decoder never makes a file undecodable, and never changes a picture:
|
||||
|
||||
- The factory declines (the software decoder is used) when Settings ▸ Playback ▸ Hardware
|
||||
decoding is Off, for formats the hardware path does not take (field-coded H.264, bit depths
|
||||
other than 8 / 10, 4:4:4, mismatched luma / chroma depth), and when the OS cannot create a
|
||||
hardware session for the stream (hardware decoding is *required*, so the OS's own software
|
||||
decoder is never used instead of ours).
|
||||
- A hardware decoder that fails mid-stream (decode error, session invalidated by a GPU change or
|
||||
sleep, in-band parameter sets that differ from the sample entry) becomes the software decoder
|
||||
for that stream without the caller noticing: `HybridDecoder` replays the samples since the last
|
||||
restart point through it and drops pictures already returned. It is logged and counted
|
||||
(`perf.stats` `decode.hardware.fallbacks`).
|
||||
- Pictures are interchangeable with the software decoder's: same planes (bit-exact on the H.264
|
||||
High, HEVC Main and Main 10 parity fixtures, including after seeks and flushes), colour, pixel
|
||||
aspect, pts and presentation order. Tests compare the two on every run on macOS, and the
|
||||
fallback logic is tested on every platform with a stand-in hardware decoder.
|
||||
|
||||
## Consequences
|
||||
|
||||
- 4K H.264 / HEVC decode costs ~4 ms of CPU per frame instead of 85–120 ms
|
||||
([performance.md](../performance.md)).
|
||||
- Better patent posture for the most encumbered formats: H.264 / HEVC decoding is done by the
|
||||
operating system, which carries the licences, wherever hardware decoding is used.
|
||||
- A new review burden: changes to `crates/platform` need the same scrutiny as any FFI code.
|
||||
Anything else that wants `unsafe` needs a new decision, not an extension of this one.
|
||||
|
|
@ -21,7 +21,7 @@ Design principles:
|
|||
|
||||
```text
|
||||
L6 apps/filmcraft · apps/filmcraft-cli · apps/filmcraft-web
|
||||
L5 ui-egui · automation
|
||||
L5 ui-egui · automation · platform
|
||||
L4 engine
|
||||
L3 render · gpu · export · golden (test-only)
|
||||
L2 edit · codecs · interchange · captions · speech
|
||||
|
|
@ -71,6 +71,7 @@ and `filmcraft-cli`.
|
|||
| `engine` | L4 | `Session`, command registry, undo history, media pool, jobs, interchange glue |
|
||||
| `ui-egui` | L5 | the egui frontend: docking, panels, timeline, monitors, playback, control-channel handlers |
|
||||
| `automation` | L5 | MCP server (`rmcp`, stdio), headless or bridged to the running app |
|
||||
| `platform` | L5 | OS media FFI only: hardware video decoding (VideoToolbox H.264 / HEVC on macOS; a no-op elsewhere) behind `codecs::VideoDecoder`, with transparent fallback to our decoders. The one crate allowed `unsafe` ([ADR 0001](adr/0001-platform-ffi.md), [README](../crates/platform/README.md)) |
|
||||
| `filmcraft` | L6 | desktop binary: eframe/wgpu window, cpal audio output, file dialogs, native macOS menu, TCP control server |
|
||||
| `filmcraft-cli` | L6 | headless CLI: `exec`, `run`, `inspect`, `describe`, `commands`, `import`, `export`, `render`, `probe`, `mcp`; `--bridge` targets the running app |
|
||||
| `filmcraft-web` | L6 | the browser app (wasm32): eframe web runner on WebGPU/WebGL2, Blob-backed services, OPFS recovery, WebAudio, WebCodecs, `window.filmcraft` API ([web.md](web.md)) |
|
||||
|
|
@ -91,7 +92,8 @@ are exempt):
|
|||
|
||||
`cargo xtask wasm` runs `cargo check --target wasm32-unknown-unknown` on every L0–L4 crate, the
|
||||
egui UI and the web app, so everything up to the engine stays web-portable and the web app builds
|
||||
([web.md](web.md)). `unsafe_code = "deny"` applies workspace-wide.
|
||||
([web.md](web.md)). `unsafe_code = "forbid"` applies workspace-wide, except in `platform`
|
||||
(`deny`, allowed only on its FFI modules; [ADR 0001](adr/0001-platform-ffi.md)).
|
||||
|
||||
## 2. Time base
|
||||
|
||||
|
|
@ -196,7 +198,8 @@ pub struct CommandSpec {
|
|||
|
||||
```text
|
||||
file ──► codecs (MP4/MOV, MKV, audio) demux + decode, GOP-aware seek
|
||||
│ decoder registry: h264, hevc, vp9, av1, prores, dnx, mjpeg (+ any registered first)
|
||||
│ decoder registry: h264, hevc, vp9, av1, prores, dnx, mjpeg (+ any registered first:
|
||||
│ platform's VideoToolbox H.264 / HEVC on macOS, falling back to ours)
|
||||
▼
|
||||
media::MediaSource ──► frame cache (byte-budgeted LRU, shared)
|
||||
▼
|
||||
|
|
@ -225,6 +228,24 @@ file ──► codecs (MP4/MOV, MKV, audio) demux + decode, GOP-aware see
|
|||
every frame of the GOP. Sequential playback reuses the decoder. Decoders implement
|
||||
`codecs::VideoDecoder`. `register_video_decoder` puts a factory in front of the built-in ones, so a
|
||||
hardware decoder can take precedence. The GOP cache never holds its lock while decoding.
|
||||
- **Hardware decoding.** The apps (desktop, CLI / headless MCP, bench) call
|
||||
`filmcraft_platform::register()` at startup, which on macOS registers a VideoToolbox factory for
|
||||
`avcC` / `hvcC` streams (8 / 10-bit, 4:2:0 and 4:2:2; elsewhere it does nothing). The factory
|
||||
declines (our decoder is used) when Settings ▸ Playback ▸ Hardware decoding is Off
|
||||
(`codecs::hw::set_hardware_decoding`, applied by the engine whenever preferences change), for
|
||||
formats it does not take, and when the OS cannot create a *hardware* session for the stream.
|
||||
Its decoder is a `platform::HybridDecoder`: VideoToolbox decodes asynchronously (two access
|
||||
units in flight), a reorder buffer of the stream's own depth (`max_num_reorder_frames` /
|
||||
`sps_max_num_reorder_pics`, from `codecs::hw::NalStreamInfo`) restores presentation order, and
|
||||
the decoded biplanar `CVPixelBuffer` is copied into planar `Yuv8` / `Yuv16`. Colour, pixel
|
||||
aspect, random-access and disposable answers come from the same helpers as the software
|
||||
decoders (`video::vui_color`, `sar_par`, `h264_disposable`, `hevc_disposable`), so the two are
|
||||
interchangeable (bit-exact in the parity tests). If the hardware fails mid-stream (decode error,
|
||||
session lost to a GPU change or sleep, changed in-band parameter sets), the hybrid builds
|
||||
`codecs::software_video_decoder` (built-in factories only), replays the samples since the last
|
||||
restart point and continues in software for that instance; it is logged and counted. A source
|
||||
already open keeps its decoder when the setting changes, until it is reopened. `perf.stats`
|
||||
`decode.hardware` reports hardware vs software frames, sessions, declines and fallbacks.
|
||||
Decoders run slices on rayon, so an export worker waiting inside a decode can pick up another
|
||||
frame of the same source. A request that finds the shared decoder busy decodes with a private
|
||||
decoder.
|
||||
|
|
@ -639,7 +660,8 @@ Protocol reference: [control-protocol.md](control-protocol.md). Agent guide: [ag
|
|||
## 8. Not built yet
|
||||
|
||||
The layer table reserves names for crates that don't exist yet: `riff`, `mjpeg`,
|
||||
`keyframe`, `effects`, `audio`, `playback` and `platform`.
|
||||
`keyframe`, `effects`, `audio` and `playback`. (`platform` exists but holds only OS media FFI;
|
||||
other OS integration stays in `apps/filmcraft`.)
|
||||
Until they exist, that work lives elsewhere: keyframes and effect definitions in `project`, effects
|
||||
and the audio mix in `render`, playback in `ui-egui`, and OS integration (cpal, rfd,
|
||||
native menus) in `apps/filmcraft`. [ROADMAP.md](../ROADMAP.md) has the milestone status.
|
||||
|
|
|
|||
|
|
@ -29,7 +29,7 @@ Methods (handlers in `crates/ui-egui/src/control.rs`):
|
|||
| `ui.playback` | `{action: play|stop|toggle, speed?}` | |
|
||||
| `ui.screenshot` | `{path?, panel?}` | PNG of the window or one panel; fails after 10 s when no frame is presented (window hidden, display asleep) |
|
||||
| `ui.resize`, `ui.focus`, `app.quit` | | `ui.focus` is the only request that activates the app and takes keyboard focus |
|
||||
| `perf.stats` | – | performance counters (also the command `perf.stats`, so `engine.execute` and MCP `command_run` reach it; headless sessions return the engine part). `decode`: GOP-cache requests / hits / `cacheHitRate`, decoder seeks, samples decoded and skipped while catching up, `draftFrames` (decoded in draft mode), `h264Threads` (frame-threading workers per H.264 decoder), decoder ms (total and per sample); `playback`: playing, shown / dropped frames and `dropRate` of the current or last play, resolution, `draftDecode` (Settings ▸ Playback ▸ Draft decoding); `frames`: frame-worker jobs, cancelled, `requestHitRate`, `decodeMs` / `renderMs` (mean, p50, p95 of the last 256 jobs), queue, render-cost estimate, cache use; `ui`: fps and frame ms; `process`: CPU seconds; `media`, `jobs`. Counters are cumulative: diff two readings to measure an interval ([performance.md](performance.md)) |
|
||||
| `perf.stats` | – | performance counters (also the command `perf.stats`, so `engine.execute` and MCP `command_run` reach it; headless sessions return the engine part). `decode`: GOP-cache requests / hits / `cacheHitRate`, decoder seeks, samples decoded and skipped while catching up, `draftFrames` (decoded in draft mode), `h264Threads` (frame-threading workers per H.264 decoder), decoder ms (total and per sample), `framesDecoded`, `hardware` (Settings ▸ Playback ▸ Hardware decoding: `enabled`, pictures from hardware decoders `frames` vs `softwareFrames`, `sessions`, `declined` streams, mid-stream `fallbacks`); `playback`: playing, shown / dropped frames and `dropRate` of the current or last play, resolution, `draftDecode` (Settings ▸ Playback ▸ Draft decoding); `frames`: frame-worker jobs, cancelled, `requestHitRate`, `decodeMs` / `renderMs` (mean, p50, p95 of the last 256 jobs), queue, render-cost estimate, cache use; `ui`: fps and frame ms; `process`: CPU seconds; `media`, `jobs`. Counters are cumulative: diff two readings to measure an interval ([performance.md](performance.md)) |
|
||||
|
||||
**Focus.** Driving the app never steals the user's keyboard. Started with `--control`, the app opens
|
||||
without activating itself. UI requests and screenshots that need a rendered frame bring the window
|
||||
|
|
|
|||
|
|
@ -13,6 +13,48 @@ the before/after comparison relies on. Each milestone below alternated base / af
|
|||
each); the most load-independent figure is the decoder's own cycle count (`proc_pid_rusage`
|
||||
instructions and cycles of a process, all threads summed).
|
||||
|
||||
## Results (HW1: VideoToolbox hardware decoding, Off → Auto)
|
||||
|
||||
Same commit, Settings ▸ Playback ▸ Hardware decoding switched with the bench flag `--hw off` /
|
||||
`--hw auto` (`cargo xtask bench --sections decode --only dec_h --repeat 3 --hw …`, then
|
||||
`--sections playback --only h264` and `--only hevc-2160`), 2026-10-05, Apple M4 Pro, load average
|
||||
**150–190** throughout (shared with other agent builds). The decoded pictures are identical either
|
||||
way (bit-exact parity tests, `crates/platform/tests/videotoolbox.rs`); only who decodes changes.
|
||||
|
||||
### Decode (every frame through the media stack)
|
||||
|
||||
| codec | size | CPU ms/frame Off → **Auto** | fps Off → **Auto** (best of 3) |
|
||||
|---|---|---|---|
|
||||
| H.264 | 1080p | 31.8 → **1.2** | 153 → **389** |
|
||||
| H.264 | 2160p | 119.3 → **4.2** | 35 → **107** |
|
||||
| HEVC | 1080p | 22.2 → **1.1** | 185 → **708** |
|
||||
| HEVC | 2160p | 85.7 → **3.7** | 49 → **217** |
|
||||
|
||||
CPU time per frame is what is left on the CPU: the sample copy into a `CMSampleBuffer`, copying
|
||||
the decoded biplanar picture out into planar Y'CbCr (chroma deinterleaved; 10-bit shifted down),
|
||||
and the GOP cache. An earlier synchronous version (one picture decoding at a time) reached only
|
||||
155 / 51 / 449 / 149 fps; asynchronous decompression with two access units in flight overlaps
|
||||
decoding with the copy-out.
|
||||
|
||||
### Program-monitor playback (8 s, GPU path)
|
||||
|
||||
| case | shown/dropped Off → **Auto** | CPU ms/frame Off → **Auto** |
|
||||
|---|---|---|
|
||||
| H.264 1080p Full | 192/0 → **192/0** | 38.1 → **3.8** |
|
||||
| H.264 2160p Full | 104/88 → **192/0** | 119.8 → **9.2** |
|
||||
| H.264 2160p 1/2 | 1/191 → **192/0** | 95.5 → **9.0** |
|
||||
| H.264 2160p 1/4 | 0/192 → **192/0** | 94.3 → **8.8** |
|
||||
| HEVC 2160p Full | 175/17 → **192/0** | 105.6 → **9.1** |
|
||||
| HEVC 2160p 1/2 | 190/2 → **192/0** | 97.4 → **8.8** |
|
||||
| HEVC 2160p 1/2 draft | 190/2 → **191/1** | 107.0 → **22.3** |
|
||||
|
||||
At this load the software decoders drop most 4K H.264 frames; with hardware decoding every case
|
||||
plays in real time at a tenth of the CPU. Draft playback costs more CPU than full-quality
|
||||
playback with hardware decoding: the hardware ignores draft mode (`set_draft` is a no-op) and
|
||||
the draft plan's box decimation of the planes, cheap next to software decoding, is now the main
|
||||
CPU cost. Zero-copy upload of the decoded `CVPixelBuffer` into wgpu (no copy-out at all) is the
|
||||
next step (issue #30).
|
||||
|
||||
## Results (GPU1: blend modes on the GPU compositor, #30, before → after)
|
||||
|
||||
Before = this change with the old whole-frame CPU fallback for non-Normal blend modes put back
|
||||
|
|
|
|||
|
|
@ -238,7 +238,7 @@ own process under `/usr/bin/time` so peak RSS is per section: `decode` (every fr
|
|||
(the real app under `egui_kittest` with a 1000-clip / 20-track sequence: update, tessellation and
|
||||
wgpu render ms per frame), `export` (H.264 + AAC and ProRes) and `project` (save / open 5000 clips).
|
||||
Options: `--sections decode,scrub`, `--only <substring>` (fixture or scenario), `--repeat N`,
|
||||
`--quick`, `--cpu`, `--label NAME` (output `target/bench/bench-<label>.{json,md}`),
|
||||
`--quick`, `--cpu`, `--hw auto|off` (Settings ▸ Playback ▸ Hardware decoding; default `auto`, as the app), `--label NAME` (output `target/bench/bench-<label>.{json,md}`),
|
||||
`--section NAME --json FILE` (one section in-process, e.g. under a profiler). Fixtures are made
|
||||
with ffmpeg in `target/fixtures/playback/` (or `$FILMCRAFT_FIXTURES/playback`). The suite's
|
||||
`playback` section plays `h264-1080`, `stack3` and `stack3-blend` at Full, `h264-2160` at Full, 1/2 and 1/4 and with
|
||||
|
|
|
|||
Loading…
Reference in New Issue