From 4cab6bc5dae89638e06e5ddf86bc3f223cc7010a Mon Sep 17 00:00:00 2001 From: Dario Carnelutti Date: Tue, 6 Oct 2026 20:04:55 -0600 Subject: [PATCH] platform: opt-in hardware H.264 encoding with VideoToolbox Export encodes H.264 on the CPU with the built-in encoder, which is most of the time of an export (about two thirds of the busy CPU in a profile of a 20 s export). On macOS the media engine can do it: a VideoToolbox compression session behind `filmcraft_export::VideoEncoder`, registered by `filmcraft_platform::register()`. - `ExportSettings::hardware_encoding` (`HardwareEncoding::Off` | `Auto`; Export > Video > Hardware Encoding, `"hardwareEncoding": "off|auto"` in `file.exportMedia`, a boolean is understood too), off by default: the built-in encoder's output is byte-identical on every machine, a hardware encoder's is not. - `videotoolbox_encode` (FFI, same containment as the decoder: SAFETY comments, no panic across the callback, safe API) and `hardware_encode` (safe adapter and factory). - The factory declines, so the built-in encoder is used, for other formats, MXF, two-pass VBR, HDR, odd sizes, non-square pixels and any configuration VideoToolbox cannot create a hardware session for. An encoder that fails mid-export stops it with an error: unlike a decoder, it cannot hand a half-written stream to another one. - No B-frames: on real 1080p footage the quality is the same with and without them (+-0.3 dB at equal bitrate) and the bitrate lands closer to the target, and without them every frame is presented at its decode time (no composition offsets, no edit list). - `export::register_encoder` ignores a factory registered twice; `encoder_registered`. Apple M1, 8 cores, single runs: a 9:29 timeline (camera clip, ProRes 4444 overlays with alpha, AAC, loudness) exports in 311 s instead of 793 s (2.5x faster); its densest 134 s in 84 s instead of 395 s (4.7x). The outputs compare at SSIM 0.990 / PSNR 46.5 dB. The encoder alone runs 200 fps at 1080p (docs/performance.md). Tests (tests/hardware_encode.rs): what is taken and declined, a round trip through our software decoder, an export that decodes in our decoder and in ffmpeg, the fallback for everything hardware declines, exact output sizes, an empty range, hostile configurations and encoders dropped at any point; the engine's `hardwareEncoding` parameter. Co-Authored-By: Claude Sonnet 5.5 --- Cargo.lock | 3 + README.md | 2 +- ROADMAP.md | 4 +- crates/engine/src/commands.rs | 2 +- crates/engine/src/export_tests.rs | 20 + crates/engine/src/export_tools.rs | 14 +- crates/export/src/lib.rs | 18 +- crates/export/src/settings.rs | 13 + crates/platform/Cargo.toml | 6 + crates/platform/README.md | 39 +- crates/platform/src/hardware_encode.rs | 158 ++++++ crates/platform/src/lib.rs | 13 +- crates/platform/src/videotoolbox_encode.rs | 572 +++++++++++++++++++++ crates/platform/tests/hardware_encode.rs | 433 ++++++++++++++++ crates/ui-egui/src/panels/export_mode.rs | 12 +- crates/ui-egui/tests/export_ui.rs | 1 + docs/adr/0001-platform-ffi.md | 20 + docs/architecture.md | 2 +- docs/performance.md | 56 ++ 19 files changed, 1375 insertions(+), 13 deletions(-) create mode 100644 crates/platform/src/hardware_encode.rs create mode 100644 crates/platform/src/videotoolbox_encode.rs create mode 100644 crates/platform/tests/hardware_encode.rs diff --git a/Cargo.lock b/Cargo.lock index 63e28933..b6ad1c92 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2089,9 +2089,12 @@ version = "0.2.1" dependencies = [ "filmcraft-codecs", "filmcraft-color", + "filmcraft-export", "filmcraft-frame", "filmcraft-isobmff", "filmcraft-media", + "filmcraft-project", + "filmcraft-render", "filmcraft-testkit", "filmcraft-time", "log", diff --git a/README.md b/README.md index 70bac481..0dd28e7b 100644 --- a/README.md +++ b/README.md @@ -286,7 +286,7 @@ We track two numbers ([ROADMAP.md](ROADMAP.md#honest-assessment-2026-10-05)): The biggest gaps today: -- **Speed on big footage.** Hardware decoding works on macOS; Windows and Linux have no hardware path yet. Blend modes and the most common effects run on the GPU, but Lumetri, keys, export and encoding still run on the CPU ([#30](https://github.com/storytold/filmcraft/issues/30)). +- **Speed on big footage.** Hardware decoding works on macOS; Windows and Linux have no hardware path yet. Blend modes and the most common effects run on the GPU, but Lumetri, keys and export rendering still run on the CPU; H.264 encoding can use the hardware encoder on macOS (opt-in) ([#30](https://github.com/storytold/filmcraft/issues/30)). - **No plugins.** No VST3 / Audio Units or OpenFX hosting. - **Delivery codecs.** H.264 is our only delivery-codec export; no HEVC or AV1 export yet. - **Real-world media and platforms.** Our decoders are bit-exact on conformance streams, but camera and phone files in the wild are less tested. Windows and Linux get far less testing than macOS. diff --git a/ROADMAP.md b/ROADMAP.md index 553a6fcf..93a62fe1 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -36,7 +36,7 @@ before choosing work. | **Correctness in depth** | not tracked | **~60–70%** | A first-time contributor found four real bugs in core paths within hours (#5–#8: AIFF import failed, Slide left linked audio behind, transitions didn't follow ripple trims, export ignored start/end times). User reports: Color Matte always grey with no picker (#29); ALSA audio error on Linux (#23). Settings are "most wired", not all. | | **Codecs and media** | ~90% | **~75%** | Our decoders are bit-exact on conformance streams. Real camera and phone media (variable frame rate, damaged files, unusual containers) is far less tested. No camera RAW (RED, BRAW, ARRIRAW), no E-AC-3. | | **Export** | ~92% | **~70%** | Delivery: H.264 only (ProRes, DNxHR and image sequences cover mastering). No HEVC or AV1 export. AAF and OMF have never been validated in Avid Media Composer or Pro Tools. Export renders and encodes on the CPU only. | -| **Performance and hardware** | ~62% | **~35–40%** | Premiere runs effects, decoding and encoding on the GPU and the hardware media engines. Here, hardware decode works on macOS only (VideoToolbox, #33; active in the desktop app only since #41, so releases up to 0.2.1 decoded in software). The GPU does compositing, blend modes (#32) and 31 common effects (GPU2): colour adjustments, crop/flip/transform, blurs, sharpen. Lumetri, keys, masks, the remaining effects, export and encoding are CPU. Linux and Windows have no hardware path. 8K and AV1 aren't real-time. The effect list shows Premiere's "GPU accelerated" badge, but only 31 effect ids run on the GPU here; the rest, Lumetri included, run on the CPU. | +| **Performance and hardware** | ~62% | **~35–40%** | Premiere runs effects, decoding and encoding on the GPU and the hardware media engines. Here, hardware decode works on macOS only (VideoToolbox, #33; active in the desktop app only since #41, so releases up to 0.2.1 decoded in software). The GPU does compositing, blend modes (#32) and 31 common effects (GPU2): colour adjustments, crop/flip/transform, blurs, sharpen. Lumetri, keys, masks, the remaining effects and export rendering are CPU; H.264 encoding can use VideoToolbox on macOS (opt-in per export, no B-frames). Linux and Windows have no hardware path. 8K and AV1 aren't real-time. The effect list shows Premiere's "GPU accelerated" badge, but only 31 effect ids run on the GPU here; the rest, Lumetri included, run on the CPU. | | **Stability** | not tracked | **improving, unproven** | The never-crash pass (no panics in product code, last-resort guards) landed on 2026-10-04. There's no field record from real users' projects yet. | | **Plugins and pro ecosystem** | not tracked | **~0–10%** | No audio plugin hosting (VST3 / Audio Units) and no third-party video effects (OpenFX). For many professional editors this alone rules FilmCraft out. Team Projects and Productions are out of scope by design. | | **AI features** | partial | **~25–35%** | Speech to text exists but is off in default builds (the `whisper` feature). Enhance Speech is a DSP chain, not a model. Auto Reframe is approximate. No Generative Extend, and no media-intelligence search or auto colour. | @@ -51,7 +51,7 @@ In priority order. Agents should prefer this work over adding more checklist ite - hardware decode on Linux (VA-API) and Windows (Media Foundation); - zero-copy decoded frames into wgpu; - the remaining effects on the GPU: Lumetri, keys, Vignette, Video Limiter, masks (31 common effects are done); - - GPU export, then hardware encode (H.264 / HEVC). + - GPU export, then hardware encode: H.264 on macOS is in (opt-in); HEVC and other systems remain. 2. **A measured parity number.** Add `cargo xtask parity`, run in CI. It should cover menus, commands, effects, transitions, panels, preferences and formats against the Premiere snapshot in `plan/premiere/`. It should report presence and fidelity separately (exact, approximate, stub). diff --git a/crates/engine/src/commands.rs b/crates/engine/src/commands.rs index 23f2dccd..dfc9744b 100644 --- a/crates/engine/src/commands.rs +++ b/crates/engine/src/commands.rs @@ -1018,7 +1018,7 @@ fn build() -> Vec { "Media…", ["File", "Export"], None, - r#"{"path":str,"preset":str?,"settings":ExportSettings?,"format":"h264|prores|dnxhr|apv|mjpeg|mxf-op1a|mxf-opatom|png|tiff|bmp|gif|wav|aiff"?,"width":u32?,"height":u32?,"fps":f64?,"bitrateKbps":u32?,"bitrateMode":"cbr|vbr1Pass|vbr2Pass"?,"scale":f32=1,"audio":bool=true,"quality":0..100,"burnCaptions":bool=false,"captionSidecar":"srt|vtt"?,"loudnessLufs":f64?,"proresProfile":"proxy|lt|standard|hq"?,"dnxProfile":"lb|sq|hq|hqx"?,"apvProfile":"422-10|422-12|444-10|444-12"?,"mxfVideoCodec":"dnxhr|proRes|h264"?,"sequence":id?,"range":"entire|inOut|workArea|custom"?,"startSeconds":f64?,"endSeconds":f64?,"wait":bool=false}"#, + r#"{"path":str,"preset":str?,"settings":ExportSettings?,"format":"h264|prores|dnxhr|apv|mjpeg|mxf-op1a|mxf-opatom|png|tiff|bmp|gif|wav|aiff"?,"width":u32?,"height":u32?,"fps":f64?,"bitrateKbps":u32?,"bitrateMode":"cbr|vbr1Pass|vbr2Pass"?,"hardwareEncoding":"off|auto"?,"scale":f32=1,"audio":bool=true,"quality":0..100,"burnCaptions":bool=false,"captionSidecar":"srt|vtt"?,"loudnessLufs":f64?,"proresProfile":"proxy|lt|standard|hq"?,"dnxProfile":"lb|sq|hq|hqx"?,"apvProfile":"422-10|422-12|444-10|444-12"?,"mxfVideoCodec":"dnxhr|proRes|h264"?,"sequence":id?,"range":"entire|inOut|workArea|custom"?,"startSeconds":f64?,"endSeconds":f64?,"wait":bool=false}"#, has_seq, crate::export_tools::export_media ), diff --git a/crates/engine/src/export_tests.rs b/crates/engine/src/export_tests.rs index 62f22093..337c4772 100644 --- a/crates/engine/src/export_tests.rs +++ b/crates/engine/src/export_tests.rs @@ -363,6 +363,26 @@ fn queue_cancels_a_running_export_and_retries_a_failed_one() { assert!(queue(&mut s).iter().all(|i| i["id"] != bad)); } +#[test] +fn hardware_encoding_is_off_unless_asked_for() { + use filmcraft_export::HardwareEncoding::{Auto, Off}; + let s = demo(); + let setting = |p: Value| crate::export_tools::settings_from_params(&s, &p, "file.exportMedia").map(|(_, st)| st.hardware_encoding); + assert_eq!(setting(json!({"path": "x.mp4"})).unwrap(), Off); + assert_eq!(setting(json!({"path": "x.mp4", "hardwareEncoding": "auto"})).unwrap(), Auto); + assert_eq!(setting(json!({"path": "x.mp4", "hardwareEncoding": "off"})).unwrap(), Off); + // a boolean is understood too: true is auto + assert_eq!(setting(json!({"path": "x.mp4", "hardwareEncoding": true})).unwrap(), Auto); + assert_eq!(setting(json!({"path": "x.mp4", "hardwareEncoding": false})).unwrap(), Off); + // anything else names the choices + let e = setting(json!({"path": "x.mp4", "hardwareEncoding": "gpu"})).unwrap_err().to_string(); + assert!(e.contains("off | auto"), "{e}"); + // and it travels in the settings object, with the default filling in when it is missing + assert_eq!(setting(json!({"path": "x.mp4", "settings": {"hardwareEncoding": "auto"}})).unwrap(), Auto); + let old: filmcraft_export::ExportSettings = serde_json::from_value(json!({"format": "h264"})).unwrap(); + assert_eq!(old.hardware_encoding, Off); +} + #[test] fn queue_exports_several_sequences_and_ranges() { let mut s = demo(); diff --git a/crates/engine/src/export_tools.rs b/crates/engine/src/export_tools.rs index a6cee128..fc83a5c3 100644 --- a/crates/engine/src/export_tools.rs +++ b/crates/engine/src/export_tools.rs @@ -26,7 +26,7 @@ use std::sync::Arc; use std::sync::atomic::Ordering; use filmcraft_export::presets::{DEFAULT_PRESET, preset_key}; -use filmcraft_export::{ExportPreset, ExportSettings, Format}; +use filmcraft_export::{ExportPreset, ExportSettings, Format, HardwareEncoding}; use filmcraft_project::{ItemId, Project}; use filmcraft_time::{FrameRate, Tick, TimeRange}; use serde::{Deserialize, Serialize}; @@ -200,6 +200,18 @@ pub fn settings_from_params(s: &Session, p: &Value, cmd: &str) -> Result<(Option if let Some(v) = u64_p(p, "keyframeDistance") { settings.keyframe_distance = Some(v as u32); } + if let Some(v) = p.get("hardwareEncoding") { + settings.hardware_encoding = match v { + Value::Bool(on) => { + if *on { + HardwareEncoding::Auto + } else { + HardwareEncoding::Off + } + } + other => serde_json::from_value(other.clone()).map_err(|_| bad(cmd, "hardwareEncoding: off | auto"))?, + }; + } if let Some(v) = bool_p(p, "burnCaptions") { settings.burn_captions = v; } diff --git a/crates/export/src/lib.rs b/crates/export/src/lib.rs index d50e1f30..af576cad 100644 --- a/crates/export/src/lib.rs +++ b/crates/export/src/lib.rs @@ -248,6 +248,11 @@ pub struct ExportSettings { /// stream needs is raised. pub h264_level: Option, pub bitrate_mode: BitrateMode, + /// May H.264 be encoded by the system's hardware encoder (VideoToolbox on macOS)? Off unless + /// asked for: hardware output depends on the machine, so it is not byte-reproducible like the + /// built-in encoder's (`determinism_tests`). + #[serde(default)] + pub hardware_encoding: HardwareEncoding, /// VBR maximum bitrate (None = 1.5 × target). pub max_bitrate_kbps: Option, /// Adaptive bitrate (the Match Source presets): bits per pixel per frame; replaces @@ -429,6 +434,7 @@ impl Default for ExportSettings { h264_profile: H264Profile::High, h264_level: None, bitrate_mode: BitrateMode::default(), + hardware_encoding: HardwareEncoding::default(), max_bitrate_kbps: None, adaptive_bitrate: None, keyframe_distance: None, @@ -553,8 +559,18 @@ fn audio_factories() -> &'static RwLock> { F.get_or_init(|| RwLock::new(vec![aac_factory])) } +/// Register a video encoder factory (tried before the built-in ones and those registered earlier). +/// Registering the same factory twice is harmless. pub fn register_encoder(f: EncoderFactory) { - video_factories().write().unwrap_or_else(|e| e.into_inner()).insert(0, f); + let mut g = video_factories().write().unwrap_or_else(|e| e.into_inner()); + if !g.iter().any(|x| std::ptr::fn_addr_eq(*x, f)) { + g.insert(0, f); + } +} + +/// Whether `f` is among the registered video encoder factories (startup diagnostics, tests). +pub fn encoder_registered(f: EncoderFactory) -> bool { + video_factories().read().unwrap_or_else(|e| e.into_inner()).iter().any(|x| std::ptr::fn_addr_eq(*x, f)) } pub fn register_audio_encoder(f: AudioEncoderFactory) { audio_factories().write().unwrap_or_else(|e| e.into_inner()).insert(0, f); diff --git a/crates/export/src/settings.rs b/crates/export/src/settings.rs index 76a4443e..fd24d48b 100644 --- a/crates/export/src/settings.rs +++ b/crates/export/src/settings.rs @@ -61,6 +61,19 @@ impl H264Profile { } } +/// Export ▸ Hardware encoding: whether H.264 may be encoded by the system's hardware video encoder +/// (VideoToolbox on macOS) instead of FilmCraft's own encoder. Off by default: hardware encoders make +/// different streams, and exports are otherwise byte-identical from run to run and machine to machine. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub enum HardwareEncoding { + #[default] + Off, + /// Use the hardware encoder when the system has one and it takes the settings; otherwise the + /// software encoder. + Auto, +} + /// Bitrate encoding of bitrate-driven codecs (H.264). #[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Serialize, Deserialize)] #[serde(rename_all = "camelCase")] diff --git a/crates/platform/Cargo.toml b/crates/platform/Cargo.toml index 152110eb..9ca850cc 100644 --- a/crates/platform/Cargo.toml +++ b/crates/platform/Cargo.toml @@ -27,6 +27,7 @@ log = { workspace = true } [target.'cfg(target_os = "macos")'.dependencies] filmcraft-color = { workspace = true } +filmcraft-export = { workspace = true } filmcraft-frame = { workspace = true } filmcraft-time = { workspace = true } objc2-core-foundation = "0.3.2" @@ -38,3 +39,8 @@ objc2-video-toolbox = "0.3.2" filmcraft-testkit = { workspace = true } filmcraft-media = { workspace = true } filmcraft-frame = { workspace = true } + +[target.'cfg(target_os = "macos")'.dev-dependencies] +filmcraft-project = { workspace = true } +filmcraft-render = { workspace = true } +filmcraft-time = { workspace = true } diff --git a/crates/platform/README.md b/crates/platform/README.md index e725a75d..fd1d1bf8 100644 --- a/crates/platform/README.md +++ b/crates/platform/README.md @@ -1,7 +1,8 @@ # filmcraft-platform OS media integration for FilmCraft (layer L5): hardware video decoding through the operating -system's codecs, behind `filmcraft_codecs::VideoDecoder`. It holds OS media FFI and nothing else. +system's codecs, behind `filmcraft_codecs::VideoDecoder`, and hardware H.264 encoding, behind +`filmcraft_export::VideoEncoder`. It holds OS media FFI and nothing else. It is the one crate of the workspace allowed to contain `unsafe`, under the rules of [ADR 0001](../../docs/adr/0001-platform-ffi.md) and [AGENTS.md](../../AGENTS.md) §0.3. @@ -31,6 +32,31 @@ let availability = filmcraft_platform::register(); // Available("VideoToolbox") log is bounded (600 samples / 256 MB); beyond it one error is returned and the next seek restarts in software. Streams our decoders cannot decode (HEVC 4:2:2) have no fallback: the error stands. +## Hardware H.264 encoding (macOS) + +`videotoolbox_encode.rs` (FFI) wraps a VideoToolbox compression session with a hardware encoder +*required*; `hardware_encode.rs` (safe code) is the `VideoEncoder` adapter and the factory that +`register()` puts in front of the built-in encoders (`filmcraft_export::register_encoder`). + +- **Opt-in per export:** `ExportSettings::hardware_encoding` (`Off` | `Auto`; Export ▸ Video ▸ + Hardware Encoding, `"hardwareEncoding": "off|auto"` in `file.exportMedia`), off by default. The built-in encoder's + output is byte-identical on every machine; a hardware encoder's depends on the machine. +- **What it takes:** H.264 in MP4 / MOV, 8-bit SDR, even picture sizes up to 8192, Baseline / Main / + High (the level is chosen by the OS), constant or one-pass variable bitrate with the settings' + target and ceiling, the keyframe distance (closed GOPs: every keyframe is an IDR picture). + Pictures are converted exactly like the built-in encoder's (BT.709, limited range), 4:2:0 NV12. +- **No B-frames.** On real 1080p footage the quality is the same with and without them (±0.3 dB at + equal bitrate) and the bitrate lands closer to the target without them, so frame reordering is + off: compressed frames come out in presentation order, with no composition offsets or edit list. +- **What it declines** (the built-in encoder is used, nothing fails): the setting off, other formats, + MXF (Annex B), two-pass VBR, HDR, odd sizes (4:2:0 cannot crop an odd number of samples), + non-square pixels, and any configuration VideoToolbox cannot create a hardware session for + (logged at `info`). +- **A hardware encoder that fails in the middle of an export is an error**, unlike the decoder: + an encoder cannot hand a half-written stream to another one, so the export stops with the reason. +- The first frame is completed straight away: the SPS / PPS the container needs come with the first + compressed frame, and the muxer asks for them after the first group of pictures. + ## Guarantees - **Never undecodable:** the factory declines (returns `None`, so the software decoder is used) @@ -52,12 +78,21 @@ let availability = filmcraft_platform::register(); // Available("VideoToolbox") | `tests/videotoolbox.rs` (macOS) | H.264 High, HEVC Main (open GOP: CRA + RASL) and HEVC Main 10, 640×360 (coded 368: cropping) with B-frames: every picture **bit-exact** with our software decoder, same pts order, count, colour and aspect, also after `reset` + reseek to every later sync sample, mid-stream `flush`, and a full pass after resets; forced mid-stream failures (`VtDecoder::fail_after`) at five points continue with the software decoder's exact output; seeded mutation of samples and parameter sets (bit flips, truncation, corrupt length prefixes) never panics or hangs; HEVC 4:2:2 10-bit is bit-exact with ffmpeg's decode | | `tests/fallback.rs` (every OS) | `HybridDecoder` with a stand-in hardware decoder failing after N samples (every sync sample ± a few, first / last sample, after a seek): output identical to the software decoder; in-band parameter sets identical to the sample entry's stay in hardware, different ones switch to software | | `tests/setting.rs` | Hardware decoding Off gives the software decoder through `make_video_decoder` and the media stack (no hardware frames); Auto gives VideoToolbox where available | +| `tests/hardware_encode.rs` (macOS) | what the hardware path takes and declines; round trip through our software decoder (every picture, in order, luma PSNR above 30 dB, keyframes no further apart than asked, no composition offsets); an export through `filmcraft_export` that decodes in our decoder and in ffmpeg / ffprobe (profile, size, frame count, BT.709); the built-in encoder still exporting everything hardware declines; exact output size at sizes that are not multiples of 16; hostile configurations (zero, huge, odd sizes, frame rates, bitrates, keyframe intervals, wrong planes) give errors and never panic; encoders dropped at any point do not crash or hang | Fixtures are made with ffmpeg into `target/fixtures/platform/` (generator only, never linked); tests skip without ffmpeg or without a hardware decoder. ## Performance +Hardware H.264 encoding, Apple M1 (8 cores), single runs: 1080p25 camera footage through the +encoder alone 200 fps at 4, 8 and 16 Mb/s (8× real time); a 9:29 timeline (camera clip, ProRes 4444 +overlays, AAC, loudness) exported in 311 s against 793 s with the built-in encoder (84 s against 395 s +for its densest 134 s), the outputs at SSIM 0.990 / PSNR 46.5 dB. Details in +[docs/performance.md](../../docs/performance.md). + +Hardware decoding, M4 Pro: + M4 Pro, load 150–190 (`cargo xtask bench --hw off|auto`): CPU per decoded frame H.264 2160p 119 → 4.2 ms, HEVC 2160p 86 → 3.7 ms; decode 35 → 107 fps and 49 → 217 fps; 4K H.264 and HEVC playback with no dropped frames at Full, 1/2 and 1/4. Details in @@ -65,5 +100,5 @@ playback with no dropped frames at Full, 1/2 and 1/4. Details in ## Not yet -Zero-copy upload of `CVPixelBuffer`s into wgpu textures; hardware encoding; Media Foundation / +Zero-copy upload of `CVPixelBuffer`s into wgpu textures; hardware HEVC encoding and B-frames; Media Foundation / D3D11 (Windows) and VA-API (Linux) decoders; field-coded H.264. diff --git a/crates/platform/src/hardware_encode.rs b/crates/platform/src/hardware_encode.rs new file mode 100644 index 00000000..96b72cc9 --- /dev/null +++ b/crates/platform/src/hardware_encode.rs @@ -0,0 +1,158 @@ +//! The VideoToolbox H.264 encoder behind `filmcraft_export::VideoEncoder` (macOS; safe code around +//! [`crate::videotoolbox_encode`]). +//! +//! Hardware encoding is opt-in per export (`ExportSettings::hardware_encoding`, `Auto`): its output depends +//! on the machine, while the built-in encoder's is byte-identical everywhere. The factory declines, +//! so the built-in encoder is used, for everything the hardware path does not take: other formats, +//! MXF (Annex B), two-pass VBR, HDR, odd picture sizes, non-square pixels, and any configuration +//! VideoToolbox cannot create a hardware session for. A hardware encoder that fails in the middle of an export reports +//! an error (the export stops); it cannot hand the stream over to another encoder. + +use filmcraft_export::{BitrateMode, EncodedPacket, EncoderFrame, ExportError, ExportSettings, Format, H264Pass, H264Profile, HardwareEncoding, VideoEncoder}; +use filmcraft_isobmff::{AvcConfig, SampleEntry}; +use filmcraft_time::FrameRate; + +use crate::videotoolbox_encode::{VtConfig, VtEncoder, VtPacket, VtProfile, VtRate}; + +/// The hardware encoder's configuration for an export, or why it is not used for it. +pub fn config_for(w: u32, h: u32, rate: FrameRate, s: &ExportSettings) -> Result { + if s.hardware_encoding != HardwareEncoding::Auto { + return Err("hardware encoding is off".into()); + } + if s.format.is_mxf() { + return Err("MXF carries an Annex B stream".into()); + } + if s.bitrate_mode == BitrateMode::Vbr2Pass || !matches!(s.h264_pass, H264Pass::Single) { + return Err("two-pass encoding".into()); + } + if s.signal.is_hdr() { + return Err("HDR".into()); + } + if !w.is_multiple_of(2) || !h.is_multiple_of(2) { + return Err("odd picture size".into()); + } + if s.pixel_aspect.is_some_and(|(n, d)| n != d) { + return Err("non-square pixels".into()); + } + let timescale = u32::try_from(rate.num).ok().filter(|n| *n > 0).ok_or("invalid frame rate")?; + let frame_duration = u32::try_from(rate.den).ok().filter(|d| *d > 0).ok_or("invalid frame rate")?; + let kbps = s.bitrate_kbps.max(100); + let max_kbps = s.max_bitrate_kbps.filter(|m| *m >= kbps).unwrap_or(kbps.saturating_mul(3) / 2); + let fps = f64::from(timescale) / f64::from(frame_duration); + Ok(VtConfig { + width: w, + height: h, + timescale, + frame_duration, + keyframe_interval: s.keyframe_distance.filter(|k| *k > 0).unwrap_or_else(|| (fps * 2.0).round().max(1.0) as u32), + profile: match s.h264_profile { + H264Profile::Baseline => VtProfile::Baseline, + H264Profile::Main => VtProfile::Main, + H264Profile::High => VtProfile::High, + }, + rate: match s.bitrate_mode { + BitrateMode::Cbr => VtRate::Cbr { kbps }, + _ => VtRate::Vbr { target_kbps: kbps, max_kbps }, + }, + }) +} + +/// The encoder factory `register` puts in front of the built-in ones: a hardware H.264 encoder when +/// the export asks for one and the OS takes the configuration, `None` (the built-in encoder) otherwise. +pub fn videotoolbox_encoder_factory( + format: Format, + w: u32, + h: u32, + rate: FrameRate, + s: &ExportSettings, +) -> Option>> { + if format != Format::H264 || s.hardware_encoding != HardwareEncoding::Auto { + return None; + } + let config = match config_for(w, h, rate, s) { + Ok(c) => c, + Err(why) => { + log::info!("hardware H.264 encoding declined: {why}"); + return None; + } + }; + match VtEncoder::new(config) { + Ok(vt) => Some(Ok(Box::new(HardwareH264Encoder { vt, y: Vec::new(), u: Vec::new(), v: Vec::new(), decoded: 0 }))), + Err(why) => { + log::info!("hardware H.264 encoding declined: {why}"); + None + } + } +} + +/// [`VtEncoder`] as a `VideoEncoder`: RGBA pictures in, MP4 samples out. +pub struct HardwareH264Encoder { + vt: VtEncoder, + y: Vec, + u: Vec, + v: Vec, + /// Compressed frames returned so far. + decoded: i64, +} + +impl HardwareH264Encoder { + /// MP4 samples from compressed frames. Without frame reordering every frame is presented at + /// its decode time, so composition offsets are 0; a frame whose time is not `n × duration` + /// after the first (a dropped or reordered frame) is an error, never a file with wrong times. + fn samples(&mut self, packets: Vec) -> filmcraft_export::Result> { + let duration = self.vt.config().frame_duration; + let mut out = Vec::with_capacity(packets.len()); + for p in packets { + let n = self.decoded; + self.decoded += 1; + if n.checked_mul(i64::from(duration)) != Some(p.pts) || p.pts != p.dts { + return Err(ExportError::Encode(format!( + "VideoToolbox returned frame {n} at time {} (decode time {}), expected {}", + p.pts, + p.dts, + n.saturating_mul(i64::from(duration)) + ))); + } + out.push(EncodedPacket { data: p.data, key: p.key, duration, composition_offset: 0 }); + } + Ok(out) + } +} + +impl VideoEncoder for HardwareH264Encoder { + fn sample_entry(&self) -> SampleEntry { + let c = self.vt.config(); + let (sps, pps) = self.vt.parameter_sets().map(|p| (p.sps, p.pps)).unwrap_or_default(); + SampleEntry::avc(AvcConfig::new(sps, pps, 4), c.width.min(u32::from(u16::MAX)) as u16, c.height.min(u32::from(u16::MAX)) as u16) + } + + fn timescale(&self) -> u32 { + self.vt.config().timescale + } + + fn encode(&mut self, f: &EncoderFrame) -> filmcraft_export::Result> { + let c = self.vt.config(); + if f.hdr.is_some() { + return Err(ExportError::Unsupported("hardware H.264 encoding does not take HDR pictures".into())); + } + if (f.width, f.height) != (c.width, c.height) { + return Err(ExportError::Encode(format!("picture is {}x{}, the encoder was created for {}x{}", f.width, f.height, c.width, c.height))); + } + let (w, h) = (f.width as usize, f.height as usize); + if f.rgba.len() < w.saturating_mul(h).saturating_mul(4) { + return Err(ExportError::Encode("the RGBA picture is smaller than its size".into())); + } + filmcraft_export::rgba_to_yuv420_8(f.rgba, w, h, &mut self.y, &mut self.u, &mut self.v); + let packets = self.vt.encode(f.index, &self.y, &self.u, &self.v).map_err(ExportError::Encode)?; + // the muxer needs the parameter sets after the first group of pictures + if f.index == 0 && self.vt.parameter_sets().is_none() { + return Err(ExportError::Encode("VideoToolbox returned no parameter sets for the first frame".into())); + } + self.samples(packets) + } + + fn flush(&mut self) -> filmcraft_export::Result> { + let packets = self.vt.flush().map_err(ExportError::Encode)?; + self.samples(packets) + } +} diff --git a/crates/platform/src/lib.rs b/crates/platform/src/lib.rs index 5a2a6802..81f27b24 100644 --- a/crates/platform/src/lib.rs +++ b/crates/platform/src/lib.rs @@ -1,9 +1,12 @@ -//! OS media integration (layer L5): hardware video decoding through the operating system's codecs. +//! OS media integration (layer L5): hardware video decoding and encoding through the operating +//! system's codecs. //! //! [`register`] puts the platform's hardware decoder factory in front of FilmCraft's own decoders //! (`filmcraft_codecs::register_video_decoder`). Today that is VideoToolbox on macOS for H.264 //! (`avcC`) and HEVC (`hvcC`) streams, 8- and 10-bit, 4:2:0 and 4:2:2; on other systems -//! registration does nothing and reports [`Availability::Unavailable`]. +//! registration does nothing and reports [`Availability::Unavailable`]. It also registers a +//! hardware H.264 encoder factory (`filmcraft_export::register_encoder`) that only acts when an +//! export asks for it (`ExportSettings::hardware_encoding` = `Auto`), see [`hardware_encode`]. //! //! Hardware decoding never makes a file undecodable: //! @@ -22,10 +25,15 @@ #![cfg_attr(not(test), deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable))] +#[cfg(target_os = "macos")] +pub mod hardware_encode; pub mod hybrid; #[cfg(target_os = "macos")] #[allow(unsafe_code)] pub mod videotoolbox; +#[cfg(target_os = "macos")] +#[allow(unsafe_code)] +pub mod videotoolbox_encode; pub use hybrid::HybridDecoder; @@ -45,6 +53,7 @@ pub fn register() -> Availability { #[cfg(target_os = "macos")] { filmcraft_codecs::register_video_decoder(videotoolbox_factory); + filmcraft_export::register_encoder(hardware_encode::videotoolbox_encoder_factory); Availability::Available("VideoToolbox") } #[cfg(not(target_os = "macos"))] diff --git a/crates/platform/src/videotoolbox_encode.rs b/crates/platform/src/videotoolbox_encode.rs new file mode 100644 index 00000000..48b35135 --- /dev/null +++ b/crates/platform/src/videotoolbox_encode.rs @@ -0,0 +1,572 @@ +//! VideoToolbox (macOS) hardware H.264 encoding. +//! +//! An FFI module of this crate (docs/adr/0001-platform-ffi.md): every `unsafe` block has a +//! `// SAFETY:` comment, no panic may unwind into VideoToolbox (the output callback runs under +//! `catch_unwind`), and every failure is a `Result` the caller answers by using the built-in +//! encoder instead. +//! +//! A compression session is created with a hardware encoder *required*, so a machine without one +//! is declined up front. Frame reordering (B-frames) is off, so compressed frames come out in +//! presentation order with decode time = presentation time. +//! +//! Pictures go in as 8-bit 4:2:0 (NV12) pixel buffers filled from planar BT.709 limited-range +//! Y'CbCr, the layout of `filmcraft_export::rgba_to_yuv420_8`, so the colours match the built-in +//! encoder's. VideoToolbox encodes asynchronously; its output callback (on a VideoToolbox thread) +//! copies each compressed frame (length-prefixed NAL units) into a queue that [`VtEncoder::encode`] +//! and [`VtEncoder::flush`] drain. +//! +//! The first frame is completed straight away: the SPS / PPS the container needs come with the +//! first compressed frame, and the muxer asks for them after the first group of frames. + +use std::ffi::{c_int, c_void}; +use std::panic::AssertUnwindSafe; +use std::ptr::NonNull; +use std::sync::{Mutex, PoisonError}; + +use objc2_core_foundation::{CFArray, CFBoolean, CFDictionary, CFNumber, CFRetained, CFString, CFType}; +use objc2_core_media::{CMSampleBuffer, CMTime, CMTimeFlags, CMVideoFormatDescriptionGetH264ParameterSetAtIndex, kCMVideoCodecType_H264}; +use objc2_core_video::{ + CVImageBuffer, CVPixelBufferCreate, CVPixelBufferGetBaseAddressOfPlane, CVPixelBufferGetBytesPerRowOfPlane, CVPixelBufferGetHeightOfPlane, + CVPixelBufferGetWidthOfPlane, CVPixelBufferLockBaseAddress, CVPixelBufferLockFlags, CVPixelBufferPool, CVPixelBufferUnlockBaseAddress, + kCVImageBufferColorPrimaries_ITU_R_709_2, kCVImageBufferTransferFunction_ITU_R_709_2, kCVImageBufferYCbCrMatrix_ITU_R_709_2, kCVPixelBufferHeightKey, + kCVPixelBufferIOSurfacePropertiesKey, kCVPixelBufferPixelFormatTypeKey, kCVPixelBufferWidthKey, +}; +use objc2_video_toolbox::{ + VTCompressionSession, VTEncodeInfoFlags, VTSessionSetProperty, kVTCompressionPropertyKey_AllowFrameReordering, kVTCompressionPropertyKey_AllowOpenGOP, + kVTCompressionPropertyKey_AverageBitRate, kVTCompressionPropertyKey_ColorPrimaries, kVTCompressionPropertyKey_ConstantBitRate, + kVTCompressionPropertyKey_DataRateLimits, kVTCompressionPropertyKey_ExpectedFrameRate, kVTCompressionPropertyKey_MaxKeyFrameInterval, + kVTCompressionPropertyKey_ProfileLevel, kVTCompressionPropertyKey_RealTime, kVTCompressionPropertyKey_TransferFunction, + kVTCompressionPropertyKey_YCbCrMatrix, kVTProfileLevel_H264_Baseline_AutoLevel, kVTProfileLevel_H264_High_AutoLevel, kVTProfileLevel_H264_Main_AutoLevel, + kVTVideoEncoderSpecification_RequireHardwareAcceleratedVideoEncoder, +}; + +/// The pixel format of the buffers we hand over: biplanar Y + interleaved CbCr, video range. +const NV12_VIDEO: u32 = u32::from_be_bytes(*b"420v"); +/// Largest compressed frame accepted from the callback (a sanity cap on a size we do not control). +const MAX_FRAME_BYTES: usize = 256 << 20; +/// Largest picture the hardware encoder is asked for. +const MAX_SIDE: u32 = 8192; + +/// H.264 profiles VideoToolbox offers. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum VtProfile { + Baseline, + Main, + High, +} + +/// How the bitrate is controlled (kilobits per second). +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum VtRate { + /// Constant bitrate. + Cbr { kbps: u32 }, + /// Average bitrate with a ceiling over one second. + Vbr { target_kbps: u32, max_kbps: u32 }, +} + +/// What to encode and how. Times are in a timescale of `timescale` units per second, one frame +/// lasting `frame_duration` units (the frame rate's numerator and denominator). +#[derive(Clone, Debug)] +pub struct VtConfig { + /// Picture size in luma samples: even, at most 8192 each way. + pub width: u32, + pub height: u32, + pub timescale: u32, + pub frame_duration: u32, + /// Frames between keyframes (every keyframe is an IDR picture). + pub keyframe_interval: u32, + pub profile: VtProfile, + pub rate: VtRate, +} + +/// One compressed frame in decoding order, which is presentation order (times in the config's timescale). +#[derive(Clone, Debug)] +pub struct VtPacket { + /// Length-prefixed NAL units (4-byte lengths, as in an `avcC` MP4 sample). + pub data: Vec, + pub key: bool, + pub pts: i64, + pub dts: i64, +} + +/// The parameter sets of the stream (NAL units without length prefix), for the `avcC` box. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct ParamSets { + pub sps: Vec>, + pub pps: Vec>, +} + +/// One output-callback result. +enum Output { + Packet(VtPacket), + Failed(String), +} + +/// State shared with the output callback (its `outputCallbackRefCon`). Boxed by the encoder and +/// kept alive until the session is invalidated. +struct Shared { + timescale: i64, + out: Mutex>, + params: Mutex>, +} + +impl Shared { + fn push(&self, o: Output) { + self.out.lock().unwrap_or_else(PoisonError::into_inner).push(o); + } +} + +/// A compression session with its callback state. Dropping it completes the pending frames and +/// invalidates the session before the callback state is freed. +struct Session { + session: CFRetained, + /// Pointed to by the session's callback: must outlive `session`. + shared: Box, +} + +// SAFETY: a VTCompressionSession and the CoreMedia / CoreVideo objects it uses may be used from +// any thread (Apple: "thread safe"); the encoder owning a `Session` is used by one thread at a time +// (`&mut self`), and the callback state is behind `Mutex`es. +unsafe impl Send for Session {} + +impl Drop for Session { + fn drop(&mut self) { + // SAFETY: the session is valid until released. Completing the frames lets any callback in + // flight finish; invalidating guarantees none runs afterwards, so `shared` can be freed + // when this function returns. + unsafe { + let _ = self.session.complete_frames(invalid_time()); + self.session.invalidate(); + } + } +} + +/// `kCMTimeInvalid`: "all pending frames" for `complete_frames`. +fn invalid_time() -> CMTime { + CMTime { value: 0, timescale: 0, flags: CMTimeFlags(0), epoch: 0 } +} + +fn cm_time(value: i64, timescale: u32) -> CMTime { + CMTime { value, timescale: timescale.min(i32::MAX as u32) as i32, flags: CMTimeFlags::Valid, epoch: 0 } +} + +/// `t` in units of `1 / timescale` s, rounded to nearest; None for an invalid or non-numeric time. +fn ticks(t: CMTime, timescale: i64) -> Option { + let (value, scale, flags) = (t.value, t.timescale, t.flags); + if !flags.contains(CMTimeFlags::Valid) || flags.contains(CMTimeFlags::PositiveInfinity) || flags.contains(CMTimeFlags::NegativeInfinity) || scale <= 0 { + return None; + } + let (num, den) = (i128::from(value) * i128::from(timescale), i128::from(scale)); + i64::try_from((2 * num + den).div_euclid(2 * den)).ok() +} + +/// Whether length-prefixed NAL units hold an IDR slice (nal_unit_type 5). +fn has_idr(data: &[u8]) -> Result { + let mut pos = 0usize; + let mut idr = false; + while pos < data.len() { + let len_bytes = data.get(pos..pos + 4).ok_or("truncated NAL length")?; + let len = u32::from_be_bytes([len_bytes[0], len_bytes[1], len_bytes[2], len_bytes[3]]) as usize; + pos += 4; + let end = pos.checked_add(len).filter(|e| *e <= data.len()).ok_or("NAL unit longer than the frame")?; + let header = *data.get(pos).ok_or("empty NAL unit")?; + if header & 0x1f == 5 { + idr = true; + } + pos = end; + } + Ok(idr) +} + +/// SPS / PPS of the first compressed frame's format description. +fn read_param_sets(sample: &CMSampleBuffer) -> Result { + // SAFETY: `sample` is a valid sample buffer for the duration of the callback. + let format = unsafe { sample.format_description() }.ok_or("compressed frame without a format description")?; + let (mut sps, mut pps) = (Vec::new(), Vec::new()); + let (mut index, mut count) = (0usize, 1usize); + while index < count && index < 16 { + let (mut ptr, mut size, mut total, mut nal_len): (*const u8, usize, usize, c_int) = (std::ptr::null(), 0, 0, 0); + // SAFETY: the out-pointers are valid for the call; the returned pointer is only read + // while `format` (retained above) is alive. + let status = unsafe { CMVideoFormatDescriptionGetH264ParameterSetAtIndex(&format, index, &mut ptr, &mut size, &mut total, &mut nal_len) }; + if status != 0 { + return Err(format!("cannot read the H.264 parameter sets ({status})")); + } + if nal_len != 4 { + return Err(format!("unexpected NAL length size {nal_len}")); + } + count = total; + if ptr.is_null() || size == 0 || size > 65_535 { + return Err("empty or oversized parameter set".into()); + } + // SAFETY: VideoToolbox returned `size` readable bytes at `ptr`, owned by `format`. + let bytes = unsafe { std::slice::from_raw_parts(ptr, size) }.to_vec(); + match bytes.first().map(|b| b & 0x1f) { + Some(7) => sps.push(bytes), + Some(8) => pps.push(bytes), + _ => {} + } + index += 1; + } + if sps.is_empty() || pps.is_empty() { + return Err("the format description has no SPS / PPS".into()); + } + Ok(ParamSets { sps, pps }) +} + +/// One compressed frame: its bytes, key flag and timestamps (and, once, the parameter sets). +fn read_packet(sample: &CMSampleBuffer, shared: &Shared) -> Result { + // SAFETY: `sample` is a valid sample buffer for the duration of the callback. + let block = unsafe { sample.data_buffer() }.ok_or("compressed frame without data")?; + // SAFETY: `block` is a valid block buffer (retained above). + let len = unsafe { block.data_length() }; + if len == 0 || len > MAX_FRAME_BYTES { + return Err(format!("compressed frame of {len} bytes")); + } + let mut data = vec![0u8; len]; + let dest = NonNull::new(data.as_mut_ptr().cast::()).ok_or("empty frame buffer")?; + // SAFETY: `dest` points to `len` writable bytes, the buffer's own length. + let status = unsafe { block.copy_data_bytes(0, len, dest) }; + if status != 0 { + return Err(format!("cannot copy a compressed frame ({status})")); + } + // SAFETY: valid sample buffer, see above. + let (pts, dts) = unsafe { (sample.presentation_time_stamp(), sample.decode_time_stamp()) }; + let pts = ticks(pts, shared.timescale).ok_or("compressed frame without a presentation time")?; + let dts = ticks(dts, shared.timescale).unwrap_or(pts); + let key = has_idr(&data)?; + { + let mut params = shared.params.lock().unwrap_or_else(PoisonError::into_inner); + if params.is_none() { + *params = Some(read_param_sets(sample)?); + } + } + Ok(VtPacket { data, key, pts, dts }) +} + +/// The output callback. Never unwinds: panics are caught and reported as a failed frame. +unsafe extern "C-unwind" fn output_callback( + refcon: *mut c_void, + _frame_refcon: *mut c_void, + status: i32, + flags: VTEncodeInfoFlags, + sample: *mut CMSampleBuffer, +) { + if refcon.is_null() { + return; + } + // SAFETY: `refcon` is the `Shared` the session was created with; it lives in a `Box` owned by + // the `Session`, which invalidates the session (no more callbacks) before freeing it. + let shared = unsafe { &*(refcon as *const Shared) }; + let result = std::panic::catch_unwind(AssertUnwindSafe(|| { + if status != 0 { + return Output::Failed(format!("VideoToolbox encode error {status}")); + } + if flags.contains(VTEncodeInfoFlags::FrameDropped) { + return Output::Failed("VideoToolbox dropped a frame".into()); + } + let Some(sample) = NonNull::new(sample) else { + return Output::Failed("VideoToolbox returned no data for a frame".into()); + }; + // SAFETY: VideoToolbox passes a valid sample buffer that stays alive for the duration of + // the callback; we only borrow it here. + let sample = unsafe { sample.as_ref() }; + match read_packet(sample, shared) { + Ok(p) => Output::Packet(p), + Err(e) => Output::Failed(e), + } + })); + shared.push(result.unwrap_or_else(|_| Output::Failed("panic while reading a compressed frame".into()))); +} + +/// Unlocks a pixel buffer's base address when dropped. +struct Locked<'a>(&'a CVImageBuffer); + +impl Drop for Locked<'_> { + fn drop(&mut self) { + // SAFETY: `fill` locked the buffer with the same flags, and it is still alive (borrowed). + unsafe { CVPixelBufferUnlockBaseAddress(self.0, CVPixelBufferLockFlags(0)) }; + } +} + +/// One writable plane of a locked pixel buffer. +fn plane_mut<'l>(pb: &CVImageBuffer, _lock: &'l mut Locked<'_>, i: usize) -> Result<(&'l mut [u8], usize, usize, usize), String> { + let base = CVPixelBufferGetBaseAddressOfPlane(pb, i) as *mut u8; + let stride = CVPixelBufferGetBytesPerRowOfPlane(pb, i); + let (width, height) = (CVPixelBufferGetWidthOfPlane(pb, i), CVPixelBufferGetHeightOfPlane(pb, i)); + if base.is_null() || stride == 0 || height == 0 { + return Err(format!("plane {i} of the input buffer is not mapped")); + } + let len = stride.checked_mul(height).ok_or("plane size overflows")?; + // SAFETY: the buffer's base address is locked for writing while `_lock` lives and the returned + // slice borrows it ('l); CoreVideo maps `bytes_per_row * height` bytes from each plane's base. + let data = unsafe { std::slice::from_raw_parts_mut(base, len) }; + Ok((data, stride, width, height)) +} + +/// Copy planar Y / U / V (`w`×`h` luma, `⌈w/2⌉`×`⌈h/2⌉` chroma, tightly packed) into an NV12 buffer. +fn fill(pb: &CVImageBuffer, w: usize, h: usize, y: &[u8], u: &[u8], v: &[u8]) -> Result<(), String> { + let (cw, ch) = (w.div_ceil(2), h.div_ceil(2)); + if y.len() < w * h || u.len() < cw * ch || v.len() < cw * ch { + return Err("the Y'CbCr planes are smaller than the picture".into()); + } + // SAFETY: `pb` is a valid pixel buffer (the caller holds it retained). + if unsafe { CVPixelBufferLockBaseAddress(pb, CVPixelBufferLockFlags(0)) } != 0 { + return Err("cannot lock the input buffer".into()); + } + let mut lock = Locked(pb); + let (luma, ls, lw, lh) = plane_mut(pb, &mut lock, 0)?; + if lw < w || lh < h || ls < w { + return Err("the input buffer's luma plane is smaller than the picture".into()); + } + for (row, src) in luma.chunks_mut(ls).zip(y.chunks_exact(w)).take(h) { + row[..w].copy_from_slice(src); + } + let (chroma, cs, cpw, cph) = plane_mut(pb, &mut lock, 1)?; + if cpw < cw || cph < ch || cs < cw * 2 { + return Err("the input buffer's chroma plane is smaller than the picture".into()); + } + for ((row, us), vs) in chroma.chunks_mut(cs).zip(u.chunks_exact(cw)).zip(v.chunks_exact(cw)).take(ch) { + for ((pair, &cb), &cr) in row.as_chunks_mut::<2>().0.iter_mut().zip(us).zip(vs) { + pair[0] = cb; + pair[1] = cr; + } + } + drop(lock); + Ok(()) +} + +/// A hardware H.264 encoder: feed pictures, collect compressed frames (in presentation order, which +/// is decoding order since nothing is reordered). +pub struct VtEncoder { + session: Session, + config: VtConfig, + pool: Option>, + attrs: CFRetained>, + /// Pictures submitted so far. + submitted: u64, +} + +// SAFETY: the session and its callback state are `Send` (see `Session`); the dictionary of source +// buffer attributes is never mutated after creation and CoreFoundation dictionaries and +// CoreVideo pixel buffer pools are thread-safe; the encoder is used by one thread at a time +// (`&mut self`), so moving it to another thread (the export job's) is sound. +unsafe impl Send for VtEncoder {} + +fn set_bool(session: &VTCompressionSession, key: &CFString, on: bool) -> i32 { + set_value(session, key, CFBoolean::new(on).as_ref()) +} + +fn set_value(session: &VTCompressionSession, key: &CFString, value: &CFType) -> i32 { + // SAFETY: the session is valid, `key` is a framework constant and `value` a live CF object. + unsafe { VTSessionSetProperty(session.as_ref(), key, Some(value)) } +} + +/// Fail with the property's name when VideoToolbox refuses it. +fn require(status: i32, what: &str) -> Result<(), String> { + if status == 0 { Ok(()) } else { Err(format!("VideoToolbox refused {what} ({status})")) } +} + +impl VtEncoder { + /// Create the session, or say why hardware encoding is not used for this configuration. + pub fn new(config: VtConfig) -> Result { + let (w, h) = (config.width, config.height); + if w == 0 || h == 0 || w > MAX_SIDE || h > MAX_SIDE { + return Err(format!("picture size {w}x{h}")); + } + if !w.is_multiple_of(2) || !h.is_multiple_of(2) { + // 4:2:0 H.264 crops in units of two samples: an odd size would come out one sample smaller + return Err(format!("odd picture size {w}x{h}")); + } + if config.timescale == 0 || config.frame_duration == 0 || config.timescale > i32::MAX as u32 { + return Err("invalid frame rate".into()); + } + let shared = Box::new(Shared { timescale: i64::from(config.timescale), out: Mutex::new(Vec::new()), params: Mutex::new(None) }); + + // SAFETY: reading immutable framework constants. + let require_key: &CFString = unsafe { kVTVideoEncoderSpecification_RequireHardwareAcceleratedVideoEncoder }; + let spec = CFDictionary::::from_slices(&[require_key], &[CFBoolean::new(true).as_ref()]); + + let (pf, cw, ch) = (CFNumber::new_i32(NV12_VIDEO as i32), CFNumber::new_i32(w as i32), CFNumber::new_i32(h as i32)); + let io = CFDictionary::::from_slices(&[], &[]); + // SAFETY: reading immutable framework constants. + let (pf_key, w_key, h_key, io_key): (&CFString, &CFString, &CFString, &CFString) = + unsafe { (kCVPixelBufferPixelFormatTypeKey, kCVPixelBufferWidthKey, kCVPixelBufferHeightKey, kCVPixelBufferIOSurfacePropertiesKey) }; + let attrs = CFDictionary::::from_slices(&[pf_key, w_key, h_key, io_key], &[pf.as_ref(), cw.as_ref(), ch.as_ref(), io.as_ref()]); + + let mut out: *mut VTCompressionSession = std::ptr::null_mut(); + // SAFETY: every pointer is valid for the call (VideoToolbox copies what it keeps); the + // callback's refcon points into `shared`, which the returned `Session` keeps alive until + // the session is invalidated. + let status = unsafe { + VTCompressionSession::create( + None, + w as i32, + h as i32, + kCMVideoCodecType_H264, + Some(spec.as_ref()), + Some(attrs.as_ref()), + None, + Some(output_callback), + &*shared as *const Shared as *mut c_void, + NonNull::from(&mut out), + ) + }; + let session = NonNull::new(out).filter(|_| status == 0).ok_or_else(|| format!("VTCompressionSessionCreate failed ({status})"))?; + // SAFETY: a created session is returned retained (+1); `CFRetained` takes that reference over. + let session = Session { session: unsafe { CFRetained::from_raw(session) }, shared }; + Self::configure(&session.session, &config)?; + // SAFETY: the session is valid. Preparing allocates the encoder's resources now, so a + // configuration the hardware cannot take fails here and not on the first frame. + require(unsafe { session.session.prepare_to_encode_frames() }, "to prepare the encoder")?; + // SAFETY: the session is valid; the pool (if any) is returned retained. + let pool = unsafe { session.session.pixel_buffer_pool() }; + Ok(VtEncoder { session, config, pool, attrs, submitted: 0 }) + } + + fn configure(session: &VTCompressionSession, c: &VtConfig) -> Result<(), String> { + // SAFETY: reading immutable framework constants. + let (profile_key, profile) = unsafe { + ( + kVTCompressionPropertyKey_ProfileLevel, + match c.profile { + VtProfile::Baseline => kVTProfileLevel_H264_Baseline_AutoLevel, + VtProfile::Main => kVTProfileLevel_H264_Main_AutoLevel, + VtProfile::High => kVTProfileLevel_H264_High_AutoLevel, + }, + ) + }; + require(set_value(session, profile_key, profile.as_ref()), "the profile")?; + // SAFETY: reading immutable framework constants. + let (realtime, reorder, open_gop) = + unsafe { (kVTCompressionPropertyKey_RealTime, kVTCompressionPropertyKey_AllowFrameReordering, kVTCompressionPropertyKey_AllowOpenGOP) }; + require(set_bool(session, realtime, false), "offline encoding")?; + // No B-frames: on real 1080p footage the hardware encoder's quality is the same with and + // without them (±0.3 dB at equal bitrate), the bitrate lands closer to the target, and + // decode and presentation order coincide, so the muxer needs no composition offsets or edit list. + require(set_bool(session, reorder, false), "frame reordering")?; + // closed GOPs: every keyframe is an IDR picture, so every MP4 sync sample is a clean seek point + let _ = set_bool(session, open_gop, false); + + let keyint = c.keyframe_interval.clamp(1, i32::MAX as u32) as i32; + // SAFETY: reading an immutable framework constant. + let keyint_key = unsafe { kVTCompressionPropertyKey_MaxKeyFrameInterval }; + require(set_value(session, keyint_key, CFNumber::new_i32(keyint).as_ref()), "the keyframe interval")?; + // SAFETY: reading an immutable framework constant. + let fps_key = unsafe { kVTCompressionPropertyKey_ExpectedFrameRate }; + let fps = f64::from(c.timescale) / f64::from(c.frame_duration); + let _ = set_value(session, fps_key, CFNumber::new_f64(fps).as_ref()); + + // SAFETY: reading immutable framework constants. + let (avg_key, limits_key, cbr_key) = + unsafe { (kVTCompressionPropertyKey_AverageBitRate, kVTCompressionPropertyKey_DataRateLimits, kVTCompressionPropertyKey_ConstantBitRate) }; + let bps = |kbps: u32| i32::try_from(u64::from(kbps).saturating_mul(1000)).unwrap_or(i32::MAX); + match c.rate { + VtRate::Cbr { kbps } => { + // true constant bitrate where the OS offers it (macOS 13+); otherwise an average + // with a one-second ceiling at the same rate + if set_value(session, cbr_key, CFNumber::new_i32(bps(kbps)).as_ref()) != 0 { + require(set_value(session, avg_key, CFNumber::new_i32(bps(kbps)).as_ref()), "the bitrate")?; + let limits = CFArray::::from_objects(&[CFNumber::new_i32(bps(kbps) / 8).as_ref(), CFNumber::new_f64(1.0).as_ref()]); + let _ = set_value(session, limits_key, limits.as_ref()); + } + } + VtRate::Vbr { target_kbps, max_kbps } => { + require(set_value(session, avg_key, CFNumber::new_i32(bps(target_kbps)).as_ref()), "the bitrate")?; + let limits = + CFArray::::from_objects(&[CFNumber::new_i32(bps(max_kbps.max(target_kbps)) / 8).as_ref(), CFNumber::new_f64(1.0).as_ref()]); + let _ = set_value(session, limits_key, limits.as_ref()); + } + } + + // BT.709, like the pictures we convert: written to the stream's VUI + // SAFETY: reading immutable framework constants. + let (pk, tk, mk, pv, tv, mv): (&CFString, &CFString, &CFString, &CFString, &CFString, &CFString) = unsafe { + ( + kVTCompressionPropertyKey_ColorPrimaries, + kVTCompressionPropertyKey_TransferFunction, + kVTCompressionPropertyKey_YCbCrMatrix, + kCVImageBufferColorPrimaries_ITU_R_709_2, + kCVImageBufferTransferFunction_ITU_R_709_2, + kCVImageBufferYCbCrMatrix_ITU_R_709_2, + ) + }; + let _ = set_value(session, pk, pv.as_ref()); + let _ = set_value(session, tk, tv.as_ref()); + let _ = set_value(session, mk, mv.as_ref()); + Ok(()) + } + + /// The configuration this encoder was created with. + pub fn config(&self) -> &VtConfig { + &self.config + } + + /// The stream's SPS / PPS, known once the first compressed frame came out. + pub fn parameter_sets(&self) -> Option { + self.session.shared.params.lock().unwrap_or_else(PoisonError::into_inner).clone() + } + + fn new_buffer(&self) -> Result, String> { + let mut raw: *mut CVImageBuffer = std::ptr::null_mut(); + let status = match &self.pool { + // SAFETY: the pool is valid and `raw` is a valid out-pointer. + Some(pool) => unsafe { CVPixelBufferPool::create_pixel_buffer(None, pool, NonNull::from(&mut raw)) }, + // SAFETY: `attrs` is a valid dictionary of CF types and `raw` a valid out-pointer. + None => unsafe { + CVPixelBufferCreate( + None, + self.config.width as usize, + self.config.height as usize, + NV12_VIDEO, + Some(self.attrs.as_ref()), + NonNull::from(&mut raw), + ) + }, + }; + let raw = NonNull::new(raw).filter(|_| status == 0).ok_or_else(|| format!("cannot allocate an input buffer ({status})"))?; + // SAFETY: both creation functions return the buffer retained (+1). + Ok(unsafe { CFRetained::from_raw(raw) }) + } + + /// Submit picture number `index` (planar BT.709 limited-range Y'CbCr, tightly packed) and return + /// the compressed frames that came out meanwhile. + pub fn encode(&mut self, index: u64, y: &[u8], u: &[u8], v: &[u8]) -> Result, String> { + let buffer = self.new_buffer()?; + fill(&buffer, self.config.width as usize, self.config.height as usize, y, u, v)?; + let value = i64::try_from(index).ok().and_then(|i| i.checked_mul(i64::from(self.config.frame_duration))).ok_or("frame index out of range")?; + let (pts, duration) = (cm_time(value, self.config.timescale), cm_time(i64::from(self.config.frame_duration), self.config.timescale)); + // SAFETY: the session and buffer are valid (VideoToolbox retains the buffer as long as it + // needs it); no frame properties, refcon or info-flags out-pointer. + let status = unsafe { self.session.session.encode_frame(&buffer, pts, duration, None, std::ptr::null_mut(), std::ptr::null_mut()) }; + if status != 0 { + return Err(format!("VTCompressionSessionEncodeFrame failed ({status})")); + } + self.submitted += 1; + if self.submitted == 1 { + // the first compressed frame carries the SPS / PPS the container needs + // SAFETY: the session is valid. + require(unsafe { self.session.session.complete_frames(pts) }, "to finish the first frame")?; + } + self.drain() + } + + /// Finish every pending frame and return them. + pub fn flush(&mut self) -> Result, String> { + // SAFETY: the session is valid; an invalid time means "all pending frames". + require(unsafe { self.session.session.complete_frames(invalid_time()) }, "to finish the pending frames")?; + self.drain() + } + + fn drain(&mut self) -> Result, String> { + let outputs = std::mem::take(&mut *self.session.shared.out.lock().unwrap_or_else(PoisonError::into_inner)); + let mut packets = Vec::with_capacity(outputs.len()); + for o in outputs { + match o { + Output::Packet(p) => packets.push(p), + Output::Failed(why) => return Err(why), + } + } + Ok(packets) + } +} diff --git a/crates/platform/tests/hardware_encode.rs b/crates/platform/tests/hardware_encode.rs new file mode 100644 index 00000000..eeed379a --- /dev/null +++ b/crates/platform/tests/hardware_encode.rs @@ -0,0 +1,433 @@ +//! VideoToolbox H.264 encoding (macOS): what the hardware path takes and declines, a round trip +//! through our own software decoder (picture quality, frame count and order, keyframes, the +//! timestamps the muxer needs), a full export through `filmcraft_export` that decodes, the fallback +//! to the built-in encoder for everything hardware does not take, and hostile configurations that +//! must give errors and never crash. Tests that need a hardware encoder skip without one. +#![cfg(target_os = "macos")] + +use std::panic::AssertUnwindSafe; +use std::sync::Arc; + +use filmcraft_export::{ + BitrateMode, EncodedPacket, EncoderFrame, ExportSettings, Format, H264Pass, H264Profile, HardwareEncoding, Progress, VideoEncoder, export, rgba_to_yuv420_8, +}; +use filmcraft_frame::PixelData; +use filmcraft_isobmff::{CodecConfig, SampleEntry}; +use filmcraft_media::generators::GeneratorSource; +use filmcraft_media::{DemoScene, MediaSource}; +use filmcraft_platform::hardware_encode::{config_for, videotoolbox_encoder_factory}; +use filmcraft_platform::videotoolbox_encode::{VtConfig, VtEncoder, VtProfile, VtRate}; +use filmcraft_project::{ItemKind, Label, MediaClip, MediaRef, Project, SequenceSettings, TrackKind}; +use filmcraft_render::SourceMap; +use filmcraft_time::{FrameRate, Tick, TimeRange}; + +const W: u32 = 640; +const H: u32 = 360; +const FPS: FrameRate = FrameRate::FPS_30; + +/// A moving test picture: gradients plus a bar travelling across. +fn picture(i: u64) -> Vec { + let (w, h) = (W as usize, H as usize); + let mut px = vec![255u8; w * h * 4]; + let bar = (i as usize * 7) % (w - 40); + for y in 0..h { + for x in 0..w { + let o = (y * w + x) * 4; + let on_bar = x >= bar && x < bar + 40 && y > h / 3 && y < 2 * h / 3; + px[o] = if on_bar { 255 } else { ((x * 255 / w + i as usize * 3) % 256) as u8 }; + px[o + 1] = if on_bar { 255 } else { (y * 255 / h) as u8 }; + px[o + 2] = if on_bar { 255 } else { (((x + y) / 2 + i as usize * 5) % 256) as u8 }; + } + } + px +} + +fn hardware_settings() -> ExportSettings { + ExportSettings { format: Format::H264, hardware_encoding: HardwareEncoding::Auto, bitrate_kbps: 8_000, keyframe_distance: Some(30), ..Default::default() } +} + +/// The hardware encoder for the settings, or `None` (skip) when this machine has none. +fn hardware(s: &ExportSettings, w: u32, h: u32) -> Option> { + match videotoolbox_encoder_factory(Format::H264, w, h, FPS, s) { + Some(Ok(e)) => Some(e), + Some(Err(e)) => panic!("factory error: {e}"), + None => { + eprintln!("SKIPPED: no VideoToolbox hardware H.264 encoder for these settings"); + None + } + } +} + +fn encode_all(enc: &mut Box, frames: u64) -> Vec { + let mut packets = Vec::new(); + for i in 0..frames { + let rgba = picture(i); + packets.extend(enc.encode(&EncoderFrame { width: W, height: H, rgba: &rgba, hdr: None, index: i }).expect("encode")); + } + packets.extend(enc.flush().expect("flush")); + packets +} + +/// Decode `packets` (decoding order) with our software decoder: the pictures in presentation order. +fn decode(entry: &SampleEntry, packets: &[EncodedPacket]) -> Vec<(i64, Vec)> { + let mut dec = filmcraft_codecs::software_video_decoder(entry).expect("software decoder"); + let mut out = Vec::new(); + let mut dts = 0i64; + let take = |frames: Vec, out: &mut Vec<(i64, Vec)>| { + for f in frames { + let PixelData::Yuv8 { planes, .. } = &f.frame.data else { panic!("expected 8-bit planes") }; + out.push((f.pts, planes[0].to_vec())); + } + }; + for p in packets { + take(dec.decode(&p.data, dts + i64::from(p.composition_offset)).expect("decode"), &mut out); + dts += i64::from(p.duration); + } + take(dec.flush(), &mut out); + out.sort_by_key(|(pts, _)| *pts); + out +} + +fn psnr(a: &[u8], b: &[u8]) -> f64 { + assert_eq!(a.len(), b.len()); + let mse = a.iter().zip(b).map(|(x, y)| (f64::from(*x) - f64::from(*y)).powi(2)).sum::() / a.len() as f64; + if mse == 0.0 { 99.0 } else { 10.0 * (255.0f64 * 255.0 / mse).log10() } +} + +#[test] +fn declines_what_the_hardware_path_does_not_take() { + let ok = hardware_settings(); + assert!(config_for(W, H, FPS, &ok).is_ok()); + let declined = |s: &ExportSettings, what: &str| assert!(config_for(W, H, FPS, s).is_err(), "{what} must not use the hardware encoder"); + declined(&ExportSettings { hardware_encoding: HardwareEncoding::Off, ..ok.clone() }, "hardware encoding off"); + declined(&ExportSettings { bitrate_mode: BitrateMode::Vbr2Pass, ..ok.clone() }, "two-pass VBR"); + declined(&ExportSettings { h264_pass: H264Pass::First, ..ok.clone() }, "an analysis pass"); + declined(&ExportSettings { signal: filmcraft_export::ColorSignal::PQ, ..ok.clone() }, "HDR"); + declined(&ExportSettings { pixel_aspect: Some((4, 3)), ..ok.clone() }, "non-square pixels"); + declined(&ExportSettings { format: Format::MxfOp1a, ..ok.clone() }, "MXF"); + assert!(config_for(W, H, FrameRate { num: 0, den: 1 }, &ok).is_err()); + assert!(config_for(W, H, FrameRate { num: 30, den: 0 }, &ok).is_err()); + assert!(config_for(W, H, FrameRate { num: -30, den: 1 }, &ok).is_err()); + // square pixels, explicit or implied, are fine + assert!(config_for(W, H, FPS, &ExportSettings { pixel_aspect: Some((1, 1)), ..ok.clone() }).is_ok()); + // the configuration follows the settings + let c = + config_for(W, H, FPS, &ExportSettings { h264_profile: H264Profile::Main, bitrate_mode: BitrateMode::Cbr, bitrate_kbps: 5000, ..ok.clone() }).unwrap(); + assert_eq!((c.profile, c.rate, c.keyframe_interval), (VtProfile::Main, VtRate::Cbr { kbps: 5000 }, 30)); + let c = config_for(W, H, FPS, &ExportSettings { keyframe_distance: None, max_bitrate_kbps: Some(20_000), ..ok }).unwrap(); + assert_eq!((c.rate, c.keyframe_interval), (VtRate::Vbr { target_kbps: 8000, max_kbps: 20_000 }, 60)); +} + +#[test] +fn factory_only_acts_when_asked() { + let s = hardware_settings(); + for format in [Format::ProRes, Format::DnxHr, Format::Mjpeg, Format::PngSequence] { + assert!(videotoolbox_encoder_factory(format, W, H, FPS, &s).is_none(), "{format:?}"); + } + assert!(videotoolbox_encoder_factory(Format::H264, W, H, FPS, &ExportSettings { hardware_encoding: HardwareEncoding::Off, ..s }).is_none()); +} + +#[test] +fn round_trip_through_our_software_decoder() { + let s = hardware_settings(); + let Some(mut enc) = hardware(&s, W, H) else { return }; + let n = 75u64; + let packets = encode_all(&mut enc, n); + let entry = enc.sample_entry(); + let CodecConfig::Avc(avc) = &entry.codec else { panic!("expected an AVC sample entry, got {}", entry.codec.name()) }; + assert!(!avc.sps.is_empty() && !avc.pps.is_empty()); + assert_eq!(avc.length_size, 4); + assert_eq!(enc.timescale(), 30); + + // one compressed frame per picture, the first a keyframe, keyframes no further apart than asked + assert_eq!(packets.len() as u64, n); + assert!(packets[0].key); + assert!(packets.iter().all(|p| p.duration == 1 && p.composition_offset >= 0 && !p.data.is_empty())); + let keys: Vec = packets.iter().enumerate().filter(|(_, p)| p.key).map(|(i, _)| i).collect(); + assert!(keys.windows(2).all(|w| w[1] - w[0] <= 30), "keyframes at {keys:?}"); + + // no frame reordering: every frame is presented at its decode time (the muxer numbers decode + // times n × duration), so there are no composition offsets and no edit list + assert!(packets.iter().all(|p| p.composition_offset == 0), "composition offsets {:?}", packets.iter().map(|p| p.composition_offset).collect::>()); + assert_eq!(enc.media_start(), None); + + // our decoder reproduces every picture, in order, at a good quality + let decoded = decode(&entry, &packets); + assert_eq!(decoded.len() as u64, n); + assert_eq!(decoded.iter().map(|(p, _)| *p).collect::>(), (0..n as i64).collect::>()); + let (mut y, mut u, mut v) = (Vec::new(), Vec::new(), Vec::new()); + let mut worst = 99.0f64; + for (i, (_, luma)) in decoded.iter().enumerate() { + rgba_to_yuv420_8(&picture(i as u64), W as usize, H as usize, &mut y, &mut u, &mut v); + worst = worst.min(psnr(&y, luma)); + } + assert!(worst > 30.0, "worst luma PSNR {worst:.1} dB"); + eprintln!("hardware H.264: {} bytes for {n} frames, worst luma PSNR {worst:.1} dB", packets.iter().map(|p| p.data.len()).sum::()); +} + +/// A project with an animated demo clip as the whole sequence. +fn project(frames: i64) -> (Arc, filmcraft_project::ItemId, SourceMap) { + let mut p = Project::new("hw"); + let g = GeneratorSource::demo(DemoScene::Plasma); + let info = g.info().clone(); + let item = p.add_item( + &info.name.clone(), + Label::Iris, + ItemKind::Media(MediaClip { + media: MediaRef::Generator(g.generator.clone()), + info, + interpret: Default::default(), + mark_in: None, + mark_out: None, + markers: vec![], + offline: false, + proxy: None, + identity: None, + }), + None, + ); + let seq = p.new_sequence("s", SequenceSettings { width: W, height: H, frame_rate: FPS, ..Default::default() }, 1, 1, None); + let mut clip = p.make_track_item(item, TrackKind::Video, Tick::ZERO, TimeRange::new(Tick::ZERO, FPS.tick_of(frames)), FPS).unwrap(); + clip.scale_to_frame = true; + p.sequence_mut(seq).unwrap().video_tracks[0].items.push(clip); + let mut m = SourceMap::default(); + m.0.insert(item, Arc::new(g)); + (Arc::new(p), seq, m) +} + +fn tmp(name: &str) -> String { + let d = std::env::temp_dir().join(format!("fc-hw-encode-{}", std::process::id())); + std::fs::create_dir_all(&d).unwrap(); + d.join(name).to_string_lossy().to_string() +} + +/// (sample count, decoded picture count, sync sample count) of an MP4's video track, decoded by our decoder. +fn inspect(path: &str) -> (usize, usize, usize) { + let bytes = std::fs::read(path).unwrap(); + let file = filmcraft_isobmff::open(bytes.as_slice()).unwrap(); + let vt = file.track_of_kind(filmcraft_isobmff::TrackKind::Video).unwrap(); + let track = &file.tracks[vt]; + let entry = track.entries.first().expect("a sample entry"); + let CodecConfig::Avc(_) = &entry.codec else { panic!("not AVC") }; + let mut dec = filmcraft_codecs::software_video_decoder(entry).expect("software decoder"); + let mut pictures = 0; + for (i, s) in track.samples.iter().enumerate() { + let data = file.read_sample(bytes.as_slice(), vt, i).unwrap(); + pictures += dec.decode(&data, s.pts).expect("decode").len(); + } + pictures += dec.flush().len(); + (track.samples.len(), pictures, track.samples.iter().filter(|s| s.is_sync).count()) +} + +#[test] +fn export_through_the_pipeline_with_hardware_encoding() { + filmcraft_platform::register(); + let path = tmp("hw.mp4"); + let s = ExportSettings { path: path.clone(), include_audio: false, ..hardware_settings() }; + // is the hardware encoder available at all on this machine? + if hardware(&s, W, H).is_none() { + return; + } + let frames = 70; + let (p, seq, m) = project(frames); + let report = export(&p, seq, &s, &m, &Progress::default()).expect("export"); + assert_eq!(report.frames, frames as u64); + let (samples, pictures, syncs) = inspect(&path); + assert_eq!((samples, pictures), (frames as usize, frames as usize)); + assert!(syncs >= 3, "keyframes every 30 frames: {syncs} sync samples"); +} + +#[test] +fn what_hardware_does_not_take_still_exports_with_the_built_in_encoder() { + filmcraft_platform::register(); + let (frames, (p, seq, m)) = (30, project(30)); + for (name, settings) in [ + ("two-pass", ExportSettings { bitrate_mode: BitrateMode::Vbr2Pass, ..hardware_settings() }), + ("off", ExportSettings { hardware_encoding: HardwareEncoding::Off, ..hardware_settings() }), + ("odd-aspect", ExportSettings { pixel_aspect: Some((4, 3)), ..hardware_settings() }), + ] { + let path = tmp(&format!("{name}.mp4")); + let s = ExportSettings { path: path.clone(), include_audio: false, ..settings }; + export(&p, seq, &s, &m, &Progress::default()).unwrap_or_else(|e| panic!("{name}: {e}")); + let (samples, pictures, _) = inspect(&path); + assert_eq!((samples, pictures), (frames, frames), "{name}"); + } +} + +/// Run `f` and require that it neither panics nor hangs the test. +fn calm(what: &str, f: impl FnOnce() -> R) -> R { + std::panic::catch_unwind(AssertUnwindSafe(f)).unwrap_or_else(|_| panic!("{what} panicked")) +} + +#[test] +fn hostile_configurations_give_errors_not_crashes() { + let base = VtConfig { + width: W, + height: H, + timescale: 30, + frame_duration: 1, + keyframe_interval: 30, + profile: VtProfile::High, + rate: VtRate::Vbr { target_kbps: 8000, max_kbps: 12_000 }, + }; + let variants: Vec<(&str, VtConfig)> = vec![ + ("zero width", VtConfig { width: 0, ..base.clone() }), + ("zero height", VtConfig { height: 0, ..base.clone() }), + ("huge width", VtConfig { width: u32::MAX, ..base.clone() }), + ("huge height", VtConfig { height: 100_000, ..base.clone() }), + ("1x1", VtConfig { width: 1, height: 1, ..base.clone() }), + ("odd size", VtConfig { width: 321, height: 181, ..base.clone() }), + ("odd width", VtConfig { width: 321, ..base.clone() }), + ("zero timescale", VtConfig { timescale: 0, ..base.clone() }), + ("zero frame duration", VtConfig { frame_duration: 0, ..base.clone() }), + ("huge timescale", VtConfig { timescale: u32::MAX, ..base.clone() }), + ("zero keyframe interval", VtConfig { keyframe_interval: 0, ..base.clone() }), + ("huge keyframe interval", VtConfig { keyframe_interval: u32::MAX, ..base.clone() }), + ("zero bitrate", VtConfig { rate: VtRate::Vbr { target_kbps: 0, max_kbps: 0 }, ..base.clone() }), + ("huge bitrate", VtConfig { rate: VtRate::Vbr { target_kbps: u32::MAX, max_kbps: u32::MAX }, ..base.clone() }), + ("max below target", VtConfig { rate: VtRate::Vbr { target_kbps: 8000, max_kbps: 1 }, ..base.clone() }), + ("zero cbr", VtConfig { rate: VtRate::Cbr { kbps: 0 }, ..base.clone() }), + ("baseline", VtConfig { profile: VtProfile::Baseline, ..base.clone() }), + ("main", VtConfig { profile: VtProfile::Main, ..base.clone() }), + ("cbr", VtConfig { rate: VtRate::Cbr { kbps: 6000 }, ..base.clone() }), + ]; + for (name, config) in variants { + match calm(name, || VtEncoder::new(config)) { + // when a hardware session exists, pictures of the wrong size or planes must be refused + Ok(mut enc) => { + let (w, h) = (enc.config().width as usize, enc.config().height as usize); + let (cw, ch) = (w.div_ceil(2), h.div_ceil(2)); + assert!(calm(name, || enc.encode(0, &[], &[], &[])).is_err(), "{name}: empty planes"); + assert!(calm(name, || enc.encode(0, &vec![0; w * h - 1], &vec![128; cw * ch], &vec![128; cw * ch])).is_err(), "{name}: short luma"); + let _ = calm(name, || enc.encode(0, &vec![16; w * h], &vec![128; cw * ch], &vec![128; cw * ch])); + let _ = calm(name, || enc.flush()); + } + Err(why) => eprintln!("{name}: declined ({why})"), + } + } +} + +#[test] +fn encoder_refuses_pictures_of_another_size() { + let s = hardware_settings(); + let Some(mut enc) = hardware(&s, W, H) else { return }; + let rgba = vec![0u8; 100 * 100 * 4]; + assert!(enc.encode(&EncoderFrame { width: 100, height: 100, rgba: &rgba, hdr: None, index: 0 }).is_err()); + let short = vec![0u8; 16]; + assert!(enc.encode(&EncoderFrame { width: W, height: H, rgba: &short, hdr: None, index: 0 }).is_err()); + let hdr = vec![0.0f32; (W * H * 3) as usize]; + assert!(enc.encode(&EncoderFrame { width: W, height: H, rgba: &vec![0u8; (W * H * 4) as usize], hdr: Some(&hdr), index: 0 }).is_err()); +} + +#[test] +fn encoders_can_be_dropped_at_any_point() { + let s = hardware_settings(); + for frames in [0u64, 1, 2, 5] { + let Some(mut enc) = hardware(&s, W, H) else { return }; + for i in 0..frames { + let rgba = picture(i); + let _ = calm("encode", || enc.encode(&EncoderFrame { width: W, height: H, rgba: &rgba, hdr: None, index: i })); + } + drop(enc); // pending frames, no flush: must not crash or hang + } +} + +#[test] +fn ffmpeg_decodes_what_the_hardware_encoder_writes() { + filmcraft_platform::register(); + let Some(ffmpeg) = filmcraft_testkit::ffmpeg_or_skip("hardware H.264 export") else { return }; + let Some(ffprobe) = filmcraft_testkit::ffprobe_or_skip("hardware H.264 export") else { return }; + let path = tmp("oracle.mp4"); + let s = ExportSettings { path: path.clone(), include_audio: false, ..hardware_settings() }; + if hardware(&s, W, H).is_none() { + return; + } + let frames = 65; + let (p, seq, m) = project(frames); + export(&p, seq, &s, &m, &Progress::default()).expect("export"); + // an independent decoder reads the whole file without a complaint + let out = std::process::Command::new(&ffmpeg).args(["-v", "error", "-i", &path, "-f", "null", "-"]).output().expect("run ffmpeg"); + assert!(out.status.success() && out.stderr.is_empty(), "ffmpeg: {}", String::from_utf8_lossy(&out.stderr)); + // and finds the stream we asked for + let out = std::process::Command::new(&ffprobe) + .args([ + "-v", + "error", + "-count_frames", + "-select_streams", + "v:0", + "-show_entries", + "stream=profile,width,height,nb_read_frames,has_b_frames,color_space", + "-of", + "default=nw=1", + &path, + ]) + .output() + .expect("run ffprobe"); + let info = String::from_utf8_lossy(&out.stdout).to_string(); + for want in ["profile=High", &format!("width={W}"), &format!("height={H}"), &format!("nb_read_frames={frames}"), "has_b_frames=0", "color_space=bt709"] { + assert!(info.contains(want), "ffprobe lacks `{want}`:\n{info}"); + } +} + +#[test] +fn pictures_keep_their_exact_size() { + let s = hardware_settings(); + // odd sizes cannot be cropped exactly in 4:2:0: the built-in encoder takes them + assert!(config_for(321, 181, FPS, &s).is_err() && config_for(640, 361, FPS, &s).is_err()); + assert!(videotoolbox_encoder_factory(Format::H264, 321, 181, FPS, &s).is_none()); + // even sizes that are not multiples of 16 come out at exactly their size + for (w, h) in [(640u32, 362u32), (642, 360), (1280, 720), (16, 16)] { + let Some(mut enc) = hardware(&s, w, h) else { continue }; + let rgba: Vec = (0..w * h * 4).map(|i| (i % 251) as u8).collect(); + let mut packets = Vec::new(); + for i in 0..6u64 { + packets.extend(enc.encode(&EncoderFrame { width: w, height: h, rgba: &rgba, hdr: None, index: i }).expect("encode")); + } + packets.extend(enc.flush().expect("flush")); + assert_eq!(packets.len(), 6, "{w}x{h}"); + let entry = enc.sample_entry(); + let mut dec = filmcraft_codecs::software_video_decoder(&entry).expect("software decoder"); + let mut pictures = Vec::new(); + for (i, p) in packets.iter().enumerate() { + pictures.extend(dec.decode(&p.data, i as i64).expect("decode")); + } + pictures.extend(dec.flush()); + assert_eq!(pictures.len(), 6, "{w}x{h}"); + assert!( + pictures.iter().all(|f| (f.frame.width, f.frame.height) == (w, h)), + "{w}x{h}: decoded {}x{}", + pictures[0].frame.width, + pictures[0].frame.height + ); + } +} + +#[test] +fn an_empty_range_never_leaves_a_broken_file() { + filmcraft_platform::register(); + let probe = ExportSettings { include_audio: false, ..hardware_settings() }; + if hardware(&probe, W, H).is_none() { + return; + } + for (name, hardware_encoding) in [("empty-hw", HardwareEncoding::Auto), ("empty-sw", HardwareEncoding::Off)] { + let path = tmp(&format!("{name}.mp4")); + let _ = std::fs::remove_file(&path); + let s = ExportSettings { path: path.clone(), range: Some(TimeRange::new(Tick::ZERO, Tick::ZERO)), hardware_encoding, ..probe.clone() }; + let (p, seq, m) = project(30); + let result = export(&p, seq, &s, &m, &Progress::default()); + let file = std::fs::read(&path).ok(); + eprintln!("{name}: {:?}, file {:?} bytes", result.as_ref().map(|r| r.frames).map_err(|e| e.to_string()), file.as_ref().map(Vec::len)); + match (result, file) { + // a clean error is fine; so is a file with no pictures that parses + (Err(_), _) => {} + (Ok(r), Some(bytes)) => { + assert_eq!(r.frames, 0, "{name}"); + let f = filmcraft_isobmff::open(bytes.as_slice()).unwrap_or_else(|e| panic!("{name}: the empty file does not parse: {e}")); + let vt = f.track_of_kind(filmcraft_isobmff::TrackKind::Video).expect("a video track"); + assert!(f.tracks[vt].samples.is_empty(), "{name}"); + } + (Ok(_), None) => panic!("{name}: reported success without a file"), + } + } +} diff --git a/crates/ui-egui/src/panels/export_mode.rs b/crates/ui-egui/src/panels/export_mode.rs index c9446a7f..9c278929 100644 --- a/crates/ui-egui/src/panels/export_mode.rs +++ b/crates/ui-egui/src/panels/export_mode.rs @@ -22,8 +22,8 @@ use egui::{Align2, Color32, Rect, Sense, pos2, vec2}; use filmcraft_engine::export::presets::{DEFAULT_PRESET, preset_key}; use filmcraft_engine::export::{ - AudioCodec, BitrateMode, ExportSettings, FieldOrder, Format, H264Profile, Multiplexer, MxfVideoCodec, Placement, Scaling, TextOverlay, builtin_presets, - format_bytes, + AudioCodec, BitrateMode, ExportSettings, FieldOrder, Format, H264Profile, HardwareEncoding, Multiplexer, MxfVideoCodec, Placement, Scaling, TextOverlay, + builtin_presets, format_bytes, }; use filmcraft_engine::time::{FrameRate, Tick}; use serde::{Deserialize, Serialize}; @@ -653,6 +653,14 @@ fn video_section(ui: &mut egui::Ui, reg: &mut Reg, s: &mut ExportSettings, t: &T *k = v as u32; } }); + // The operating system's hardware encoder where there is one (macOS); everything it does + // not take (two-pass, HDR, MXF) and every machine without one keeps the built-in encoder. + row(ui, t, "Hardware Encoding", |ui| { + let mut on = s.hardware_encoding == HardwareEncoding::Auto; + if check(ui, reg, "export.video.hardwareEncoding", &mut on, "Use the hardware encoder when available") { + s.hardware_encoding = if on { HardwareEncoding::Auto } else { HardwareEncoding::Off }; + } + }); } Format::ProRes => { row(ui, t, "Profile", |ui| { diff --git a/crates/ui-egui/tests/export_ui.rs b/crates/ui-egui/tests/export_ui.rs index ae15af44..4a1dc55a 100644 --- a/crates/ui-egui/tests/export_ui.rs +++ b/crates/ui-egui/tests/export_ui.rs @@ -141,6 +141,7 @@ fn export_mode_settings_and_summary() { "export.section.multiplexer", "export.video.matchSize", "export.video.bitrateMode", + "export.video.hardwareEncoding", "export.video.maxQuality", "export.audio.sampleRate", "export.range", diff --git a/docs/adr/0001-platform-ffi.md b/docs/adr/0001-platform-ffi.md index 7720cc5c..9458027b 100644 --- a/docs/adr/0001-platform-ffi.md +++ b/docs/adr/0001-platform-ffi.md @@ -63,3 +63,23 @@ Registering a hardware decoder never makes a file undecodable, and never changes operating system, which carries the licences, wherever hardware decoding is used. - A new review burden: changes to `crates/platform` need the same scrutiny as any FFI code. Anything else that wants `unsafe` needs a new decision, not an extension of this one. + +## Addendum (2026-10-06): hardware H.264 encoding + +The crate's second use of the FFI is the one this decision named, hardware encoding: a VideoToolbox +H.264 encoder behind `filmcraft_export::VideoEncoder` (`videotoolbox_encode`, with the same +containment rules; `hardware_encode` is safe code). It differs from decoding in two ways, both +deliberate: + +- **It is opt-in per export** (`ExportSettings::hardware_encoding`, `Off` by default). Exports from + the built-in encoder are byte-identical across machines; a hardware encoder's output depends on the + machine, so it must be asked for. +- **There is no mid-stream fallback.** A decoder can replay samples through another decoder; an + encoder cannot hand a half-written stream to another one. The factory declines up front (the + built-in encoder is used) for everything the hardware path does not take, and for any machine + where the OS cannot create a hardware session; a hardware encoder that fails during an export + stops it with an error. + +Everything else in this record applies unchanged: `unsafe` stays in `crates/platform`, every block +has a `// SAFETY:` comment, no panic crosses the FFI boundary, the public API is safe, and the crate +compiles everywhere (`register()` is a no-op off macOS). diff --git a/docs/architecture.md b/docs/architecture.md index 619e1c7d..22d02250 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -72,7 +72,7 @@ and `filmcraft-cli`. | `engine` | L4 | `Session`, command registry, undo history, media pool, jobs, interchange glue | | `ui-egui` | L5 | the egui frontend: docking, panels, timeline, monitors, playback, control-channel handlers | | `automation` | L5 | MCP server (`rmcp`, stdio), headless or bridged to the running app | -| `platform` | L5 | OS media FFI only: hardware video decoding (VideoToolbox H.264 / HEVC on macOS; a no-op elsewhere) behind `codecs::VideoDecoder`, with transparent fallback to our decoders. The one crate allowed `unsafe` ([ADR 0001](adr/0001-platform-ffi.md), [README](../crates/platform/README.md)) | +| `platform` | L5 | OS media FFI only: hardware video decoding (VideoToolbox H.264 / HEVC on macOS; a no-op elsewhere) behind `codecs::VideoDecoder`, with transparent fallback to our decoders, and opt-in hardware H.264 encoding (VideoToolbox) behind `export::VideoEncoder`, which declines to the built-in encoder for what it does not take. The one crate allowed `unsafe` ([ADR 0001](adr/0001-platform-ffi.md), [README](../crates/platform/README.md)) | | `filmcraft` | L6 | desktop binary: eframe/wgpu window, cpal audio output, file dialogs, native macOS menu, TCP control server | | `filmcraft-cli` | L6 | headless CLI: `exec`, `run`, `inspect`, `describe`, `commands`, `import`, `export`, `render`, `probe`, `mcp`; `--bridge` targets the running app | | `filmcraft-web` | L6 | the browser app (wasm32): eframe web runner on WebGPU/WebGL2, Blob-backed services, OPFS recovery, WebAudio, WebCodecs, `window.filmcraft` API ([web.md](web.md)) | diff --git a/docs/performance.md b/docs/performance.md index f284313b..1ba2e51b 100644 --- a/docs/performance.md +++ b/docs/performance.md @@ -125,6 +125,62 @@ the draft plan's box decimation of the planes, cheap next to software decoding, CPU cost. Zero-copy upload of the decoded `CVPixelBuffer` into wgpu (no copy-out at all) is the next step (issue #30). +## Results (HW2: VideoToolbox hardware H.264 encoding, built-in → hardware) + +Apple M1 (8 cores, 16 GB), 2026-10-06, single runs on an otherwise idle machine, release build. +Hardware encoding is opt-in per export (`hardware_encoding: auto`); the built-in encoder is unchanged. + +### The encoder alone (100 frames of 1920×1080 25 fps camera footage, High, one-pass VBR, keyframe every 50) + +Frames go in as planar Y'CbCr and out as compressed frames (NV12 fill, `VTCompressionSessionEncodeFrame`, +flush); quality is the luma PSNR of our software decoder's output against the input. + +| target | B-frames | real bitrate | luma PSNR | frames/s | +|---|---|---|---|---| +| 4 Mb/s | off | 3982 kb/s | 41.66 dB | 202 | +| 8 Mb/s | off | 7987 kb/s | 43.44 dB | 202 | +| 16 Mb/s | off | 15972 kb/s | 45.55 dB | 202 | +| 4 Mb/s | on | 4220 kb/s | 41.37 dB | 183 | +| 8 Mb/s | on | 8268 kb/s | 43.41 dB | 197 | +| 16 Mb/s | on | 16388 kb/s | 45.66 dB | 199 | + +B-frames buy nothing measurable here (±0.3 dB) and overshoot the target by up to 5 %, so the hardware +encoder runs without them: frames come out in presentation order and the muxer needs no composition +offsets or edit list. + +### Whole exports (1080p25 camera clip with ProRes 4444 overlays with alpha, AAC, loudness normalised to −14 LUFS) + +A 9:29 timeline (569 s, 14 227 frames): the camera clip, an intro, six banners, three subscribe bars +and an outro. + +| encoder | wall time | frames/s | vs real time | file | +|---|---|---|---|---| +| built-in (`filmcraft-h264enc`) | 793 s | 17.9 | 1.4× slower | 1445 MB, 20.0 Mb/s | +| hardware (VideoToolbox) | **311 s** | **45.7** | 1.8× faster | 1417 MB, 19.6 Mb/s | + +2.5× faster. The first 134 s of the same timeline, where the overlays are densest (intro, a banner, a +subscribe bar and the outro in 134 s): + +| encoder | wall time | frames/s | vs real time | file | +|---|---|---|---|---| +| built-in (`filmcraft-h264enc`) | 395 s | 8.5 | 2.9× slower | 340 MB, 20.0 Mb/s | +| hardware (VideoToolbox) | **84 s** | **39.9** | 1.6× faster | 312 MB, 18.3 Mb/s | + +4.7× faster there. The gain depends on what else an export spends its time on: with the encoder +off the CPU, what is left is decoding the camera clip, the CPU compositor (every layer, alpha +included) and the RGBA → Y'CbCr conversion. + +The pairs of files have the same frame count and duration, decode in ffmpeg without a message, and +compare at SSIM 0.990 / PSNR 46.7 dB over the 134 s (lowest frame: 45.1 dB) and SSIM 0.989–0.990 / +PSNR 46.3–46.6 dB at six 3-second spots of the full export. Audio is identical. + +Where the built-in export's time goes (`sample` on the process during a 20 s export, inclusive CPU +time in the call tree; the figures overlap because work-stealing mixes the tasks): about two thirds +of the busy CPU is in the encoder, about a third in compositing and layer decoding. Moving the encoder to +the media engine removes most of the first and lets the next batch render while the previous one +encodes (the exporter renders a batch of frames in parallel, then encodes it). What is left is the +CPU compositor, the next target of #30 ("GPU export"). + ## Results (GPU1: blend modes on the GPU compositor, #30, before → after) Before = this change with the old whole-frame CPU fallback for non-Normal blend modes put back