From 790b3a06978a99ad0ca89daaa0257c8f97e12390 Mon Sep 17 00:00:00 2001 From: Dario Carnelutti Date: Wed, 7 Oct 2026 14:00:08 -0600 Subject: [PATCH] platform: H.265 (HEVC) export with VideoToolbox Adds `Format::Hevc` (Export > Format > H.265 (HEVC); `"format": "hevc"` in `file.exportMedia`): HEVC Main, 8-bit 4:2:0, in MP4 (`hvc1`) or QuickTime, with AAC audio, encoded by the macOS hardware encoder. There is no built-in H.265 encoder (one in pure Rust is a project of its own, and x265 is GPL), so the format exists only where the OS has one. - `videotoolbox_encode`: the session wrapper creates HEVC sessions too (`VtProfile::HevcMain`; the profile also says which codec). Keyframes are the HEVC random access points (NAL types 16 to 21); the parameter sets are VPS / SPS / PPS; the `hvcC` record in the MP4 sample entry is the one VideoToolbox writes (sample description extension atoms), so profile, level and flags are the encoder's own. `VtEncoder::uses_hardware()` asks VideoToolbox whether the session runs on a hardware encoder (a test checks H.264 and HEVC), so a requirement dropped by accident could never become a silent software encode. - `hardware_encode`: one adapter for both codecs. Choosing the format is the opt-in (the `hardware_encoding` setting is for H.264). `hevc_available()` (one small hardware session, created once) is registered with the new `filmcraft_export::register_format_probe`, so `available(Format::Hevc)`, the format list of Export mode and `exportFormats` follow the machine. `Format::has_builtin_encoder` says which formats need such a probe. - `export`: `Format::is_h26x` where H.264 and H.265 behave alike (MP4 or QuickTime, AAC, size estimate, summary line). Two-pass VBR is refused up front ("H.265 export has no two-pass mode") and HDR sequences export as SDR, because there is no encoder to fall back to. - Export mode: H.265 shows H.264's multiplexer, audio, metadata and bitrate controls, without profile, level, two-pass or the hardware checkbox, and says "Encoder: Hardware". Apple M1 (8 cores), 1080p25, a 9:29 timeline (camera clip, ProRes 4444 overlays, AAC, loudness): hardware H.265 at 10 Mb/s 195 s, hardware H.264 at 20 Mb/s 204 s (CPU 1029 s and 1042 s; the same H.264 export took 193 s and 200 s on other runs). It is not faster: the codec that compresses further is the more expensive one (raw encoders: 166 and 189 frames/s), and the export is bound by the CPU compositor, not by the media engine. Quality per bit, 30 s of the same timeline against a ProRes 422 HQ export (average PSNR over the three planes, SSIM): 20 Mb/s 46.88 dB / 0.9902 (H.264) and 46.91 dB / 0.9903 (H.265); 10 Mb/s 44.96 / 0.9863 and 45.09 / 0.9863; 6 Mb/s 43.72 / 0.9833 and 44.31 / 0.9843; 4 Mb/s 42.91 / 0.9807 and 43.64 / 0.9825; 3 Mb/s 42.37 / 0.9786 and 43.05 / 0.9807. The same picture at 10 Mb/s and above; at 3 to 6 Mb/s H.265 needs 22 to 31 % less bitrate for the same PSNR. Both formats therefore keep the same default bitrate. Details in docs/performance.md (HW3). Tests: tests/hardware_encode.rs gets the HEVC counterparts of the H.264 ones (what is taken and declined, a round trip through our HEVC decoder with luma PSNR above 30 dB, keyframes, MP4 and QuickTime exports with AAC, ffmpeg / ffprobe reading `hevc` / `Main` / `hvc1` / `yuv420p` / BT.709, exact sizes, hostile configurations, encoders dropped at any point, the format list agreeing with the probe, sessions on the hardware encoder); settings_tests.rs checks names, ids, defaults, the summary, two-pass refusal and the probe; export_ui.rs the controls H.265 shows and hides. Stacked on the hardware H.264 encoder (the first commit of this branch). Co-Authored-By: Claude Sonnet 5.5 --- README.md | 2 +- ROADMAP.md | 6 +- apps/filmcraft-cli/src/main.rs | 2 +- crates/engine/src/commands.rs | 2 +- crates/export/src/job.rs | 6 +- crates/export/src/lib.rs | 53 +++- crates/export/src/settings.rs | 20 +- crates/export/src/settings_tests.rs | 52 ++++ crates/platform/README.md | 44 +++- crates/platform/src/hardware_encode.rs | 117 ++++++--- crates/platform/src/lib.rs | 1 + crates/platform/src/videotoolbox_encode.rs | 165 +++++++++--- crates/platform/tests/hardware_encode.rs | 292 ++++++++++++++++++++- crates/ui-egui/src/panels/export_mode.rs | 70 ++--- crates/ui-egui/tests/export_ui.rs | 32 +++ docs/adr/0001-platform-ffi.md | 9 + docs/architecture.md | 2 +- docs/performance.md | 37 +++ 18 files changed, 779 insertions(+), 133 deletions(-) diff --git a/README.md b/README.md index 0dd28e7b..f879b6c7 100644 --- a/README.md +++ b/README.md @@ -286,7 +286,7 @@ We track two numbers ([ROADMAP.md](ROADMAP.md#honest-assessment-2026-10-05)): The biggest gaps today: -- **Speed on big footage.** Hardware decoding works on macOS; Windows and Linux have no hardware path yet. Blend modes and the most common effects run on the GPU, but Lumetri, keys and export rendering still run on the CPU; H.264 encoding can use the hardware encoder on macOS (opt-in) ([#30](https://github.com/storytold/filmcraft/issues/30)). +- **Speed on big footage.** Hardware decoding works on macOS; Windows and Linux have no hardware path yet. Blend modes and the most common effects run on the GPU, but Lumetri, keys and export rendering still run on the CPU; H.264 encoding can use the hardware encoder on macOS (opt-in), and H.265 export is hardware-only there ([#30](https://github.com/storytold/filmcraft/issues/30)). - **No plugins.** No VST3 / Audio Units or OpenFX hosting. - **Delivery codecs.** H.264 is our only delivery-codec export; no HEVC or AV1 export yet. - **Real-world media and platforms.** Our decoders are bit-exact on conformance streams, but camera and phone files in the wild are less tested. Windows and Linux get far less testing than macOS. diff --git a/ROADMAP.md b/ROADMAP.md index 93a62fe1..fca8be25 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -36,7 +36,7 @@ before choosing work. | **Correctness in depth** | not tracked | **~60–70%** | A first-time contributor found four real bugs in core paths within hours (#5–#8: AIFF import failed, Slide left linked audio behind, transitions didn't follow ripple trims, export ignored start/end times). User reports: Color Matte always grey with no picker (#29); ALSA audio error on Linux (#23). Settings are "most wired", not all. | | **Codecs and media** | ~90% | **~75%** | Our decoders are bit-exact on conformance streams. Real camera and phone media (variable frame rate, damaged files, unusual containers) is far less tested. No camera RAW (RED, BRAW, ARRIRAW), no E-AC-3. | | **Export** | ~92% | **~70%** | Delivery: H.264 only (ProRes, DNxHR and image sequences cover mastering). No HEVC or AV1 export. AAF and OMF have never been validated in Avid Media Composer or Pro Tools. Export renders and encodes on the CPU only. | -| **Performance and hardware** | ~62% | **~35–40%** | Premiere runs effects, decoding and encoding on the GPU and the hardware media engines. Here, hardware decode works on macOS only (VideoToolbox, #33; active in the desktop app only since #41, so releases up to 0.2.1 decoded in software). The GPU does compositing, blend modes (#32) and 31 common effects (GPU2): colour adjustments, crop/flip/transform, blurs, sharpen. Lumetri, keys, masks, the remaining effects and export rendering are CPU; H.264 encoding can use VideoToolbox on macOS (opt-in per export, no B-frames). Linux and Windows have no hardware path. 8K and AV1 aren't real-time. The effect list shows Premiere's "GPU accelerated" badge, but only 31 effect ids run on the GPU here; the rest, Lumetri included, run on the CPU. | +| **Performance and hardware** | ~62% | **~35–40%** | Premiere runs effects, decoding and encoding on the GPU and the hardware media engines. Here, hardware decode works on macOS only (VideoToolbox, #33; active in the desktop app only since #41, so releases up to 0.2.1 decoded in software). The GPU does compositing, blend modes (#32) and 31 common effects (GPU2): colour adjustments, crop/flip/transform, blurs, sharpen. Lumetri, keys, masks, the remaining effects and export rendering are CPU; H.264 encoding can use VideoToolbox on macOS (opt-in per export, no B-frames), and H.265 (HEVC) export exists there only, through the same hardware encoder. Linux and Windows have no hardware path. 8K and AV1 aren't real-time. The effect list shows Premiere's "GPU accelerated" badge, but only 31 effect ids run on the GPU here; the rest, Lumetri included, run on the CPU. | | **Stability** | not tracked | **improving, unproven** | The never-crash pass (no panics in product code, last-resort guards) landed on 2026-10-04. There's no field record from real users' projects yet. | | **Plugins and pro ecosystem** | not tracked | **~0–10%** | No audio plugin hosting (VST3 / Audio Units) and no third-party video effects (OpenFX). For many professional editors this alone rules FilmCraft out. Team Projects and Productions are out of scope by design. | | **AI features** | partial | **~25–35%** | Speech to text exists but is off in default builds (the `whisper` feature). Enhance Speech is a DSP chain, not a model. Auto Reframe is approximate. No Generative Extend, and no media-intelligence search or auto colour. | @@ -51,7 +51,7 @@ In priority order. Agents should prefer this work over adding more checklist ite - hardware decode on Linux (VA-API) and Windows (Media Foundation); - zero-copy decoded frames into wgpu; - the remaining effects on the GPU: Lumetri, keys, Vignette, Video Limiter, masks (31 common effects are done); - - GPU export, then hardware encode: H.264 on macOS is in (opt-in); HEVC and other systems remain. + - GPU export, then hardware encode: H.264 and H.265 on macOS are in (H.264 opt-in; H.265 has no other encoder); other systems remain. 2. **A measured parity number.** Add `cargo xtask parity`, run in CI. It should cover menus, commands, effects, transitions, panels, preferences and formats against the Premiere snapshot in `plan/premiere/`. It should report presence and fidelity separately (exact, approximate, stub). @@ -63,7 +63,7 @@ In priority order. Agents should prefer this work over adding more checklist ite through XML / AAF. 5. **Windows and Linux at runtime.** CI that runs the test suite and a headless playback/export smoke test on both, not just release builds. Fix the Linux audio path (#23). -6. **Delivery codecs.** HEVC and AV1 export: hardware encoders first, and `rav1e` (BSD-2) for AV1. +6. **Delivery codecs.** HEVC export is in on macOS (VideoToolbox, 8-bit, no software encoder); still to do: HEVC elsewhere, 10-bit / HDR HEVC, and AV1 (`rav1e`, BSD-2). 7. **Plugin hosting** (VST3 / Audio Units, then OpenFX). Needs FFI, so it falls under the isolated `unsafe` crate rule (AGENTS.md §0.3). **Owner decision needed** before starting. 8. **AI features** with openly licensed local models: transcription on by default, text-based diff --git a/apps/filmcraft-cli/src/main.rs b/apps/filmcraft-cli/src/main.rs index 4753551a..66055b6b 100644 --- a/apps/filmcraft-cli/src/main.rs +++ b/apps/filmcraft-cli/src/main.rs @@ -30,7 +30,7 @@ SUBCOMMANDS [--scale f] [--quality 0-100] [--no-audio] [--queue] export the active sequence and wait for it to finish: with an export preset (`export --list-presets`; built-in or the user's), or - a format (h264|prores|dnxhr|apv|mjpeg|mxf-op1a|mxf-opatom|png|tiff|bmp|gif|wav|aiff, guessed + a format (h264|hevc|prores|dnxhr|apv|mjpeg|mxf-op1a|mxf-opatom|png|tiff|bmp|gif|wav|aiff, guessed from the extension); --range entire|inOut|workArea, or a custom range in seconds; --settings is ExportSettings JSON merged over the preset; --scale renders at a fraction of the frame size (0.5 = diff --git a/crates/engine/src/commands.rs b/crates/engine/src/commands.rs index dfc9744b..ee831eaa 100644 --- a/crates/engine/src/commands.rs +++ b/crates/engine/src/commands.rs @@ -1018,7 +1018,7 @@ fn build() -> Vec { "Media…", ["File", "Export"], None, - r#"{"path":str,"preset":str?,"settings":ExportSettings?,"format":"h264|prores|dnxhr|apv|mjpeg|mxf-op1a|mxf-opatom|png|tiff|bmp|gif|wav|aiff"?,"width":u32?,"height":u32?,"fps":f64?,"bitrateKbps":u32?,"bitrateMode":"cbr|vbr1Pass|vbr2Pass"?,"hardwareEncoding":"off|auto"?,"scale":f32=1,"audio":bool=true,"quality":0..100,"burnCaptions":bool=false,"captionSidecar":"srt|vtt"?,"loudnessLufs":f64?,"proresProfile":"proxy|lt|standard|hq"?,"dnxProfile":"lb|sq|hq|hqx"?,"apvProfile":"422-10|422-12|444-10|444-12"?,"mxfVideoCodec":"dnxhr|proRes|h264"?,"sequence":id?,"range":"entire|inOut|workArea|custom"?,"startSeconds":f64?,"endSeconds":f64?,"wait":bool=false}"#, + r#"{"path":str,"preset":str?,"settings":ExportSettings?,"format":"h264|hevc|prores|dnxhr|apv|mjpeg|mxf-op1a|mxf-opatom|png|tiff|bmp|gif|wav|aiff"?,"width":u32?,"height":u32?,"fps":f64?,"bitrateKbps":u32?,"bitrateMode":"cbr|vbr1Pass|vbr2Pass"?,"hardwareEncoding":"off|auto"?,"scale":f32=1,"audio":bool=true,"quality":0..100,"burnCaptions":bool=false,"captionSidecar":"srt|vtt"?,"loudnessLufs":f64?,"proresProfile":"proxy|lt|standard|hq"?,"dnxProfile":"lb|sq|hq|hqx"?,"apvProfile":"422-10|422-12|444-10|444-12"?,"mxfVideoCodec":"dnxhr|proRes|h264"?,"sequence":id?,"range":"entire|inOut|workArea|custom"?,"startSeconds":f64?,"endSeconds":f64?,"wait":bool=false}"#, has_seq, crate::export_tools::export_media ), diff --git a/crates/export/src/job.rs b/crates/export/src/job.rs index f127747e..c9ce7b8f 100644 --- a/crates/export/src/job.rs +++ b/crates/export/src/job.rs @@ -82,7 +82,7 @@ pub struct Exporter { /// Whether [`Exporter`] handles a format. pub fn stepped(format: Format) -> bool { - matches!(format, Format::H264 | Format::ProRes | Format::DnxHr | Format::Apv | Format::Mjpeg | Format::MxfOp1a | Format::MxfOpAtom) + matches!(format, Format::H264 | Format::Hevc | Format::ProRes | Format::DnxHr | Format::Apv | Format::Mjpeg | Format::MxfOp1a | Format::MxfOpAtom) } fn make_venc(settings: &ExportSettings, w: u32, h: u32, rate: FrameRate) -> Result> { @@ -102,7 +102,7 @@ impl Exporter { } settings.validate()?; let q = project.sequence(seq).ok_or(ExportError::NoSequence)?; - // HDR sequences export HDR (H.264 / ProRes / DNxHR / APV) unless SDR is asked for + // HDR sequences export HDR (H.264 / ProRes / DNxHR / APV) unless SDR is asked for (H.265 export is 8-bit SDR) let pipe = q.settings.color; let hdr_out = pipe.working.is_hdr() && !settings.sdr && matches!(settings.video_format(), Format::H264 | Format::ProRes | Format::DnxHr | Format::Apv); let mut settings = settings.clone(); @@ -128,7 +128,7 @@ impl Exporter { progress.set_status(format!("Exporting {} frames ({})", nframes, settings.format.label())); } let venc = make_venc(&settings, pipe.w, pipe.h, pipe.rate)?; - let brand = if settings.format == Format::H264 && settings.multiplexer == Multiplexer::Mp4 { Brand::Mp4 } else { Brand::Mov }; + let brand = if settings.format.is_h26x() && settings.multiplexer == Multiplexer::Mp4 { Brand::Mp4 } else { Brand::Mov }; let audio = if settings.has_audio() { Some(AudioOut::new(project.clone(), seq, &settings, range)?) } else { None }; let aenc: Option> = match (&audio, settings.audio_codec()) { (Some(a), AudioCodec::Aac) => { diff --git a/crates/export/src/lib.rs b/crates/export/src/lib.rs index af576cad..0f94ae3e 100644 --- a/crates/export/src/lib.rs +++ b/crates/export/src/lib.rs @@ -61,6 +61,11 @@ pub enum Format { #[default] #[serde(rename = "h264", alias = "H264")] H264, + /// MPEG-4 (or QuickTime, see [`Multiplexer`]), H.265 / HEVC Main (8-bit) video + AAC audio. There is no + /// built-in encoder: a platform hardware encoder registers one ([`register_encoder`]) and says so + /// with [`register_format_probe`]; [`available`] is false without it. + #[serde(rename = "hevc", alias = "Hevc")] + Hevc, /// QuickTime, Apple ProRes 422 (HQ unless the settings pick another flavour) + PCM. #[serde(rename = "prores", alias = "ProRes")] ProRes, @@ -102,6 +107,7 @@ impl Format { pub fn from_name(s: &str) -> Option { Some(match s.to_ascii_lowercase().replace([' ', '-', '_', '.'], "").as_str() { "h264" | "mp4" | "avc" | "m4v" => Format::H264, + "hevc" | "h265" | "hvc1" | "hev1" | "x265" => Format::Hevc, "prores" | "mov" | "appleprores" => Format::ProRes, "dnxhr" | "dnxhd" | "dnx" | "avid" | "aviddnxhr" | "aviddnxhd" | "vc3" => Format::DnxHr, "apv" | "apv1" => Format::Apv, @@ -121,6 +127,7 @@ impl Format { pub fn id(self) -> &'static str { match self { Format::H264 => "h264", + Format::Hevc => "hevc", Format::ProRes => "prores", Format::DnxHr => "dnxhr", Format::Apv => "apv", @@ -137,7 +144,7 @@ impl Format { } pub fn extension(self) -> &'static str { match self { - Format::H264 => "mp4", + Format::H264 | Format::Hevc => "mp4", Format::ProRes | Format::DnxHr | Format::Apv | Format::Mjpeg => "mov", Format::PngSequence => "png", Format::TiffSequence => "tif", @@ -151,6 +158,7 @@ impl Format { pub fn label(self) -> &'static str { match self { Format::H264 => "H.264", + Format::Hevc => "H.265 (HEVC)", Format::ProRes => "Apple ProRes", Format::DnxHr => "Avid DNxHR", Format::Apv => "APV", @@ -169,8 +177,18 @@ impl Format { pub fn is_mxf(self) -> bool { matches!(self, Format::MxfOp1a | Format::MxfOpAtom) } - pub const ALL: [Format; 13] = [ + /// Whether the crate carries an encoder for the format; the others need one registered at runtime + /// ([`register_format_probe`]). + pub fn has_builtin_encoder(self) -> bool { + self != Format::Hevc + } + /// H.264 or H.265: MPEG-4 (or QuickTime) with AAC audio, set up with the same bitrate controls. + pub fn is_h26x(self) -> bool { + matches!(self, Format::H264 | Format::Hevc) + } + pub const ALL: [Format; 14] = [ Format::H264, + Format::Hevc, Format::ProRes, Format::DnxHr, Format::Apv, @@ -461,6 +479,9 @@ impl ExportSettings { if self.effects.image_overlay.enabled && self.effects.image_overlay.path.trim().is_empty() { return Err(ExportError::Unsupported("image overlay: no image file chosen".into())); } + if self.format == Format::Hevc && self.bitrate_mode == BitrateMode::Vbr2Pass { + return Err(ExportError::Unsupported("H.265 export has no two-pass mode: choose CBR or VBR, 1 pass".into())); + } Ok(()) } } @@ -576,10 +597,30 @@ pub fn register_audio_encoder(f: AudioEncoderFactory) { audio_factories().write().unwrap_or_else(|e| e.into_inner()).insert(0, f); } -/// Whether a format can currently be exported. Every [`Format`] has a built-in encoder; this stays -/// as the hook for formats whose encoders are registered at runtime. +type FormatProbe = (Format, fn() -> bool); + +fn format_probes() -> &'static RwLock> { + static P: OnceLock>> = OnceLock::new(); + P.get_or_init(|| RwLock::new(Vec::new())) +} + +/// Say how to find out whether the encoder of a format without a built-in one ([`Format::has_builtin_encoder`]) +/// works on this machine: `probe` runs when [`available`] asks (it should cache its answer). +/// Registering the same probe twice is harmless. +pub fn register_format_probe(format: Format, probe: fn() -> bool) { + let mut g = format_probes().write().unwrap_or_else(|e| e.into_inner()); + if !g.iter().any(|(f, p)| *f == format && std::ptr::fn_addr_eq(*p, probe)) { + g.push((format, probe)); + } +} + +/// Whether a format can currently be exported: it has a built-in encoder, or a registered probe +/// says its encoder works here. pub fn available(format: Format) -> bool { - Format::ALL.contains(&format) + if !Format::ALL.contains(&format) { + return false; + } + format.has_builtin_encoder() || format_probes().read().unwrap_or_else(|e| e.into_inner()).iter().any(|(f, probe)| *f == format && probe()) } struct MjpegEncoder { @@ -1241,7 +1282,7 @@ pub fn export(project: &Arc, seq: ItemId, settings: &ExportSettings, so } (total, count) } - Format::H264 | Format::ProRes | Format::DnxHr | Format::Apv | Format::Mjpeg | Format::MxfOp1a | Format::MxfOpAtom => { + Format::H264 | Format::Hevc | Format::ProRes | Format::DnxHr | Format::Apv | Format::Mjpeg | Format::MxfOp1a | Format::MxfOpAtom => { // Handled by the stepped exporter above; reaching here would be a dispatch bug. return Err(ExportError::Unsupported(format!("{:?} must run as a stepped export", settings.format))); } diff --git a/crates/export/src/settings.rs b/crates/export/src/settings.rs index fd24d48b..4ced7f2e 100644 --- a/crates/export/src/settings.rs +++ b/crates/export/src/settings.rs @@ -405,8 +405,8 @@ impl ExportSettings { pub fn audio_codec(&self) -> AudioCodec { match (self.format, self.audio.codec) { (Format::Wav | Format::Aiff | Format::MxfOp1a | Format::MxfOpAtom, _) => AudioCodec::Pcm, - (Format::H264, _) if self.multiplexer == Multiplexer::Mp4 => AudioCodec::Aac, - (_, AudioCodec::Auto) if self.format == Format::H264 => AudioCodec::Aac, + (f, _) if f.is_h26x() && self.multiplexer == Multiplexer::Mp4 => AudioCodec::Aac, + (_, AudioCodec::Auto) if self.format.is_h26x() => AudioCodec::Aac, (_, AudioCodec::Auto) => AudioCodec::Pcm, (_, c) => c, } @@ -424,7 +424,7 @@ impl ExportSettings { /// File extension of the output. pub fn extension(&self) -> &'static str { - if self.format == Format::H264 && self.multiplexer == Multiplexer::Mov { "mov" } else { self.format.extension() } + if self.format.is_h26x() && self.multiplexer == Multiplexer::Mov { "mov" } else { self.format.extension() } } /// Estimated output size in bytes for `duration` of a sequence (`seq_w`×`seq_h` at `seq_rate`). @@ -434,7 +434,7 @@ impl ExportSettings { let fps = r.rate.num as f64 / r.rate.den as f64; let px = r.width as f64 * r.height as f64; let video_bps = match self.video_format() { - Format::H264 => r.target_kbps as f64 * 1000.0, + Format::H264 | Format::Hevc => r.target_kbps as f64 * 1000.0, Format::ProRes => crate::prores_profile(&self.prores_profile).nominal_mbps_1080p30() * 1e6 * px / (1920.0 * 1080.0) * fps / 29.97, Format::DnxHr => { // nominal 1080p29.97 data rates of the DNxHR profiles (Mb/s) @@ -507,6 +507,14 @@ impl ExportSettings { }; v += &format!(", keyframe every {} frames", r.keyint); } + Format::Hevc => { + v += &format!(", HEVC Main (hardware encoder), {}", self.bitrate_mode.label()); + v += &match self.bitrate_mode { + BitrateMode::Cbr => format!(", {}", mbps(r.target_kbps)), + _ => format!(", Target {}, Max {}", mbps(r.target_kbps), mbps(r.max_kbps)), + }; + v += &format!(", keyframe every {} frames", r.keyint); + } Format::ProRes => { use filmcraft_prores::Profile; v += match crate::prores_profile(&self.prores_profile) { @@ -537,8 +545,8 @@ impl ExportSettings { "No audio".to_string() }; let container = match self.format { - Format::H264 if self.multiplexer == Multiplexer::Mov => "QuickTime", - Format::H264 => "MP4", + Format::H264 | Format::Hevc if self.multiplexer == Multiplexer::Mov => "QuickTime", + Format::H264 | Format::Hevc => "MP4", Format::ProRes | Format::DnxHr | Format::Apv | Format::Mjpeg => "QuickTime", Format::MxfOp1a | Format::MxfOpAtom => self.mxf_video_codec.label(), Format::PngSequence | Format::TiffSequence | Format::BmpSequence => "Image sequence", diff --git a/crates/export/src/settings_tests.rs b/crates/export/src/settings_tests.rs index 4a7562fe..2c39c4b1 100644 --- a/crates/export/src/settings_tests.rs +++ b/crates/export/src/settings_tests.rs @@ -410,3 +410,55 @@ fn aac_bitrate_is_capped_for_low_sample_rates() { assert!(enc.is_ok(), "{rate} Hz × {ch}: {:?}", enc.err()); } } + +#[test] +fn h265_is_a_format_that_needs_a_registered_encoder() { + use std::sync::atomic::{AtomicBool, Ordering}; + // names, ids, files and serialized settings + for name in ["hevc", "H.265", "h265", "HVC1", "x265"] { + assert_eq!(Format::from_name(name), Some(Format::Hevc), "{name}"); + } + assert_eq!(Format::from_name(Format::Hevc.id()), Some(Format::Hevc)); + assert_eq!((Format::Hevc.extension(), Format::Hevc.label()), ("mp4", "H.265 (HEVC)")); + assert!(Format::ALL.contains(&Format::Hevc) && Format::ALL.len() == 14); + let s: ExportSettings = serde_json::from_value(serde_json::json!({"format": "hevc"})).unwrap(); + assert_eq!(s.format, Format::Hevc); + assert_eq!(serde_json::to_value(&s).unwrap()["format"], "hevc"); + + // it is H.264's family: MP4 with AAC, or QuickTime; the same size estimate + let h = ExportSettings { format: Format::Hevc, ..Default::default() }; + assert_eq!((h.audio_codec(), h.extension()), (AudioCodec::Aac, "mp4")); + assert_eq!(ExportSettings { multiplexer: Multiplexer::Mov, ..h.clone() }.extension(), "mov"); + // 10 s of 20 Mbps video + 320 kbps AAC ≈ 25.4 MB, like H.264 + let b = h.estimate_bytes(1920, 1080, FrameRate::FPS_25, 48_000, Tick(10 * TICKS_PER_SECOND)); + assert!((b as f64 - 25.4e6).abs() < 0.2e6, "{b}"); + let summary = h.summary(1920, 1080, FrameRate::FPS_25, 48_000, Tick(10 * TICKS_PER_SECOND)); + assert_eq!(summary.format, "H.265 (HEVC) (MP4)"); + assert!(summary.video.contains("HEVC Main") && summary.video.contains("Target 20.00 Mbps"), "{}", summary.video); + + // a hardware encoder has one pass + let e = ExportSettings { bitrate_mode: BitrateMode::Vbr2Pass, ..h.clone() }.validate().unwrap_err(); + assert!(e.to_string().contains("two-pass"), "{e}"); + assert!(ExportSettings { bitrate_mode: BitrateMode::Cbr, ..h.clone() }.validate().is_ok()); + + // formats with a built-in encoder are always available; this one only when a registered probe says so + assert!(Format::ALL.iter().filter(|f| f.has_builtin_encoder()).all(|f| available(*f))); + assert!(!Format::Hevc.has_builtin_encoder()); + static HERE: AtomicBool = AtomicBool::new(false); + fn probe() -> bool { + HERE.load(Ordering::SeqCst) + } + assert!(!available(Format::Hevc), "nothing registered an encoder for it"); + // and an export says so, honestly + let (p, seq, m) = matte([0.0, 0.0, 1.0, 1.0], 160, 90, None); + let sc = Scratch::new("h265-missing"); + let err = export(&p, seq, &ExportSettings { path: sc.path("x.mp4"), include_audio: false, ..h.clone() }, &m, &Progress::default()).unwrap_err().to_string(); + assert!(err.contains("H.265") && err.contains("encoder not available"), "{err}"); + register_format_probe(Format::Hevc, probe); + register_format_probe(Format::Hevc, probe); // the same probe twice is harmless + assert!(!available(Format::Hevc)); + HERE.store(true, Ordering::SeqCst); + assert!(available(Format::Hevc)); + HERE.store(false, Ordering::SeqCst); + assert!(!available(Format::Hevc)); +} diff --git a/crates/platform/README.md b/crates/platform/README.md index fd1d1bf8..61380c43 100644 --- a/crates/platform/README.md +++ b/crates/platform/README.md @@ -1,8 +1,8 @@ # filmcraft-platform OS media integration for FilmCraft (layer L5): hardware video decoding through the operating -system's codecs, behind `filmcraft_codecs::VideoDecoder`, and hardware H.264 encoding, behind -`filmcraft_export::VideoEncoder`. It holds OS media FFI and nothing else. +system's codecs, behind `filmcraft_codecs::VideoDecoder`, and hardware H.264 and H.265 (HEVC) +encoding, behind `filmcraft_export::VideoEncoder`. It holds OS media FFI and nothing else. It is the one crate of the workspace allowed to contain `unsafe`, under the rules of [ADR 0001](../../docs/adr/0001-platform-ffi.md) and [AGENTS.md](../../AGENTS.md) §0.3. @@ -57,6 +57,36 @@ let availability = filmcraft_platform::register(); // Available("VideoToolbox") - The first frame is completed straight away: the SPS / PPS the container needs come with the first compressed frame, and the muxer asks for them after the first group of pictures. +## Hardware H.265 (HEVC) encoding (macOS) + +The same session wrapper, created for `kCMVideoCodecType_HEVC` with the HEVC Main profile +(`VtProfile::HevcMain`: the profile also says which codec the session is). `Format::Hevc` has no +built-in encoder, which changes three things from H.264: + +- **Choosing the format is the opt-in.** No `hardware_encoding` setting: Export ▸ Format ▸ H.265 + (HEVC), or `"format": "hevc"` in `file.exportMedia` (`h265` is accepted too). The format is + listed as available only on a machine with a hardware HEVC encoder: `register()` hands + `hardware_encode::hevc_available` (one small hardware session, created on the first question) to + `filmcraft_export::register_format_probe`. +- **There is no fallback encoder.** What the hardware path does not take is an error, not a + different encoder: two-pass VBR is refused up front (`ExportSettings::validate`), HDR sequences + are exported as SDR (the H.265 path is 8-bit), and odd sizes, non-square pixels or a machine + without the encoder end in "H.265 (HEVC) encoder not available yet". + +The sessions are created with a hardware encoder *required*, and `VtEncoder::uses_hardware()` asks +VideoToolbox whether it agrees (`UsingHardwareAcceleratedVideoEncoder`; a test checks it for H.264 and +HEVC), so a requirement dropped by accident could never turn into a silent software encode. Export +mode says "Encoder: Hardware" for H.265, and the summary line "HEVC Main (hardware encoder)". + +What it takes is H.264's list: MP4 (`hvc1`, the tag QuickTime reads) or QuickTime, AAC +audio, 8-bit 4:2:0 BT.709 limited range, even sizes up to 8192, constant or one-pass variable +bitrate, the keyframe distance (closed GOPs, IDR pictures), and no B-frames. Keyframes are the +random access points of types 16–21 (BLA, IDR, CRA). + +The sample entry's `hvcC` record is the one VideoToolbox wrote for the stream (read from the +format description's sample description extension atoms, with the VPS / SPS / PPS), so profile, +level and flags are the encoder's own. If it is missing the export stops with that reason. + ## Guarantees - **Never undecodable:** the factory declines (returns `None`, so the software decoder is used) @@ -78,7 +108,7 @@ let availability = filmcraft_platform::register(); // Available("VideoToolbox") | `tests/videotoolbox.rs` (macOS) | H.264 High, HEVC Main (open GOP: CRA + RASL) and HEVC Main 10, 640×360 (coded 368: cropping) with B-frames: every picture **bit-exact** with our software decoder, same pts order, count, colour and aspect, also after `reset` + reseek to every later sync sample, mid-stream `flush`, and a full pass after resets; forced mid-stream failures (`VtDecoder::fail_after`) at five points continue with the software decoder's exact output; seeded mutation of samples and parameter sets (bit flips, truncation, corrupt length prefixes) never panics or hangs; HEVC 4:2:2 10-bit is bit-exact with ffmpeg's decode | | `tests/fallback.rs` (every OS) | `HybridDecoder` with a stand-in hardware decoder failing after N samples (every sync sample ± a few, first / last sample, after a seek): output identical to the software decoder; in-band parameter sets identical to the sample entry's stay in hardware, different ones switch to software | | `tests/setting.rs` | Hardware decoding Off gives the software decoder through `make_video_decoder` and the media stack (no hardware frames); Auto gives VideoToolbox where available | -| `tests/hardware_encode.rs` (macOS) | what the hardware path takes and declines; round trip through our software decoder (every picture, in order, luma PSNR above 30 dB, keyframes no further apart than asked, no composition offsets); an export through `filmcraft_export` that decodes in our decoder and in ffmpeg / ffprobe (profile, size, frame count, BT.709); the built-in encoder still exporting everything hardware declines; exact output size at sizes that are not multiples of 16; hostile configurations (zero, huge, odd sizes, frame rates, bitrates, keyframe intervals, wrong planes) give errors and never panic; encoders dropped at any point do not crash or hang | +| `tests/hardware_encode.rs` (macOS) | what the hardware path takes and declines; round trip through our software decoder (every picture, in order, luma PSNR above 30 dB, keyframes no further apart than asked, no composition offsets); an export through `filmcraft_export` that decodes in our decoder and in ffmpeg / ffprobe (profile, size, frame count, BT.709); the built-in encoder still exporting everything hardware declines; exact output size at sizes that are not multiples of 16; hostile configurations (zero, huge, odd sizes, frame rates, bitrates, keyframe intervals, wrong planes) give errors and never panic; encoders dropped at any point do not crash or hang. The same for **HEVC**: Main profile, 8-bit 4:2:0, `hvc1` entry with VPS / SPS / PPS and 4-byte lengths, MP4 and QuickTime, AAC audio, two-pass refused, the format list agreeing with the probe, ffprobe reading `codec_name=hevc`, `profile=Main`, `codec_tag_string=hvc1`, `pix_fmt=yuv420p`, BT.709 | Fixtures are made with ffmpeg into `target/fixtures/platform/` (generator only, never linked); tests skip without ffmpeg or without a hardware decoder. @@ -91,6 +121,10 @@ overlays, AAC, loudness) exported in 311 s against 793 s with the built-in encod for its densest 134 s), the outputs at SSIM 0.990 / PSNR 46.5 dB. Details in [docs/performance.md](../../docs/performance.md). +Hardware H.265 encoding, same machine: no faster than H.264 (the 9:29 timeline in 195 s against +204 s, both bound by the CPU compositor); same picture quality at 10 Mb/s and above, and 22–31 % less +bitrate for the same PSNR at 3–6 Mb/s ([docs/performance.md](../../docs/performance.md), HW3). + Hardware decoding, M4 Pro: M4 Pro, load 150–190 (`cargo xtask bench --hw off|auto`): CPU per decoded frame H.264 2160p @@ -100,5 +134,5 @@ playback with no dropped frames at Full, 1/2 and 1/4. Details in ## Not yet -Zero-copy upload of `CVPixelBuffer`s into wgpu textures; hardware HEVC encoding and B-frames; Media Foundation / -D3D11 (Windows) and VA-API (Linux) decoders; field-coded H.264. +Zero-copy upload of `CVPixelBuffer`s into wgpu textures; B-frames; 10-bit and HDR HEVC (Main 10); +Media Foundation / D3D11 (Windows) and VA-API (Linux) decoders; field-coded H.264. diff --git a/crates/platform/src/hardware_encode.rs b/crates/platform/src/hardware_encode.rs index 96b72cc9..498b68ea 100644 --- a/crates/platform/src/hardware_encode.rs +++ b/crates/platform/src/hardware_encode.rs @@ -1,22 +1,30 @@ -//! The VideoToolbox H.264 encoder behind `filmcraft_export::VideoEncoder` (macOS; safe code around -//! [`crate::videotoolbox_encode`]). +//! The VideoToolbox H.264 and HEVC encoders behind `filmcraft_export::VideoEncoder` (macOS; safe code +//! around [`crate::videotoolbox_encode`]). //! -//! Hardware encoding is opt-in per export (`ExportSettings::hardware_encoding`, `Auto`): its output depends -//! on the machine, while the built-in encoder's is byte-identical everywhere. The factory declines, -//! so the built-in encoder is used, for everything the hardware path does not take: other formats, -//! MXF (Annex B), two-pass VBR, HDR, odd picture sizes, non-square pixels, and any configuration -//! VideoToolbox cannot create a hardware session for. A hardware encoder that fails in the middle of an export reports -//! an error (the export stops); it cannot hand the stream over to another encoder. +//! H.264 hardware encoding is opt-in per export (`ExportSettings::hardware_encoding`, `Auto`): its output +//! depends on the machine, while the built-in encoder's is byte-identical everywhere. The factory +//! declines, so the built-in encoder is used, for everything the hardware path does not take: other +//! formats, MXF (Annex B), two-pass VBR, HDR, odd picture sizes, non-square pixels, and any +//! configuration VideoToolbox cannot create a hardware session for. A hardware encoder that fails in +//! the middle of an export reports an error (the export stops); it cannot hand the stream over to +//! another encoder. +//! +//! HEVC (`Format::Hevc`) has no built-in encoder, so choosing the format is the opt-in, the same +//! configurations are declined, and a declined export fails with "encoder not available" instead of +//! falling back; [`hevc_available`] tells the format list whether this machine can do it at all. + +use std::sync::OnceLock; use filmcraft_export::{BitrateMode, EncodedPacket, EncoderFrame, ExportError, ExportSettings, Format, H264Pass, H264Profile, HardwareEncoding, VideoEncoder}; -use filmcraft_isobmff::{AvcConfig, SampleEntry}; +use filmcraft_isobmff::{AvcConfig, HevcConfig, SampleEntry}; use filmcraft_time::FrameRate; -use crate::videotoolbox_encode::{VtConfig, VtEncoder, VtPacket, VtProfile, VtRate}; +use crate::videotoolbox_encode::{ParamSets, VtCodec, VtConfig, VtEncoder, VtPacket, VtProfile, VtRate}; /// The hardware encoder's configuration for an export, or why it is not used for it. pub fn config_for(w: u32, h: u32, rate: FrameRate, s: &ExportSettings) -> Result { - if s.hardware_encoding != HardwareEncoding::Auto { + let hevc = s.format == Format::Hevc; + if !hevc && s.hardware_encoding != HardwareEncoding::Auto { return Err("hardware encoding is off".into()); } if s.format.is_mxf() { @@ -45,10 +53,11 @@ pub fn config_for(w: u32, h: u32, rate: FrameRate, s: &ExportSettings) -> Result timescale, frame_duration, keyframe_interval: s.keyframe_distance.filter(|k| *k > 0).unwrap_or_else(|| (fps * 2.0).round().max(1.0) as u32), - profile: match s.h264_profile { - H264Profile::Baseline => VtProfile::Baseline, - H264Profile::Main => VtProfile::Main, - H264Profile::High => VtProfile::High, + profile: match (hevc, s.h264_profile) { + (true, _) => VtProfile::HevcMain, + (false, H264Profile::Baseline) => VtProfile::Baseline, + (false, H264Profile::Main) => VtProfile::Main, + (false, H264Profile::High) => VtProfile::High, }, rate: match s.bitrate_mode { BitrateMode::Cbr => VtRate::Cbr { kbps }, @@ -57,8 +66,31 @@ pub fn config_for(w: u32, h: u32, rate: FrameRate, s: &ExportSettings) -> Result }) } -/// The encoder factory `register` puts in front of the built-in ones: a hardware H.264 encoder when -/// the export asks for one and the OS takes the configuration, `None` (the built-in encoder) otherwise. +/// Whether this machine has a hardware HEVC encoder (asked once, by creating a small session): what +/// makes `Format::Hevc` available. +pub fn hevc_available() -> bool { + static AVAILABLE: OnceLock = OnceLock::new(); + *AVAILABLE.get_or_init(|| { + let probe = VtConfig { + width: 640, + height: 360, + timescale: 30, + frame_duration: 1, + keyframe_interval: 30, + profile: VtProfile::HevcMain, + rate: VtRate::Vbr { target_kbps: 2000, max_kbps: 3000 }, + }; + let result = VtEncoder::new(probe); + if let Err(why) = &result { + log::info!("no hardware HEVC encoder: {why}"); + } + result.is_ok() + }) +} + +/// The encoder factory `register` puts in front of the built-in ones: a hardware encoder when the +/// export asks for one (H.264 with `hardware_encoding`; HEVC always) and the OS takes the +/// configuration, `None` (the built-in encoder, if there is one) otherwise. pub fn videotoolbox_encoder_factory( format: Format, w: u32, @@ -66,27 +98,39 @@ pub fn videotoolbox_encoder_factory( rate: FrameRate, s: &ExportSettings, ) -> Option>> { - if format != Format::H264 || s.hardware_encoding != HardwareEncoding::Auto { - return None; - } + let name = match format { + Format::H264 if s.hardware_encoding == HardwareEncoding::Auto => "H.264", + Format::Hevc => "HEVC", + _ => return None, + }; let config = match config_for(w, h, rate, s) { Ok(c) => c, Err(why) => { - log::info!("hardware H.264 encoding declined: {why}"); + log::info!("hardware {name} encoding declined: {why}"); return None; } }; match VtEncoder::new(config) { - Ok(vt) => Some(Ok(Box::new(HardwareH264Encoder { vt, y: Vec::new(), u: Vec::new(), v: Vec::new(), decoded: 0 }))), + Ok(vt) => Some(Ok(Box::new(HardwareEncoder { vt, y: Vec::new(), u: Vec::new(), v: Vec::new(), decoded: 0 }))), Err(why) => { - log::info!("hardware H.264 encoding declined: {why}"); + log::info!("hardware {name} encoding declined: {why}"); None } } } +/// The `hvcC` configuration of an HEVC stream: VideoToolbox's own record when it wrote one. +fn hevc_config(p: &ParamSets) -> Result { + let record = p.hvcc.as_deref().ok_or("VideoToolbox wrote no hvcC record for the stream")?; + let cfg = HevcConfig::parse(record).map_err(|e| format!("unreadable hvcC record: {e}"))?; + if cfg.vps().is_empty() || cfg.sps().is_empty() || cfg.pps().is_empty() || cfg.length_size != 4 { + return Err("the hvcC record lacks a parameter set or does not use 4-byte NAL lengths".into()); + } + Ok(cfg) +} + /// [`VtEncoder`] as a `VideoEncoder`: RGBA pictures in, MP4 samples out. -pub struct HardwareH264Encoder { +pub struct HardwareEncoder { vt: VtEncoder, y: Vec, u: Vec, @@ -95,7 +139,7 @@ pub struct HardwareH264Encoder { decoded: i64, } -impl HardwareH264Encoder { +impl HardwareEncoder { /// MP4 samples from compressed frames. Without frame reordering every frame is presented at /// its decode time, so composition offsets are 0; a frame whose time is not `n × duration` /// after the first (a dropped or reordered frame) is an error, never a file with wrong times. @@ -119,11 +163,18 @@ impl HardwareH264Encoder { } } -impl VideoEncoder for HardwareH264Encoder { +impl VideoEncoder for HardwareEncoder { fn sample_entry(&self) -> SampleEntry { let c = self.vt.config(); - let (sps, pps) = self.vt.parameter_sets().map(|p| (p.sps, p.pps)).unwrap_or_default(); - SampleEntry::avc(AvcConfig::new(sps, pps, 4), c.width.min(u32::from(u16::MAX)) as u16, c.height.min(u32::from(u16::MAX)) as u16) + let (w, h) = (c.width.min(u32::from(u16::MAX)) as u16, c.height.min(u32::from(u16::MAX)) as u16); + match (c.profile.codec(), self.vt.parameter_sets()) { + (VtCodec::H264, p) => { + let (sps, pps) = p.map(|p| (p.sps, p.pps)).unwrap_or_default(); + SampleEntry::avc(AvcConfig::new(sps, pps, 4), w, h) + } + // `encode` refuses to go on without a usable record, so a stream that got this far has one + (VtCodec::Hevc, p) => SampleEntry::hevc(p.as_ref().and_then(|p| hevc_config(p).ok()).unwrap_or_default(), w, h), + } } fn timescale(&self) -> u32 { @@ -132,8 +183,9 @@ impl VideoEncoder for HardwareH264Encoder { fn encode(&mut self, f: &EncoderFrame) -> filmcraft_export::Result> { let c = self.vt.config(); + let codec = c.profile.codec(); if f.hdr.is_some() { - return Err(ExportError::Unsupported("hardware H.264 encoding does not take HDR pictures".into())); + return Err(ExportError::Unsupported("hardware encoding does not take HDR pictures".into())); } if (f.width, f.height) != (c.width, c.height) { return Err(ExportError::Encode(format!("picture is {}x{}, the encoder was created for {}x{}", f.width, f.height, c.width, c.height))); @@ -145,8 +197,11 @@ impl VideoEncoder for HardwareH264Encoder { filmcraft_export::rgba_to_yuv420_8(f.rgba, w, h, &mut self.y, &mut self.u, &mut self.v); let packets = self.vt.encode(f.index, &self.y, &self.u, &self.v).map_err(ExportError::Encode)?; // the muxer needs the parameter sets after the first group of pictures - if f.index == 0 && self.vt.parameter_sets().is_none() { - return Err(ExportError::Encode("VideoToolbox returned no parameter sets for the first frame".into())); + if f.index == 0 { + let params = self.vt.parameter_sets().ok_or_else(|| ExportError::Encode("VideoToolbox returned no parameter sets for the first frame".into()))?; + if codec == VtCodec::Hevc { + hevc_config(¶ms).map_err(ExportError::Encode)?; + } } self.samples(packets) } diff --git a/crates/platform/src/lib.rs b/crates/platform/src/lib.rs index 81f27b24..ed759ff5 100644 --- a/crates/platform/src/lib.rs +++ b/crates/platform/src/lib.rs @@ -54,6 +54,7 @@ pub fn register() -> Availability { { filmcraft_codecs::register_video_decoder(videotoolbox_factory); filmcraft_export::register_encoder(hardware_encode::videotoolbox_encoder_factory); + filmcraft_export::register_format_probe(filmcraft_export::Format::Hevc, hardware_encode::hevc_available); Availability::Available("VideoToolbox") } #[cfg(not(target_os = "macos"))] diff --git a/crates/platform/src/videotoolbox_encode.rs b/crates/platform/src/videotoolbox_encode.rs index 48b35135..4e9130dd 100644 --- a/crates/platform/src/videotoolbox_encode.rs +++ b/crates/platform/src/videotoolbox_encode.rs @@ -1,4 +1,4 @@ -//! VideoToolbox (macOS) hardware H.264 encoding. +//! VideoToolbox (macOS) hardware H.264 and HEVC encoding. //! //! An FFI module of this crate (docs/adr/0001-platform-ffi.md): every `unsafe` block has a //! `// SAFETY:` comment, no panic may unwind into VideoToolbox (the output callback runs under @@ -15,16 +15,21 @@ //! copies each compressed frame (length-prefixed NAL units) into a queue that [`VtEncoder::encode`] //! and [`VtEncoder::flush`] drain. //! -//! The first frame is completed straight away: the SPS / PPS the container needs come with the -//! first compressed frame, and the muxer asks for them after the first group of frames. +//! The first frame is completed straight away: the parameter sets the container needs (SPS / PPS +//! for H.264, VPS / SPS / PPS and VideoToolbox's own `hvcC` record for HEVC) come with the first +//! compressed frame, and the muxer asks for them after the first group of frames. use std::ffi::{c_int, c_void}; use std::panic::AssertUnwindSafe; use std::ptr::NonNull; use std::sync::{Mutex, PoisonError}; -use objc2_core_foundation::{CFArray, CFBoolean, CFDictionary, CFNumber, CFRetained, CFString, CFType}; -use objc2_core_media::{CMSampleBuffer, CMTime, CMTimeFlags, CMVideoFormatDescriptionGetH264ParameterSetAtIndex, kCMVideoCodecType_H264}; +use objc2_core_foundation::{CFArray, CFBoolean, CFData, CFDictionary, CFNumber, CFRetained, CFString, CFType}; +use objc2_core_media::{ + CMFormatDescription, CMSampleBuffer, CMTime, CMTimeFlags, CMVideoFormatDescriptionGetH264ParameterSetAtIndex, + CMVideoFormatDescriptionGetHEVCParameterSetAtIndex, kCMFormatDescriptionExtension_SampleDescriptionExtensionAtoms, kCMVideoCodecType_H264, + kCMVideoCodecType_HEVC, +}; use objc2_core_video::{ CVImageBuffer, CVPixelBufferCreate, CVPixelBufferGetBaseAddressOfPlane, CVPixelBufferGetBytesPerRowOfPlane, CVPixelBufferGetHeightOfPlane, CVPixelBufferGetWidthOfPlane, CVPixelBufferLockBaseAddress, CVPixelBufferLockFlags, CVPixelBufferPool, CVPixelBufferUnlockBaseAddress, @@ -32,11 +37,12 @@ use objc2_core_video::{ kCVPixelBufferIOSurfacePropertiesKey, kCVPixelBufferPixelFormatTypeKey, kCVPixelBufferWidthKey, }; use objc2_video_toolbox::{ - VTCompressionSession, VTEncodeInfoFlags, VTSessionSetProperty, kVTCompressionPropertyKey_AllowFrameReordering, kVTCompressionPropertyKey_AllowOpenGOP, - kVTCompressionPropertyKey_AverageBitRate, kVTCompressionPropertyKey_ColorPrimaries, kVTCompressionPropertyKey_ConstantBitRate, - kVTCompressionPropertyKey_DataRateLimits, kVTCompressionPropertyKey_ExpectedFrameRate, kVTCompressionPropertyKey_MaxKeyFrameInterval, - kVTCompressionPropertyKey_ProfileLevel, kVTCompressionPropertyKey_RealTime, kVTCompressionPropertyKey_TransferFunction, - kVTCompressionPropertyKey_YCbCrMatrix, kVTProfileLevel_H264_Baseline_AutoLevel, kVTProfileLevel_H264_High_AutoLevel, kVTProfileLevel_H264_Main_AutoLevel, + VTCompressionSession, VTEncodeInfoFlags, VTSessionCopyProperty, VTSessionSetProperty, kVTCompressionPropertyKey_AllowFrameReordering, + kVTCompressionPropertyKey_AllowOpenGOP, kVTCompressionPropertyKey_AverageBitRate, kVTCompressionPropertyKey_ColorPrimaries, + kVTCompressionPropertyKey_ConstantBitRate, kVTCompressionPropertyKey_DataRateLimits, kVTCompressionPropertyKey_ExpectedFrameRate, + kVTCompressionPropertyKey_MaxKeyFrameInterval, kVTCompressionPropertyKey_ProfileLevel, kVTCompressionPropertyKey_RealTime, + kVTCompressionPropertyKey_TransferFunction, kVTCompressionPropertyKey_UsingHardwareAcceleratedVideoEncoder, kVTCompressionPropertyKey_YCbCrMatrix, + kVTProfileLevel_H264_Baseline_AutoLevel, kVTProfileLevel_H264_High_AutoLevel, kVTProfileLevel_H264_Main_AutoLevel, kVTProfileLevel_HEVC_Main_AutoLevel, kVTVideoEncoderSpecification_RequireHardwareAcceleratedVideoEncoder, }; @@ -47,12 +53,30 @@ const MAX_FRAME_BYTES: usize = 256 << 20; /// Largest picture the hardware encoder is asked for. const MAX_SIDE: u32 = 8192; -/// H.264 profiles VideoToolbox offers. +/// The codec of a session. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum VtCodec { + H264, + Hevc, +} + +/// H.264 and HEVC profiles VideoToolbox offers (the profile also picks the codec). #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub enum VtProfile { Baseline, Main, High, + /// HEVC Main: 8-bit 4:2:0. + HevcMain, +} + +impl VtProfile { + pub fn codec(self) -> VtCodec { + match self { + VtProfile::Baseline | VtProfile::Main | VtProfile::High => VtCodec::H264, + VtProfile::HevcMain => VtCodec::Hevc, + } + } } /// How the bitrate is controlled (kilobits per second). @@ -75,6 +99,7 @@ pub struct VtConfig { pub frame_duration: u32, /// Frames between keyframes (every keyframe is an IDR picture). pub keyframe_interval: u32, + /// The profile, which also says whether this is an H.264 or an HEVC session. pub profile: VtProfile, pub rate: VtRate, } @@ -82,18 +107,22 @@ pub struct VtConfig { /// One compressed frame in decoding order, which is presentation order (times in the config's timescale). #[derive(Clone, Debug)] pub struct VtPacket { - /// Length-prefixed NAL units (4-byte lengths, as in an `avcC` MP4 sample). + /// Length-prefixed NAL units (4-byte lengths, as in an `avcC` / `hvcC` MP4 sample). pub data: Vec, pub key: bool, pub pts: i64, pub dts: i64, } -/// The parameter sets of the stream (NAL units without length prefix), for the `avcC` box. +/// The parameter sets of the stream (NAL units without length prefix), for the `avcC` / `hvcC` box. #[derive(Clone, Debug, PartialEq, Eq)] pub struct ParamSets { + /// HEVC only. + pub vps: Vec>, pub sps: Vec>, pub pps: Vec>, + /// HEVC only: the `hvcC` record (HEVCDecoderConfigurationRecord) VideoToolbox wrote for the stream. + pub hvcc: Option>, } /// One output-callback result. @@ -105,6 +134,7 @@ enum Output { /// State shared with the output callback (its `outputCallbackRefCon`). Boxed by the encoder and /// kept alive until the session is invalidated. struct Shared { + codec: VtCodec, timescale: i64, out: Mutex>, params: Mutex>, @@ -160,37 +190,60 @@ fn ticks(t: CMTime, timescale: i64) -> Option { i64::try_from((2 * num + den).div_euclid(2 * den)).ok() } -/// Whether length-prefixed NAL units hold an IDR slice (nal_unit_type 5). -fn has_idr(data: &[u8]) -> Result { +/// Whether length-prefixed NAL units hold a random access point: an IDR slice (nal_unit_type 5) in +/// H.264; a BLA, IDR or CRA slice (types 16 to 21) in HEVC. +fn is_sync(data: &[u8], codec: VtCodec) -> Result { let mut pos = 0usize; - let mut idr = false; + let mut key = false; while pos < data.len() { let len_bytes = data.get(pos..pos + 4).ok_or("truncated NAL length")?; let len = u32::from_be_bytes([len_bytes[0], len_bytes[1], len_bytes[2], len_bytes[3]]) as usize; pos += 4; let end = pos.checked_add(len).filter(|e| *e <= data.len()).ok_or("NAL unit longer than the frame")?; let header = *data.get(pos).ok_or("empty NAL unit")?; - if header & 0x1f == 5 { - idr = true; - } + key |= match codec { + VtCodec::H264 => header & 0x1f == 5, + VtCodec::Hevc => (16..=21).contains(&((header >> 1) & 0x3f)), + }; pos = end; } - Ok(idr) + Ok(key) } -/// SPS / PPS of the first compressed frame's format description. -fn read_param_sets(sample: &CMSampleBuffer) -> Result { +/// The `hvcC` record VideoToolbox wrote among the sample description extension atoms of an HEVC +/// format description (the container's own codec configuration, exactly as QuickTime writes it). +fn hvcc_record(format: &CMFormatDescription) -> Option> { + // SAFETY: `format` is valid; the key is an immutable framework constant; the result is a + // retained property list (or none). + let atoms = unsafe { format.extension(kCMFormatDescriptionExtension_SampleDescriptionExtensionAtoms) }?; + let atoms = atoms.downcast::().ok()?; + // SAFETY: CoreMedia documents this extension as a dictionary from four-character-code strings + // to property lists; every CoreFoundation object is a `CFType`. + let atoms: &CFDictionary = unsafe { atoms.cast_unchecked() }; + let data = atoms.get(&CFString::from_str("hvcC"))?.downcast::().ok()?; + let bytes = data.to_vec(); + (!bytes.is_empty() && bytes.len() <= 65_535).then_some(bytes) +} + +/// The parameter sets of the first compressed frame's format description: SPS / PPS for H.264, +/// VPS / SPS / PPS and the `hvcC` record for HEVC. +fn read_param_sets(sample: &CMSampleBuffer, codec: VtCodec) -> Result { // SAFETY: `sample` is a valid sample buffer for the duration of the callback. let format = unsafe { sample.format_description() }.ok_or("compressed frame without a format description")?; - let (mut sps, mut pps) = (Vec::new(), Vec::new()); + let (mut vps, mut sps, mut pps) = (Vec::new(), Vec::new(), Vec::new()); let (mut index, mut count) = (0usize, 1usize); while index < count && index < 16 { let (mut ptr, mut size, mut total, mut nal_len): (*const u8, usize, usize, c_int) = (std::ptr::null(), 0, 0, 0); // SAFETY: the out-pointers are valid for the call; the returned pointer is only read // while `format` (retained above) is alive. - let status = unsafe { CMVideoFormatDescriptionGetH264ParameterSetAtIndex(&format, index, &mut ptr, &mut size, &mut total, &mut nal_len) }; + let status = unsafe { + match codec { + VtCodec::H264 => CMVideoFormatDescriptionGetH264ParameterSetAtIndex(&format, index, &mut ptr, &mut size, &mut total, &mut nal_len), + VtCodec::Hevc => CMVideoFormatDescriptionGetHEVCParameterSetAtIndex(&format, index, &mut ptr, &mut size, &mut total, &mut nal_len), + } + }; if status != 0 { - return Err(format!("cannot read the H.264 parameter sets ({status})")); + return Err(format!("cannot read the {codec:?} parameter sets ({status})")); } if nal_len != 4 { return Err(format!("unexpected NAL length size {nal_len}")); @@ -201,17 +254,28 @@ fn read_param_sets(sample: &CMSampleBuffer) -> Result { } // SAFETY: VideoToolbox returned `size` readable bytes at `ptr`, owned by `format`. let bytes = unsafe { std::slice::from_raw_parts(ptr, size) }.to_vec(); - match bytes.first().map(|b| b & 0x1f) { - Some(7) => sps.push(bytes), - Some(8) => pps.push(bytes), + match (codec, bytes.first()) { + (VtCodec::H264, Some(b)) => match b & 0x1f { + 7 => sps.push(bytes), + 8 => pps.push(bytes), + _ => {} + }, + (VtCodec::Hevc, Some(b)) => match (b >> 1) & 0x3f { + 32 => vps.push(bytes), + 33 => sps.push(bytes), + 34 => pps.push(bytes), + _ => {} + }, _ => {} } index += 1; } - if sps.is_empty() || pps.is_empty() { - return Err("the format description has no SPS / PPS".into()); + let complete = !sps.is_empty() && !pps.is_empty() && (codec == VtCodec::H264 || !vps.is_empty()); + if !complete { + return Err("the format description lacks parameter sets".into()); } - Ok(ParamSets { sps, pps }) + let hvcc = if codec == VtCodec::Hevc { hvcc_record(&format) } else { None }; + Ok(ParamSets { vps, sps, pps, hvcc }) } /// One compressed frame: its bytes, key flag and timestamps (and, once, the parameter sets). @@ -234,11 +298,11 @@ fn read_packet(sample: &CMSampleBuffer, shared: &Shared) -> Result Ok(()) } -/// A hardware H.264 encoder: feed pictures, collect compressed frames (in presentation order, which -/// is decoding order since nothing is reordered). +/// A hardware H.264 or HEVC encoder: feed pictures, collect compressed frames (in presentation +/// order, which is decoding order since nothing is reordered). pub struct VtEncoder { session: Session, config: VtConfig, @@ -375,13 +439,14 @@ impl VtEncoder { return Err(format!("picture size {w}x{h}")); } if !w.is_multiple_of(2) || !h.is_multiple_of(2) { - // 4:2:0 H.264 crops in units of two samples: an odd size would come out one sample smaller + // 4:2:0 video crops in units of two samples: an odd size would come out one sample smaller return Err(format!("odd picture size {w}x{h}")); } if config.timescale == 0 || config.frame_duration == 0 || config.timescale > i32::MAX as u32 { return Err("invalid frame rate".into()); } - let shared = Box::new(Shared { timescale: i64::from(config.timescale), out: Mutex::new(Vec::new()), params: Mutex::new(None) }); + let codec = config.profile.codec(); + let shared = Box::new(Shared { codec, timescale: i64::from(config.timescale), out: Mutex::new(Vec::new()), params: Mutex::new(None) }); // SAFETY: reading immutable framework constants. let require_key: &CFString = unsafe { kVTVideoEncoderSpecification_RequireHardwareAcceleratedVideoEncoder }; @@ -403,7 +468,10 @@ impl VtEncoder { None, w as i32, h as i32, - kCMVideoCodecType_H264, + match codec { + VtCodec::H264 => kCMVideoCodecType_H264, + VtCodec::Hevc => kCMVideoCodecType_HEVC, + }, Some(spec.as_ref()), Some(attrs.as_ref()), None, @@ -433,6 +501,7 @@ impl VtEncoder { VtProfile::Baseline => kVTProfileLevel_H264_Baseline_AutoLevel, VtProfile::Main => kVTProfileLevel_H264_Main_AutoLevel, VtProfile::High => kVTProfileLevel_H264_High_AutoLevel, + VtProfile::HevcMain => kVTProfileLevel_HEVC_Main_AutoLevel, }, ) }; @@ -497,12 +566,28 @@ impl VtEncoder { Ok(()) } + /// Whether VideoToolbox says this session runs on a hardware encoder. The session is created + /// with one required, so it should always hold; asking (and testing it) means a requirement + /// dropped by accident could never turn into a silent software encode. + pub fn uses_hardware(&self) -> bool { + // SAFETY: reading an immutable framework constant. + let key: &CFString = unsafe { kVTCompressionPropertyKey_UsingHardwareAcceleratedVideoEncoder }; + let mut value: *mut CFType = std::ptr::null_mut(); + // SAFETY: the session is valid and `value` is a valid out-pointer for the CFTypeRef the + // property is copied to (returned retained, +1); it stays null when the call fails. + let status = unsafe { VTSessionCopyProperty(self.session.session.as_ref(), key, None, (&raw mut value).cast()) }; + let Some(value) = NonNull::new(value).filter(|_| status == 0) else { return false }; + // SAFETY: a copied property is returned retained (+1); `CFRetained` takes that reference over. + let value = unsafe { CFRetained::from_raw(value) }; + value.downcast_ref::().is_some_and(CFBoolean::as_bool) + } + /// The configuration this encoder was created with. pub fn config(&self) -> &VtConfig { &self.config } - /// The stream's SPS / PPS, known once the first compressed frame came out. + /// The stream's parameter sets, known once the first compressed frame came out. pub fn parameter_sets(&self) -> Option { self.session.shared.params.lock().unwrap_or_else(PoisonError::into_inner).clone() } @@ -544,7 +629,7 @@ impl VtEncoder { } self.submitted += 1; if self.submitted == 1 { - // the first compressed frame carries the SPS / PPS the container needs + // the first compressed frame carries the parameter sets the container needs // SAFETY: the session is valid. require(unsafe { self.session.session.complete_frames(pts) }, "to finish the first frame")?; } diff --git a/crates/platform/tests/hardware_encode.rs b/crates/platform/tests/hardware_encode.rs index eeed379a..2962f670 100644 --- a/crates/platform/tests/hardware_encode.rs +++ b/crates/platform/tests/hardware_encode.rs @@ -1,4 +1,4 @@ -//! VideoToolbox H.264 encoding (macOS): what the hardware path takes and declines, a round trip +//! VideoToolbox H.264 and HEVC encoding (macOS): what the hardware path takes and declines, a round trip //! through our own software decoder (picture quality, frame count and order, keyframes, the //! timestamps the muxer needs), a full export through `filmcraft_export` that decodes, the fallback //! to the built-in encoder for everything hardware does not take, and hostile configurations that @@ -15,7 +15,7 @@ use filmcraft_frame::PixelData; use filmcraft_isobmff::{CodecConfig, SampleEntry}; use filmcraft_media::generators::GeneratorSource; use filmcraft_media::{DemoScene, MediaSource}; -use filmcraft_platform::hardware_encode::{config_for, videotoolbox_encoder_factory}; +use filmcraft_platform::hardware_encode::{config_for, hevc_available, videotoolbox_encoder_factory}; use filmcraft_platform::videotoolbox_encode::{VtConfig, VtEncoder, VtProfile, VtRate}; use filmcraft_project::{ItemKind, Label, MediaClip, MediaRef, Project, SequenceSettings, TrackKind}; use filmcraft_render::SourceMap; @@ -46,13 +46,18 @@ fn hardware_settings() -> ExportSettings { ExportSettings { format: Format::H264, hardware_encoding: HardwareEncoding::Auto, bitrate_kbps: 8_000, keyframe_distance: Some(30), ..Default::default() } } +/// H.265 needs no opt-in: choosing the format is the opt-in. +fn hevc_settings() -> ExportSettings { + ExportSettings { format: Format::Hevc, bitrate_kbps: 6_000, keyframe_distance: Some(30), ..Default::default() } +} + /// The hardware encoder for the settings, or `None` (skip) when this machine has none. fn hardware(s: &ExportSettings, w: u32, h: u32) -> Option> { - match videotoolbox_encoder_factory(Format::H264, w, h, FPS, s) { + match videotoolbox_encoder_factory(s.format, w, h, FPS, s) { Some(Ok(e)) => Some(e), Some(Err(e)) => panic!("factory error: {e}"), None => { - eprintln!("SKIPPED: no VideoToolbox hardware H.264 encoder for these settings"); + eprintln!("SKIPPED: no VideoToolbox hardware {} encoder for these settings", s.format.label()); None } } @@ -208,7 +213,7 @@ fn inspect(path: &str) -> (usize, usize, usize) { let vt = file.track_of_kind(filmcraft_isobmff::TrackKind::Video).unwrap(); let track = &file.tracks[vt]; let entry = track.entries.first().expect("a sample entry"); - let CodecConfig::Avc(_) = &entry.codec else { panic!("not AVC") }; + assert!(matches!(&entry.codec, CodecConfig::Avc(_) | CodecConfig::Hevc(_)), "neither AVC nor HEVC: {}", entry.codec.name()); let mut dec = filmcraft_codecs::software_video_decoder(entry).expect("software decoder"); let mut pictures = 0; for (i, s) in track.samples.iter().enumerate() { @@ -431,3 +436,280 @@ fn an_empty_range_never_leaves_a_broken_file() { } } } + +// --------------------------------------------------------------------------------------------- +// HEVC (H.265) +// --------------------------------------------------------------------------------------------- + +#[test] +fn hevc_declines_what_the_hardware_path_does_not_take() { + let ok = hevc_settings(); + let config = config_for(W, H, FPS, &ok).expect("HEVC needs no hardware flag"); + assert_eq!((config.profile, config.keyframe_interval, config.rate), (VtProfile::HevcMain, 30, VtRate::Vbr { target_kbps: 6000, max_kbps: 9000 })); + let declined = |s: &ExportSettings, what: &str| assert!(config_for(W, H, FPS, s).is_err(), "{what} must not use the hardware encoder"); + declined(&ExportSettings { bitrate_mode: BitrateMode::Vbr2Pass, ..ok.clone() }, "two-pass VBR"); + declined(&ExportSettings { signal: filmcraft_export::ColorSignal::PQ, ..ok.clone() }, "HDR"); + declined(&ExportSettings { pixel_aspect: Some((4, 3)), ..ok.clone() }, "non-square pixels"); + assert!(config_for(321, 181, FPS, &ok).is_err() && config_for(640, 361, FPS, &ok).is_err(), "odd sizes"); + assert!(config_for(W, H, FrameRate { num: 0, den: 1 }, &ok).is_err()); + // constant bitrate, and the H.264 profile setting does not matter + let c = config_for(W, H, FPS, &ExportSettings { bitrate_mode: BitrateMode::Cbr, bitrate_kbps: 4000, h264_profile: H264Profile::Baseline, ..ok }).unwrap(); + assert_eq!((c.profile, c.rate), (VtProfile::HevcMain, VtRate::Cbr { kbps: 4000 })); +} + +#[test] +fn hevc_format_follows_the_machine() { + filmcraft_platform::register(); + // the format list asks `available`: it must agree with the probe + assert_eq!(filmcraft_export::available(Format::Hevc), hevc_available()); + let s = hevc_settings(); + // no hardware flag needed, but nothing else is taken by this factory + assert_eq!(videotoolbox_encoder_factory(Format::Hevc, W, H, FPS, &s).is_some(), hevc_available()); + assert!(videotoolbox_encoder_factory(Format::Hevc, 321, 181, FPS, &s).is_none()); + assert!(videotoolbox_encoder_factory(Format::H264, W, H, FPS, &s).is_none(), "H.264 still needs the hardware flag"); +} + +#[test] +fn hevc_round_trip_through_our_software_decoder() { + let s = hevc_settings(); + let Some(mut enc) = hardware(&s, W, H) else { return }; + let n = 75u64; + let packets = encode_all(&mut enc, n); + let entry = enc.sample_entry(); + assert_eq!(&entry.format.0, b"hvc1"); + let CodecConfig::Hevc(hevc) = &entry.codec else { panic!("expected an HEVC sample entry, got {}", entry.codec.name()) }; + assert!(!hevc.vps().is_empty() && !hevc.sps().is_empty() && !hevc.pps().is_empty()); + assert_eq!(hevc.length_size, 4); + // Main profile, 8-bit 4:2:0 + assert_eq!((hevc.general_profile_idc, hevc.chroma_format_idc, hevc.bit_depth_luma, hevc.bit_depth_chroma), (1, 1, 8, 8)); + assert!(hevc.general_level_idc >= 90, "level {} cannot hold 640x360", hevc.general_level_idc); + assert_eq!(enc.timescale(), 30); + + // one compressed frame per picture, the first a keyframe, keyframes no further apart than asked + assert_eq!(packets.len() as u64, n); + assert!(packets[0].key); + assert!(packets.iter().all(|p| p.duration == 1 && p.composition_offset == 0 && !p.data.is_empty())); + let keys: Vec = packets.iter().enumerate().filter(|(_, p)| p.key).map(|(i, _)| i).collect(); + assert!(keys.windows(2).all(|w| w[1] - w[0] <= 30), "keyframes at {keys:?}"); + assert_eq!(enc.media_start(), None); + + // our decoder reproduces every picture, in order, at a good quality + let decoded = decode(&entry, &packets); + assert_eq!(decoded.len() as u64, n); + assert_eq!(decoded.iter().map(|(p, _)| *p).collect::>(), (0..n as i64).collect::>()); + let (mut y, mut u, mut v) = (Vec::new(), Vec::new(), Vec::new()); + let mut worst = 99.0f64; + for (i, (_, luma)) in decoded.iter().enumerate() { + rgba_to_yuv420_8(&picture(i as u64), W as usize, H as usize, &mut y, &mut u, &mut v); + worst = worst.min(psnr(&y, luma)); + } + assert!(worst > 30.0, "worst luma PSNR {worst:.1} dB"); + eprintln!("hardware HEVC: {} bytes for {n} frames, worst luma PSNR {worst:.1} dB", packets.iter().map(|p| p.data.len()).sum::()); +} + +#[test] +fn hevc_export_through_the_pipeline() { + filmcraft_platform::register(); + if !hevc_available() { + eprintln!("SKIPPED: no VideoToolbox hardware HEVC encoder"); + return; + } + let frames = 70; + let (p, seq, m) = project(frames); + // MP4 (the default container) and QuickTime + for (name, multiplexer) in [("hevc.mp4", filmcraft_export::Multiplexer::Mp4), ("hevc.mov", filmcraft_export::Multiplexer::Mov)] { + let path = tmp(name); + let s = ExportSettings { path: path.clone(), include_audio: false, multiplexer, ..hevc_settings() }; + let report = export(&p, seq, &s, &m, &Progress::default()).unwrap_or_else(|e| panic!("{name}: {e}")); + assert_eq!(report.frames, frames as u64, "{name}"); + let (samples, pictures, syncs) = inspect(&path); + assert_eq!((samples, pictures), (frames as usize, frames as usize), "{name}"); + assert!(syncs >= 3, "{name}: keyframes every 30 frames: {syncs} sync samples"); + } +} + +#[test] +fn hevc_export_with_audio_writes_aac_in_mp4() { + filmcraft_platform::register(); + if !hevc_available() { + eprintln!("SKIPPED: no VideoToolbox hardware HEVC encoder"); + return; + } + let (p, seq, m) = project(60); + let path = tmp("hevc-audio.mp4"); + let s = ExportSettings { path: path.clone(), include_audio: true, ..hevc_settings() }; + export(&p, seq, &s, &m, &Progress::default()).expect("export"); + let bytes = std::fs::read(&path).unwrap(); + let file = filmcraft_isobmff::open(bytes.as_slice()).unwrap(); + assert!(file.track_of_kind(filmcraft_isobmff::TrackKind::Video).is_some()); + let at = file.track_of_kind(filmcraft_isobmff::TrackKind::Audio).expect("an audio track"); + assert!(matches!(&file.tracks[at].entries[0].codec, CodecConfig::Aac(_)), "audio is {}", file.tracks[at].entries[0].codec.name()); +} + +#[test] +fn hevc_has_no_two_pass_mode() { + filmcraft_platform::register(); + let (p, seq, m) = project(10); + let s = ExportSettings { path: tmp("hevc-2pass.mp4"), include_audio: false, bitrate_mode: BitrateMode::Vbr2Pass, ..hevc_settings() }; + let err = export(&p, seq, &s, &m, &Progress::default()).expect_err("two-pass HEVC must be refused").to_string(); + assert!(err.contains("two-pass"), "{err}"); +} + +#[test] +fn hevc_hostile_configurations_give_errors_not_crashes() { + let base = VtConfig { + width: W, + height: H, + timescale: 30, + frame_duration: 1, + keyframe_interval: 30, + profile: VtProfile::HevcMain, + rate: VtRate::Vbr { target_kbps: 6000, max_kbps: 9000 }, + }; + let variants: Vec<(&str, VtConfig)> = vec![ + ("zero width", VtConfig { width: 0, ..base.clone() }), + ("zero height", VtConfig { height: 0, ..base.clone() }), + ("huge width", VtConfig { width: u32::MAX, ..base.clone() }), + ("1x1", VtConfig { width: 1, height: 1, ..base.clone() }), + ("2x2", VtConfig { width: 2, height: 2, ..base.clone() }), + ("odd size", VtConfig { width: 321, height: 181, ..base.clone() }), + ("zero timescale", VtConfig { timescale: 0, ..base.clone() }), + ("zero frame duration", VtConfig { frame_duration: 0, ..base.clone() }), + ("huge timescale", VtConfig { timescale: u32::MAX, ..base.clone() }), + ("zero keyframe interval", VtConfig { keyframe_interval: 0, ..base.clone() }), + ("huge keyframe interval", VtConfig { keyframe_interval: u32::MAX, ..base.clone() }), + ("zero bitrate", VtConfig { rate: VtRate::Vbr { target_kbps: 0, max_kbps: 0 }, ..base.clone() }), + ("huge bitrate", VtConfig { rate: VtRate::Vbr { target_kbps: u32::MAX, max_kbps: u32::MAX }, ..base.clone() }), + ("max below target", VtConfig { rate: VtRate::Vbr { target_kbps: 6000, max_kbps: 1 }, ..base.clone() }), + ("zero cbr", VtConfig { rate: VtRate::Cbr { kbps: 0 }, ..base.clone() }), + ("cbr", VtConfig { rate: VtRate::Cbr { kbps: 4000 }, ..base.clone() }), + ]; + for (name, config) in variants { + match calm(name, || VtEncoder::new(config)) { + Ok(mut enc) => { + let (w, h) = (enc.config().width as usize, enc.config().height as usize); + let (cw, ch) = (w.div_ceil(2), h.div_ceil(2)); + assert!(calm(name, || enc.encode(0, &[], &[], &[])).is_err(), "{name}: empty planes"); + assert!(calm(name, || enc.encode(0, &vec![0; w * h - 1], &vec![128; cw * ch], &vec![128; cw * ch])).is_err(), "{name}: short luma"); + let _ = calm(name, || enc.encode(0, &vec![16; w * h], &vec![128; cw * ch], &vec![128; cw * ch])); + let _ = calm(name, || enc.flush()); + } + Err(why) => eprintln!("{name}: declined ({why})"), + } + } +} + +#[test] +fn hevc_encoders_can_be_dropped_at_any_point() { + let s = hevc_settings(); + for frames in [0u64, 1, 2, 5] { + let Some(mut enc) = hardware(&s, W, H) else { return }; + for i in 0..frames { + let rgba = picture(i); + let _ = calm("encode", || enc.encode(&EncoderFrame { width: W, height: H, rgba: &rgba, hdr: None, index: i })); + } + drop(enc); // pending frames, no flush: must not crash or hang + } +} + +#[test] +fn hevc_pictures_keep_their_exact_size() { + let s = hevc_settings(); + // even sizes that are not multiples of the coding block size come out at exactly their size + for (w, h) in [(640u32, 362u32), (642, 360), (1280, 720), (1920, 1080), (64, 64)] { + let Some(mut enc) = hardware(&s, w, h) else { continue }; + let rgba: Vec = (0..w * h * 4).map(|i| (i % 251) as u8).collect(); + let mut packets = Vec::new(); + for i in 0..6u64 { + packets.extend(enc.encode(&EncoderFrame { width: w, height: h, rgba: &rgba, hdr: None, index: i }).expect("encode")); + } + packets.extend(enc.flush().expect("flush")); + assert_eq!(packets.len(), 6, "{w}x{h}"); + let entry = enc.sample_entry(); + let mut dec = filmcraft_codecs::software_video_decoder(&entry).expect("software decoder"); + let mut pictures = Vec::new(); + for (i, p) in packets.iter().enumerate() { + pictures.extend(dec.decode(&p.data, i as i64).expect("decode")); + } + pictures.extend(dec.flush()); + assert_eq!(pictures.len(), 6, "{w}x{h}"); + assert!( + pictures.iter().all(|f| (f.frame.width, f.frame.height) == (w, h)), + "{w}x{h}: decoded {}x{}", + pictures[0].frame.width, + pictures[0].frame.height + ); + } +} + +#[test] +fn ffmpeg_decodes_what_the_hevc_encoder_writes() { + filmcraft_platform::register(); + let Some(ffmpeg) = filmcraft_testkit::ffmpeg_or_skip("hardware HEVC export") else { return }; + let Some(ffprobe) = filmcraft_testkit::ffprobe_or_skip("hardware HEVC export") else { return }; + if !hevc_available() { + eprintln!("SKIPPED: no VideoToolbox hardware HEVC encoder"); + return; + } + let frames = 65; + let (p, seq, m) = project(frames); + for (name, multiplexer) in [("oracle-hevc.mp4", filmcraft_export::Multiplexer::Mp4), ("oracle-hevc.mov", filmcraft_export::Multiplexer::Mov)] { + let path = tmp(name); + let s = ExportSettings { path: path.clone(), include_audio: false, multiplexer, ..hevc_settings() }; + export(&p, seq, &s, &m, &Progress::default()).expect("export"); + // an independent decoder reads the whole file without a complaint + let out = std::process::Command::new(&ffmpeg).args(["-v", "error", "-i", &path, "-f", "null", "-"]).output().expect("run ffmpeg"); + assert!(out.status.success() && out.stderr.is_empty(), "{name}: ffmpeg: {}", String::from_utf8_lossy(&out.stderr)); + // and finds the stream we asked for: HEVC Main, tagged hvc1 (what QuickTime takes), 4:2:0 BT.709 + let out = std::process::Command::new(&ffprobe) + .args([ + "-v", + "error", + "-count_frames", + "-select_streams", + "v:0", + "-show_entries", + "stream=codec_name,profile,codec_tag_string,pix_fmt,width,height,nb_read_frames,has_b_frames,color_space", + "-of", + "default=nw=1", + &path, + ]) + .output() + .expect("run ffprobe"); + let info = String::from_utf8_lossy(&out.stdout).to_string(); + for want in [ + "codec_name=hevc", + "profile=Main", + "codec_tag_string=hvc1", + "pix_fmt=yuv420p", + &format!("width={W}"), + &format!("height={H}"), + &format!("nb_read_frames={frames}"), + "has_b_frames=0", + "color_space=bt709", + ] { + assert!(info.contains(want), "{name}: ffprobe lacks `{want}`:\n{info}"); + } + } +} + +/// The sessions are created with a hardware encoder required; ask VideoToolbox whether it agrees, so +/// that a requirement dropped by accident could never become a silent software encode. +#[test] +fn the_sessions_run_on_the_hardware_encoder() { + let config = |profile| VtConfig { + width: W, + height: H, + timescale: 30, + frame_duration: 1, + keyframe_interval: 30, + profile, + rate: VtRate::Vbr { target_kbps: 6000, max_kbps: 9000 }, + }; + for (name, profile) in [("H.264", VtProfile::High), ("HEVC", VtProfile::HevcMain)] { + match VtEncoder::new(config(profile)) { + Ok(enc) => assert!(enc.uses_hardware(), "{name}: VideoToolbox does not report a hardware encoder for the session"), + Err(why) => eprintln!("SKIPPED {name}: no hardware encoder ({why})"), + } + } +} diff --git a/crates/ui-egui/src/panels/export_mode.rs b/crates/ui-egui/src/panels/export_mode.rs index 9c278929..163be490 100644 --- a/crates/ui-egui/src/panels/export_mode.rs +++ b/crates/ui-egui/src/panels/export_mode.rs @@ -461,7 +461,7 @@ fn settings_column(app: &mut FilmcraftApp, ui: &mut egui::Ui, rect: Rect) { } if (s.has_audio() || !s.has_video() - || s.format == Format::H264 + || s.format.is_h26x() || matches!(s.format, Format::ProRes | Format::DnxHr | Format::Apv | Format::Mjpeg) || s.format.is_mxf()) && !s.is_image_sequence() @@ -470,7 +470,7 @@ fn settings_column(app: &mut FilmcraftApp, ui: &mut egui::Ui, rect: Rect) { { audio_section(ui, &mut reg, s, &t, seq_sr); } - if s.format == Format::H264 && section(ui, &mut reg, &mut ex.open_sections, "multiplexer", "Multiplexer", &t) { + if s.format.is_h26x() && section(ui, &mut reg, &mut ex.open_sections, "multiplexer", "Multiplexer", &t) { row(ui, &t, "Multiplexer", |ui| { let cur = if s.multiplexer == Multiplexer::Mp4 { "MP4" } else { "QuickTime" }; if let Some(i) = combo(ui, &mut reg, "export.multiplexer", cur, &opts(&["MP4", "QuickTime"]), 160.0) { @@ -507,7 +507,7 @@ fn settings_column(app: &mut FilmcraftApp, ui: &mut egui::Ui, rect: Rect) { if section(ui, &mut reg, &mut ex.open_sections, "effects", "Effects", &t) { pick_overlay = effects_section(ui, &mut reg, s, &t); } - if matches!(s.format, Format::H264 | Format::ProRes | Format::DnxHr | Format::Apv | Format::Mjpeg) + if matches!(s.format, Format::H264 | Format::Hevc | Format::ProRes | Format::DnxHr | Format::Apv | Format::Mjpeg) && section(ui, &mut reg, &mut ex.open_sections, "metadata", "Metadata", &t) { let m = &mut s.metadata; @@ -593,26 +593,34 @@ fn video_section(ui: &mut egui::Ui, reg: &mut Reg, s: &mut ExportSettings, t: &T }); } match s.video_format() { - Format::H264 => { - row(ui, t, "Profile", |ui| { - let o = [H264Profile::Baseline, H264Profile::Main, H264Profile::High]; - let labels: Vec<(String, bool)> = o.iter().map(|p| (p.label().to_string(), true)).collect(); - if let Some(i) = combo(ui, reg, "export.video.profile", s.h264_profile.label(), &labels, 120.0) { - s.h264_profile = o[i]; - } - }); - row(ui, t, "Level", |ui| { - let levels: [Option; 13] = - [None, Some(30), Some(31), Some(32), Some(40), Some(41), Some(42), Some(50), Some(51), Some(52), Some(60), Some(61), Some(62)]; - let lab = |l: &Option| l.map(|l| format!("{}.{}", l / 10, l % 10)).unwrap_or_else(|| "Auto".into()); - let labels: Vec<(String, bool)> = levels.iter().map(|l| (lab(l), true)).collect(); - if let Some(i) = combo(ui, reg, "export.video.level", &lab(&s.h264_level), &labels, 120.0) { - s.h264_level = levels[i]; - } - }); + f @ (Format::H264 | Format::Hevc) => { + let hevc = f == Format::Hevc; + if hevc { + // the hardware encoder writes one profile: Main, 8-bit 4:2:0, level chosen by the encoder + row(ui, t, "Profile", |ui| ui.label("Main (8-bit)")); + row(ui, t, "Encoder", |ui| ui.label("Hardware (H.265 has no software encoder)")); + } else { + row(ui, t, "Profile", |ui| { + let o = [H264Profile::Baseline, H264Profile::Main, H264Profile::High]; + let labels: Vec<(String, bool)> = o.iter().map(|p| (p.label().to_string(), true)).collect(); + if let Some(i) = combo(ui, reg, "export.video.profile", s.h264_profile.label(), &labels, 120.0) { + s.h264_profile = o[i]; + } + }); + row(ui, t, "Level", |ui| { + let levels: [Option; 13] = + [None, Some(30), Some(31), Some(32), Some(40), Some(41), Some(42), Some(50), Some(51), Some(52), Some(60), Some(61), Some(62)]; + let lab = |l: &Option| l.map(|l| format!("{}.{}", l / 10, l % 10)).unwrap_or_else(|| "Auto".into()); + let labels: Vec<(String, bool)> = levels.iter().map(|l| (lab(l), true)).collect(); + if let Some(i) = combo(ui, reg, "export.video.level", &lab(&s.h264_level), &labels, 120.0) { + s.h264_level = levels[i]; + } + }); + } row(ui, t, "Bitrate Encoding", |ui| { + // a hardware encoder has one pass: H.265 offers no two-pass mode let o = [BitrateMode::Cbr, BitrateMode::Vbr1Pass, BitrateMode::Vbr2Pass]; - let labels: Vec<(String, bool)> = o.iter().map(|m| (m.label().to_string(), true)).collect(); + let labels: Vec<(String, bool)> = o.iter().map(|m| (m.label().to_string(), !(hevc && *m == BitrateMode::Vbr2Pass))).collect(); if let Some(i) = combo(ui, reg, "export.video.bitrateMode", s.bitrate_mode.label(), &labels, 140.0) { s.bitrate_mode = o[i]; } @@ -655,12 +663,15 @@ fn video_section(ui: &mut egui::Ui, reg: &mut Reg, s: &mut ExportSettings, t: &T }); // The operating system's hardware encoder where there is one (macOS); everything it does // not take (two-pass, HDR, MXF) and every machine without one keeps the built-in encoder. - row(ui, t, "Hardware Encoding", |ui| { - let mut on = s.hardware_encoding == HardwareEncoding::Auto; - if check(ui, reg, "export.video.hardwareEncoding", &mut on, "Use the hardware encoder when available") { - s.hardware_encoding = if on { HardwareEncoding::Auto } else { HardwareEncoding::Off }; - } - }); + // H.265 has only the hardware encoder: choosing the format is the opt-in. + if !hevc { + row(ui, t, "Hardware Encoding", |ui| { + let mut on = s.hardware_encoding == HardwareEncoding::Auto; + if check(ui, reg, "export.video.hardwareEncoding", &mut on, "Use the hardware encoder when available") { + s.hardware_encoding = if on { HardwareEncoding::Auto } else { HardwareEncoding::Off }; + } + }); + } } Format::ProRes => { row(ui, t, "Profile", |ui| { @@ -714,13 +725,12 @@ fn audio_section(ui: &mut egui::Ui, reg: &mut Reg, s: &mut ExportSettings, t: &T } } row(ui, t, "Audio Format", |ui| { - let fixed = s.format == Format::H264 && s.multiplexer == Multiplexer::Mp4 || audio_only || s.format.is_mxf(); + let fixed = s.format.is_h26x() && s.multiplexer == Multiplexer::Mp4 || audio_only || s.format.is_mxf(); let cur = match s.audio_codec() { AudioCodec::Aac => "AAC", _ => "Uncompressed (PCM)", }; - let labels = - vec![("AAC".to_string(), !audio_only), ("Uncompressed (PCM)".to_string(), !(s.format == Format::H264 && s.multiplexer == Multiplexer::Mp4))]; + let labels = vec![("AAC".to_string(), !audio_only), ("Uncompressed (PCM)".to_string(), !(s.format.is_h26x() && s.multiplexer == Multiplexer::Mp4))]; if fixed { ui.label(cur); } else if let Some(i) = combo(ui, reg, "export.audio.codec", cur, &labels, 180.0) { diff --git a/crates/ui-egui/tests/export_ui.rs b/crates/ui-egui/tests/export_ui.rs index 4a1dc55a..72d34947 100644 --- a/crates/ui-egui/tests/export_ui.rs +++ b/crates/ui-egui/tests/export_ui.rs @@ -179,6 +179,38 @@ fn export_mode_settings_and_summary() { d.snapshot("export-mode"); } +#[test] +fn h265_shows_the_h264_family_controls_without_the_h264_only_ones() { + use filmcraft_engine::export::{ExportSettings, Format}; + let mut d = Driver::new("h265"); + d.ok("ui.set", json!({"mode": "export"})); + d.frames(4); + // H.264, the default: profile, level and the hardware checkbox + for id in ["export.video.profile", "export.video.level", "export.video.hardwareEncoding", "export.video.bitrateMode"] { + assert!(d.has(id), "H.264: {id}"); + } + d.app().ui.export.settings = ExportSettings { format: Format::Hevc, ..Default::default() }; + d.frames(4); + for id in [ + "export.video.bitrateMode", + "export.video.target", + "export.video.max", + "export.video.keyframeOn", + "export.section.multiplexer", + "export.section.audio", + "export.audio.sampleRate", + ] { + assert!(d.has(id), "H.265: {id}"); + } + // Main is its only profile, the level is the encoder's choice, and hardware is the only encoder + for id in ["export.video.profile", "export.video.level", "export.video.hardwareEncoding"] { + assert!(!d.has(id), "H.265 has no {id}"); + } + let sum = d.ok("ui.elements", json!({"prefix": "export.summary"}))[0]["label"].as_str().unwrap().to_string(); + assert!(sum.contains("HEVC Main") && sum.contains("Target 20.00 Mbps"), "{sum}"); + d.snapshot("export-h265"); +} + #[test] fn preset_manager_search_favourite_save_apply() { let mut d = Driver::new("manager"); diff --git a/docs/adr/0001-platform-ffi.md b/docs/adr/0001-platform-ffi.md index 9458027b..79a52cee 100644 --- a/docs/adr/0001-platform-ffi.md +++ b/docs/adr/0001-platform-ffi.md @@ -83,3 +83,12 @@ deliberate: Everything else in this record applies unchanged: `unsafe` stays in `crates/platform`, every block has a `// SAFETY:` comment, no panic crosses the FFI boundary, the public API is safe, and the crate compiles everywhere (`register()` is a no-op off macOS). + +## Addendum (2026-10-07): hardware H.265 (HEVC) encoding + +The same VideoToolbox session wrapper also creates HEVC sessions (`VtProfile::HevcMain`). The rules +above hold; HEVC adds one difference. The crate has no software HEVC encoder to fall back to, and +there will not be one (pure Rust, clean-room: x265 is GPL), so `Format::Hevc` exists only where the +OS has a hardware encoder. Choosing the format is the opt-in, `filmcraft_export::available` asks a +probe the platform crate registers, and what the hardware path does not take (two-pass, odd sizes, +a machine without the encoder) is an error naming the reason instead of a different encoder. diff --git a/docs/architecture.md b/docs/architecture.md index 22d02250..d545d239 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -72,7 +72,7 @@ and `filmcraft-cli`. | `engine` | L4 | `Session`, command registry, undo history, media pool, jobs, interchange glue | | `ui-egui` | L5 | the egui frontend: docking, panels, timeline, monitors, playback, control-channel handlers | | `automation` | L5 | MCP server (`rmcp`, stdio), headless or bridged to the running app | -| `platform` | L5 | OS media FFI only: hardware video decoding (VideoToolbox H.264 / HEVC on macOS; a no-op elsewhere) behind `codecs::VideoDecoder`, with transparent fallback to our decoders, and opt-in hardware H.264 encoding (VideoToolbox) behind `export::VideoEncoder`, which declines to the built-in encoder for what it does not take. The one crate allowed `unsafe` ([ADR 0001](adr/0001-platform-ffi.md), [README](../crates/platform/README.md)) | +| `platform` | L5 | OS media FFI only: hardware video decoding (VideoToolbox H.264 / HEVC on macOS; a no-op elsewhere) behind `codecs::VideoDecoder`, with transparent fallback to our decoders, and hardware H.264 encoding (VideoToolbox, opt-in) and H.265 encoding (VideoToolbox, the only H.265 encoder; the format exists only where a hardware encoder does) behind `export::VideoEncoder`; H.264 declines to the built-in encoder for what the hardware does not take. The one crate allowed `unsafe` ([ADR 0001](adr/0001-platform-ffi.md), [README](../crates/platform/README.md)) | | `filmcraft` | L6 | desktop binary: eframe/wgpu window, cpal audio output, file dialogs, native macOS menu, TCP control server | | `filmcraft-cli` | L6 | headless CLI: `exec`, `run`, `inspect`, `describe`, `commands`, `import`, `export`, `render`, `probe`, `mcp`; `--bridge` targets the running app | | `filmcraft-web` | L6 | the browser app (wasm32): eframe web runner on WebGPU/WebGL2, Blob-backed services, OPFS recovery, WebAudio, WebCodecs, `window.filmcraft` API ([web.md](web.md)) | diff --git a/docs/performance.md b/docs/performance.md index 1ba2e51b..e29fc717 100644 --- a/docs/performance.md +++ b/docs/performance.md @@ -181,6 +181,43 @@ the media engine removes most of the first and lets the next batch render while encodes (the exporter renders a batch of frames in parallel, then encodes it). What is left is the CPU compositor, the next target of #30 ("GPU export"). +## Results (HW3: VideoToolbox hardware H.265 (HEVC) encoding, against hardware H.264) + +Apple M1 (8 cores, 16 GB), 2026-10-07, single runs on an otherwise idle machine, release build. Both +encoders are the M1's media engine. H.265 is `Format::Hevc` (Main, 8-bit 4:2:0, one-pass VBR, no +B-frames); H.264 is the opt-in hardware encoder of HW2 (High). + +### Speed + +The raw encoders (ffmpeg's `hevc_videotoolbox` and `h264_videotoolbox` on a 1080p25 test pattern, 500 +frames): 166 and 189 frames/s. H.265 is the codec that compresses further, not the faster one to encode. + +The 9:29 timeline of HW2 (camera clip, ProRes 4444 overlays, AAC, loudness normalised to −14 LUFS), 20 Mb/s +H.264 and 10 Mb/s H.265: **204 s and 195 s** (CPU 1042 s and 1029 s), files of 1417 MB and 719 MB. The +same time: the same H.264 export took 193 s and 200 s on other runs. The export is bound by the CPU +compositor and the decoders, not by the encoder, so a different hardware codec does not change it. + +### Quality per bit + +30 s with a banner (15–45 s of that timeline), exported by FilmCraft as ProRes 422 HQ (the reference) +and by each encoder at several bitrates; ffmpeg's `psnr` and `ssim` of the decoded 4:2:0 pictures against +the reference. Average PSNR over the three planes, and SSIM (higher is better): + +| Mb/s | H.264 | H.265 | difference | H.264 Mb/s for H.265's PSNR | +|---|---|---|---|---| +| 20 | 46.88 dB, 0.9902 | 46.91 dB, 0.9903 | +0.03 dB | beyond the range measured | +| 10 | 44.96 dB, 0.9863 | 45.09 dB, 0.9863 | +0.13 dB | 10.5 | +| 6 | 43.72 dB, 0.9833 | 44.31 dB, 0.9843 | +0.59 dB | 7.7 | +| 4 | 42.91 dB, 0.9807 | 43.64 dB, 0.9825 | +0.73 dB | 5.8 | +| 3 | 42.37 dB, 0.9786 | 43.05 dB, 0.9807 | +0.68 dB | 4.3 | + +At 10 Mb/s and above the two give the same picture. At 3–6 Mb/s H.265 reaches the same PSNR with 22–31 % +less bitrate. Its worst frame is better at every bitrate (lowest per-frame PSNR 45.9 against 42.1 dB at +20 Mb/s, 43.1 against 41.5 dB at 10, 39.5 against 37.8 dB at 3). + +So there is no case for a lower default bitrate: the two formats start at the same 20 Mb/s. H.265 +pays off where the file size matters, at 3–6 Mb/s; that 9:29 timeline is 428 MB at 6 Mb/s. + ## Results (GPU1: blend modes on the GPU compositor, #30, before → after) Before = this change with the old whole-frame CPU fallback for non-Normal blend modes put back