platform: H.265 (HEVC) export with VideoToolbox

Adds `Format::Hevc` (Export > Format > H.265 (HEVC); `"format": "hevc"` in
`file.exportMedia`): HEVC Main, 8-bit 4:2:0, in MP4 (`hvc1`) or QuickTime, with AAC audio,
encoded by the macOS hardware encoder. There is no built-in H.265 encoder (one in pure Rust
is a project of its own, and x265 is GPL), so the format exists only where the OS has one.

- `videotoolbox_encode`: the session wrapper creates HEVC sessions too (`VtProfile::HevcMain`;
  the profile also says which codec). Keyframes are the HEVC random access points (NAL types
  16 to 21); the parameter sets are VPS / SPS / PPS; the `hvcC` record in the MP4 sample
  entry is the one VideoToolbox writes (sample description extension atoms), so profile,
  level and flags are the encoder's own. `VtEncoder::uses_hardware()` asks VideoToolbox
  whether the session runs on a hardware encoder (a test checks H.264 and HEVC), so a
  requirement dropped by accident could never become a silent software encode.
- `hardware_encode`: one adapter for both codecs. Choosing the format is the opt-in (the
  `hardware_encoding` setting is for H.264). `hevc_available()` (one small hardware session,
  created once) is registered with the new `filmcraft_export::register_format_probe`, so
  `available(Format::Hevc)`, the format list of Export mode and `exportFormats` follow the
  machine. `Format::has_builtin_encoder` says which formats need such a probe.
- `export`: `Format::is_h26x` where H.264 and H.265 behave alike (MP4 or QuickTime, AAC, size
  estimate, summary line). Two-pass VBR is refused up front ("H.265 export has no two-pass
  mode") and HDR sequences export as SDR, because there is no encoder to fall back to.
- Export mode: H.265 shows H.264's multiplexer, audio, metadata and bitrate controls, without
  profile, level, two-pass or the hardware checkbox, and says "Encoder: Hardware".

Apple M1 (8 cores), 1080p25, a 9:29 timeline (camera clip, ProRes 4444 overlays, AAC,
loudness): hardware H.265 at 10 Mb/s 195 s, hardware H.264 at 20 Mb/s 204 s (CPU 1029 s and
1042 s; the same H.264 export took 193 s and 200 s on other runs). It is not faster: the codec
that compresses further is the more expensive one (raw encoders: 166 and 189 frames/s), and
the export is bound by the CPU compositor, not by the media engine.

Quality per bit, 30 s of the same timeline against a ProRes 422 HQ export (average PSNR over
the three planes, SSIM): 20 Mb/s 46.88 dB / 0.9902 (H.264) and 46.91 dB / 0.9903 (H.265);
10 Mb/s 44.96 / 0.9863 and 45.09 / 0.9863; 6 Mb/s 43.72 / 0.9833 and 44.31 / 0.9843; 4 Mb/s
42.91 / 0.9807 and 43.64 / 0.9825; 3 Mb/s 42.37 / 0.9786 and 43.05 / 0.9807. The same picture
at 10 Mb/s and above; at 3 to 6 Mb/s H.265 needs 22 to 31 % less bitrate for the same PSNR.
Both formats therefore keep the same default bitrate. Details in docs/performance.md (HW3).

Tests: tests/hardware_encode.rs gets the HEVC counterparts of the H.264 ones (what is taken
and declined, a round trip through our HEVC decoder with luma PSNR above 30 dB, keyframes, MP4
and QuickTime exports with AAC, ffmpeg / ffprobe reading `hevc` / `Main` / `hvc1` /
`yuv420p` / BT.709, exact sizes, hostile configurations, encoders dropped at any point, the
format list agreeing with the probe, sessions on the hardware encoder); settings_tests.rs
checks names, ids, defaults, the summary, two-pass refusal and the probe; export_ui.rs the
controls H.265 shows and hides.

Stacked on the hardware H.264 encoder (the first commit of this branch).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
Dario Carnelutti 2026-10-07 14:00:08 -06:00
parent 4cab6bc5da
commit 790b3a0697
18 changed files with 779 additions and 133 deletions

View File

@ -286,7 +286,7 @@ We track two numbers ([ROADMAP.md](ROADMAP.md#honest-assessment-2026-10-05)):
The biggest gaps today: The biggest gaps today:
- **Speed on big footage.** Hardware decoding works on macOS; Windows and Linux have no hardware path yet. Blend modes and the most common effects run on the GPU, but Lumetri, keys and export rendering still run on the CPU; H.264 encoding can use the hardware encoder on macOS (opt-in) ([#30](https://github.com/storytold/filmcraft/issues/30)). - **Speed on big footage.** Hardware decoding works on macOS; Windows and Linux have no hardware path yet. Blend modes and the most common effects run on the GPU, but Lumetri, keys and export rendering still run on the CPU; H.264 encoding can use the hardware encoder on macOS (opt-in), and H.265 export is hardware-only there ([#30](https://github.com/storytold/filmcraft/issues/30)).
- **No plugins.** No VST3 / Audio Units or OpenFX hosting. - **No plugins.** No VST3 / Audio Units or OpenFX hosting.
- **Delivery codecs.** H.264 is our only delivery-codec export; no HEVC or AV1 export yet. - **Delivery codecs.** H.264 is our only delivery-codec export; no HEVC or AV1 export yet.
- **Real-world media and platforms.** Our decoders are bit-exact on conformance streams, but camera and phone files in the wild are less tested. Windows and Linux get far less testing than macOS. - **Real-world media and platforms.** Our decoders are bit-exact on conformance streams, but camera and phone files in the wild are less tested. Windows and Linux get far less testing than macOS.

View File

@ -36,7 +36,7 @@ before choosing work.
| **Correctness in depth** | not tracked | **~60–70%** | A first-time contributor found four real bugs in core paths within hours (#5–#8: AIFF import failed, Slide left linked audio behind, transitions didn't follow ripple trims, export ignored start/end times). User reports: Color Matte always grey with no picker (#29); ALSA audio error on Linux (#23). Settings are "most wired", not all. | | **Correctness in depth** | not tracked | **~60–70%** | A first-time contributor found four real bugs in core paths within hours (#5–#8: AIFF import failed, Slide left linked audio behind, transitions didn't follow ripple trims, export ignored start/end times). User reports: Color Matte always grey with no picker (#29); ALSA audio error on Linux (#23). Settings are "most wired", not all. |
| **Codecs and media** | ~90% | **~75%** | Our decoders are bit-exact on conformance streams. Real camera and phone media (variable frame rate, damaged files, unusual containers) is far less tested. No camera RAW (RED, BRAW, ARRIRAW), no E-AC-3. | | **Codecs and media** | ~90% | **~75%** | Our decoders are bit-exact on conformance streams. Real camera and phone media (variable frame rate, damaged files, unusual containers) is far less tested. No camera RAW (RED, BRAW, ARRIRAW), no E-AC-3. |
| **Export** | ~92% | **~70%** | Delivery: H.264 only (ProRes, DNxHR and image sequences cover mastering). No HEVC or AV1 export. AAF and OMF have never been validated in Avid Media Composer or Pro Tools. Export renders and encodes on the CPU only. | | **Export** | ~92% | **~70%** | Delivery: H.264 only (ProRes, DNxHR and image sequences cover mastering). No HEVC or AV1 export. AAF and OMF have never been validated in Avid Media Composer or Pro Tools. Export renders and encodes on the CPU only. |
| **Performance and hardware** | ~62% | **~35–40%** | Premiere runs effects, decoding and encoding on the GPU and the hardware media engines. Here, hardware decode works on macOS only (VideoToolbox, #33; active in the desktop app only since #41, so releases up to 0.2.1 decoded in software). The GPU does compositing, blend modes (#32) and 31 common effects (GPU2): colour adjustments, crop/flip/transform, blurs, sharpen. Lumetri, keys, masks, the remaining effects and export rendering are CPU; H.264 encoding can use VideoToolbox on macOS (opt-in per export, no B-frames). Linux and Windows have no hardware path. 8K and AV1 aren't real-time. The effect list shows Premiere's "GPU accelerated" badge, but only 31 effect ids run on the GPU here; the rest, Lumetri included, run on the CPU. | | **Performance and hardware** | ~62% | **~35–40%** | Premiere runs effects, decoding and encoding on the GPU and the hardware media engines. Here, hardware decode works on macOS only (VideoToolbox, #33; active in the desktop app only since #41, so releases up to 0.2.1 decoded in software). The GPU does compositing, blend modes (#32) and 31 common effects (GPU2): colour adjustments, crop/flip/transform, blurs, sharpen. Lumetri, keys, masks, the remaining effects and export rendering are CPU; H.264 encoding can use VideoToolbox on macOS (opt-in per export, no B-frames), and H.265 (HEVC) export exists there only, through the same hardware encoder. Linux and Windows have no hardware path. 8K and AV1 aren't real-time. The effect list shows Premiere's "GPU accelerated" badge, but only 31 effect ids run on the GPU here; the rest, Lumetri included, run on the CPU. |
| **Stability** | not tracked | **improving, unproven** | The never-crash pass (no panics in product code, last-resort guards) landed on 2026-10-04. There's no field record from real users' projects yet. | | **Stability** | not tracked | **improving, unproven** | The never-crash pass (no panics in product code, last-resort guards) landed on 2026-10-04. There's no field record from real users' projects yet. |
| **Plugins and pro ecosystem** | not tracked | **~0–10%** | No audio plugin hosting (VST3 / Audio Units) and no third-party video effects (OpenFX). For many professional editors this alone rules FilmCraft out. Team Projects and Productions are out of scope by design. | | **Plugins and pro ecosystem** | not tracked | **~0–10%** | No audio plugin hosting (VST3 / Audio Units) and no third-party video effects (OpenFX). For many professional editors this alone rules FilmCraft out. Team Projects and Productions are out of scope by design. |
| **AI features** | partial | **~25–35%** | Speech to text exists but is off in default builds (the `whisper` feature). Enhance Speech is a DSP chain, not a model. Auto Reframe is approximate. No Generative Extend, and no media-intelligence search or auto colour. | | **AI features** | partial | **~25–35%** | Speech to text exists but is off in default builds (the `whisper` feature). Enhance Speech is a DSP chain, not a model. Auto Reframe is approximate. No Generative Extend, and no media-intelligence search or auto colour. |
@ -51,7 +51,7 @@ In priority order. Agents should prefer this work over adding more checklist ite
- hardware decode on Linux (VA-API) and Windows (Media Foundation); - hardware decode on Linux (VA-API) and Windows (Media Foundation);
- zero-copy decoded frames into wgpu; - zero-copy decoded frames into wgpu;
- the remaining effects on the GPU: Lumetri, keys, Vignette, Video Limiter, masks (31 common effects are done); - the remaining effects on the GPU: Lumetri, keys, Vignette, Video Limiter, masks (31 common effects are done);
- GPU export, then hardware encode: H.264 on macOS is in (opt-in); HEVC and other systems remain. - GPU export, then hardware encode: H.264 and H.265 on macOS are in (H.264 opt-in; H.265 has no other encoder); other systems remain.
2. **A measured parity number.** Add `cargo xtask parity`, run in CI. It should cover menus, 2. **A measured parity number.** Add `cargo xtask parity`, run in CI. It should cover menus,
commands, effects, transitions, panels, preferences and formats against the Premiere snapshot in commands, effects, transitions, panels, preferences and formats against the Premiere snapshot in
`plan/premiere/`. It should report presence and fidelity separately (exact, approximate, stub). `plan/premiere/`. It should report presence and fidelity separately (exact, approximate, stub).
@ -63,7 +63,7 @@ In priority order. Agents should prefer this work over adding more checklist ite
through XML / AAF. through XML / AAF.
5. **Windows and Linux at runtime.** CI that runs the test suite and a headless playback/export 5. **Windows and Linux at runtime.** CI that runs the test suite and a headless playback/export
smoke test on both, not just release builds. Fix the Linux audio path (#23). smoke test on both, not just release builds. Fix the Linux audio path (#23).
6. **Delivery codecs.** HEVC and AV1 export: hardware encoders first, and `rav1e` (BSD-2) for AV1. 6. **Delivery codecs.** HEVC export is in on macOS (VideoToolbox, 8-bit, no software encoder); still to do: HEVC elsewhere, 10-bit / HDR HEVC, and AV1 (`rav1e`, BSD-2).
7. **Plugin hosting** (VST3 / Audio Units, then OpenFX). Needs FFI, so it falls under the isolated 7. **Plugin hosting** (VST3 / Audio Units, then OpenFX). Needs FFI, so it falls under the isolated
`unsafe` crate rule (AGENTS.md §0.3). **Owner decision needed** before starting. `unsafe` crate rule (AGENTS.md §0.3). **Owner decision needed** before starting.
8. **AI features** with openly licensed local models: transcription on by default, text-based 8. **AI features** with openly licensed local models: transcription on by default, text-based

View File

@ -30,7 +30,7 @@ SUBCOMMANDS
[--scale f] [--quality 0-100] [--no-audio] [--queue] [--scale f] [--quality 0-100] [--no-audio] [--queue]
export the active sequence and wait for it to finish: with an export the active sequence and wait for it to finish: with an
export preset (`export --list-presets`; built-in or the user's), or export preset (`export --list-presets`; built-in or the user's), or
a format (h264|prores|dnxhr|apv|mjpeg|mxf-op1a|mxf-opatom|png|tiff|bmp|gif|wav|aiff, guessed a format (h264|hevc|prores|dnxhr|apv|mjpeg|mxf-op1a|mxf-opatom|png|tiff|bmp|gif|wav|aiff, guessed
from the extension); --range entire|inOut|workArea, or a custom from the extension); --range entire|inOut|workArea, or a custom
range in seconds; --settings is ExportSettings JSON merged over the range in seconds; --settings is ExportSettings JSON merged over the
preset; --scale renders at a fraction of the frame size (0.5 = preset; --scale renders at a fraction of the frame size (0.5 =

View File

@ -1018,7 +1018,7 @@ fn build() -> Vec<CommandSpec> {
"Media…", "Media…",
["File", "Export"], ["File", "Export"],
None, None,
r#"{"path":str,"preset":str?,"settings":ExportSettings?,"format":"h264|prores|dnxhr|apv|mjpeg|mxf-op1a|mxf-opatom|png|tiff|bmp|gif|wav|aiff"?,"width":u32?,"height":u32?,"fps":f64?,"bitrateKbps":u32?,"bitrateMode":"cbr|vbr1Pass|vbr2Pass"?,"hardwareEncoding":"off|auto"?,"scale":f32=1,"audio":bool=true,"quality":0..100,"burnCaptions":bool=false,"captionSidecar":"srt|vtt"?,"loudnessLufs":f64?,"proresProfile":"proxy|lt|standard|hq"?,"dnxProfile":"lb|sq|hq|hqx"?,"apvProfile":"422-10|422-12|444-10|444-12"?,"mxfVideoCodec":"dnxhr|proRes|h264"?,"sequence":id?,"range":"entire|inOut|workArea|custom"?,"startSeconds":f64?,"endSeconds":f64?,"wait":bool=false}"#, r#"{"path":str,"preset":str?,"settings":ExportSettings?,"format":"h264|hevc|prores|dnxhr|apv|mjpeg|mxf-op1a|mxf-opatom|png|tiff|bmp|gif|wav|aiff"?,"width":u32?,"height":u32?,"fps":f64?,"bitrateKbps":u32?,"bitrateMode":"cbr|vbr1Pass|vbr2Pass"?,"hardwareEncoding":"off|auto"?,"scale":f32=1,"audio":bool=true,"quality":0..100,"burnCaptions":bool=false,"captionSidecar":"srt|vtt"?,"loudnessLufs":f64?,"proresProfile":"proxy|lt|standard|hq"?,"dnxProfile":"lb|sq|hq|hqx"?,"apvProfile":"422-10|422-12|444-10|444-12"?,"mxfVideoCodec":"dnxhr|proRes|h264"?,"sequence":id?,"range":"entire|inOut|workArea|custom"?,"startSeconds":f64?,"endSeconds":f64?,"wait":bool=false}"#,
has_seq, has_seq,
crate::export_tools::export_media crate::export_tools::export_media
), ),

View File

@ -82,7 +82,7 @@ pub struct Exporter {
/// Whether [`Exporter`] handles a format. /// Whether [`Exporter`] handles a format.
pub fn stepped(format: Format) -> bool { pub fn stepped(format: Format) -> bool {
matches!(format, Format::H264 | Format::ProRes | Format::DnxHr | Format::Apv | Format::Mjpeg | Format::MxfOp1a | Format::MxfOpAtom) matches!(format, Format::H264 | Format::Hevc | Format::ProRes | Format::DnxHr | Format::Apv | Format::Mjpeg | Format::MxfOp1a | Format::MxfOpAtom)
} }
fn make_venc(settings: &ExportSettings, w: u32, h: u32, rate: FrameRate) -> Result<Box<dyn VideoEncoder>> { fn make_venc(settings: &ExportSettings, w: u32, h: u32, rate: FrameRate) -> Result<Box<dyn VideoEncoder>> {
@ -102,7 +102,7 @@ impl Exporter {
} }
settings.validate()?; settings.validate()?;
let q = project.sequence(seq).ok_or(ExportError::NoSequence)?; let q = project.sequence(seq).ok_or(ExportError::NoSequence)?;
// HDR sequences export HDR (H.264 / ProRes / DNxHR / APV) unless SDR is asked for // HDR sequences export HDR (H.264 / ProRes / DNxHR / APV) unless SDR is asked for (H.265 export is 8-bit SDR)
let pipe = q.settings.color; let pipe = q.settings.color;
let hdr_out = pipe.working.is_hdr() && !settings.sdr && matches!(settings.video_format(), Format::H264 | Format::ProRes | Format::DnxHr | Format::Apv); let hdr_out = pipe.working.is_hdr() && !settings.sdr && matches!(settings.video_format(), Format::H264 | Format::ProRes | Format::DnxHr | Format::Apv);
let mut settings = settings.clone(); let mut settings = settings.clone();
@ -128,7 +128,7 @@ impl Exporter {
progress.set_status(format!("Exporting {} frames ({})", nframes, settings.format.label())); progress.set_status(format!("Exporting {} frames ({})", nframes, settings.format.label()));
} }
let venc = make_venc(&settings, pipe.w, pipe.h, pipe.rate)?; let venc = make_venc(&settings, pipe.w, pipe.h, pipe.rate)?;
let brand = if settings.format == Format::H264 && settings.multiplexer == Multiplexer::Mp4 { Brand::Mp4 } else { Brand::Mov }; let brand = if settings.format.is_h26x() && settings.multiplexer == Multiplexer::Mp4 { Brand::Mp4 } else { Brand::Mov };
let audio = if settings.has_audio() { Some(AudioOut::new(project.clone(), seq, &settings, range)?) } else { None }; let audio = if settings.has_audio() { Some(AudioOut::new(project.clone(), seq, &settings, range)?) } else { None };
let aenc: Option<Box<dyn AudioEncoder>> = match (&audio, settings.audio_codec()) { let aenc: Option<Box<dyn AudioEncoder>> = match (&audio, settings.audio_codec()) {
(Some(a), AudioCodec::Aac) => { (Some(a), AudioCodec::Aac) => {

View File

@ -61,6 +61,11 @@ pub enum Format {
#[default] #[default]
#[serde(rename = "h264", alias = "H264")] #[serde(rename = "h264", alias = "H264")]
H264, H264,
/// MPEG-4 (or QuickTime, see [`Multiplexer`]), H.265 / HEVC Main (8-bit) video + AAC audio. There is no
/// built-in encoder: a platform hardware encoder registers one ([`register_encoder`]) and says so
/// with [`register_format_probe`]; [`available`] is false without it.
#[serde(rename = "hevc", alias = "Hevc")]
Hevc,
/// QuickTime, Apple ProRes 422 (HQ unless the settings pick another flavour) + PCM. /// QuickTime, Apple ProRes 422 (HQ unless the settings pick another flavour) + PCM.
#[serde(rename = "prores", alias = "ProRes")] #[serde(rename = "prores", alias = "ProRes")]
ProRes, ProRes,
@ -102,6 +107,7 @@ impl Format {
pub fn from_name(s: &str) -> Option<Format> { pub fn from_name(s: &str) -> Option<Format> {
Some(match s.to_ascii_lowercase().replace([' ', '-', '_', '.'], "").as_str() { Some(match s.to_ascii_lowercase().replace([' ', '-', '_', '.'], "").as_str() {
"h264" | "mp4" | "avc" | "m4v" => Format::H264, "h264" | "mp4" | "avc" | "m4v" => Format::H264,
"hevc" | "h265" | "hvc1" | "hev1" | "x265" => Format::Hevc,
"prores" | "mov" | "appleprores" => Format::ProRes, "prores" | "mov" | "appleprores" => Format::ProRes,
"dnxhr" | "dnxhd" | "dnx" | "avid" | "aviddnxhr" | "aviddnxhd" | "vc3" => Format::DnxHr, "dnxhr" | "dnxhd" | "dnx" | "avid" | "aviddnxhr" | "aviddnxhd" | "vc3" => Format::DnxHr,
"apv" | "apv1" => Format::Apv, "apv" | "apv1" => Format::Apv,
@ -121,6 +127,7 @@ impl Format {
pub fn id(self) -> &'static str { pub fn id(self) -> &'static str {
match self { match self {
Format::H264 => "h264", Format::H264 => "h264",
Format::Hevc => "hevc",
Format::ProRes => "prores", Format::ProRes => "prores",
Format::DnxHr => "dnxhr", Format::DnxHr => "dnxhr",
Format::Apv => "apv", Format::Apv => "apv",
@ -137,7 +144,7 @@ impl Format {
} }
pub fn extension(self) -> &'static str { pub fn extension(self) -> &'static str {
match self { match self {
Format::H264 => "mp4", Format::H264 | Format::Hevc => "mp4",
Format::ProRes | Format::DnxHr | Format::Apv | Format::Mjpeg => "mov", Format::ProRes | Format::DnxHr | Format::Apv | Format::Mjpeg => "mov",
Format::PngSequence => "png", Format::PngSequence => "png",
Format::TiffSequence => "tif", Format::TiffSequence => "tif",
@ -151,6 +158,7 @@ impl Format {
pub fn label(self) -> &'static str { pub fn label(self) -> &'static str {
match self { match self {
Format::H264 => "H.264", Format::H264 => "H.264",
Format::Hevc => "H.265 (HEVC)",
Format::ProRes => "Apple ProRes", Format::ProRes => "Apple ProRes",
Format::DnxHr => "Avid DNxHR", Format::DnxHr => "Avid DNxHR",
Format::Apv => "APV", Format::Apv => "APV",
@ -169,8 +177,18 @@ impl Format {
pub fn is_mxf(self) -> bool { pub fn is_mxf(self) -> bool {
matches!(self, Format::MxfOp1a | Format::MxfOpAtom) matches!(self, Format::MxfOp1a | Format::MxfOpAtom)
} }
pub const ALL: [Format; 13] = [ /// Whether the crate carries an encoder for the format; the others need one registered at runtime
/// ([`register_format_probe`]).
pub fn has_builtin_encoder(self) -> bool {
self != Format::Hevc
}
/// H.264 or H.265: MPEG-4 (or QuickTime) with AAC audio, set up with the same bitrate controls.
pub fn is_h26x(self) -> bool {
matches!(self, Format::H264 | Format::Hevc)
}
pub const ALL: [Format; 14] = [
Format::H264, Format::H264,
Format::Hevc,
Format::ProRes, Format::ProRes,
Format::DnxHr, Format::DnxHr,
Format::Apv, Format::Apv,
@ -461,6 +479,9 @@ impl ExportSettings {
if self.effects.image_overlay.enabled && self.effects.image_overlay.path.trim().is_empty() { if self.effects.image_overlay.enabled && self.effects.image_overlay.path.trim().is_empty() {
return Err(ExportError::Unsupported("image overlay: no image file chosen".into())); return Err(ExportError::Unsupported("image overlay: no image file chosen".into()));
} }
if self.format == Format::Hevc && self.bitrate_mode == BitrateMode::Vbr2Pass {
return Err(ExportError::Unsupported("H.265 export has no two-pass mode: choose CBR or VBR, 1 pass".into()));
}
Ok(()) Ok(())
} }
} }
@ -576,10 +597,30 @@ pub fn register_audio_encoder(f: AudioEncoderFactory) {
audio_factories().write().unwrap_or_else(|e| e.into_inner()).insert(0, f); audio_factories().write().unwrap_or_else(|e| e.into_inner()).insert(0, f);
} }
/// Whether a format can currently be exported. Every [`Format`] has a built-in encoder; this stays type FormatProbe = (Format, fn() -> bool);
/// as the hook for formats whose encoders are registered at runtime.
fn format_probes() -> &'static RwLock<Vec<FormatProbe>> {
static P: OnceLock<RwLock<Vec<FormatProbe>>> = OnceLock::new();
P.get_or_init(|| RwLock::new(Vec::new()))
}
/// Say how to find out whether the encoder of a format without a built-in one ([`Format::has_builtin_encoder`])
/// works on this machine: `probe` runs when [`available`] asks (it should cache its answer).
/// Registering the same probe twice is harmless.
pub fn register_format_probe(format: Format, probe: fn() -> bool) {
let mut g = format_probes().write().unwrap_or_else(|e| e.into_inner());
if !g.iter().any(|(f, p)| *f == format && std::ptr::fn_addr_eq(*p, probe)) {
g.push((format, probe));
}
}
/// Whether a format can currently be exported: it has a built-in encoder, or a registered probe
/// says its encoder works here.
pub fn available(format: Format) -> bool { pub fn available(format: Format) -> bool {
Format::ALL.contains(&format) if !Format::ALL.contains(&format) {
return false;
}
format.has_builtin_encoder() || format_probes().read().unwrap_or_else(|e| e.into_inner()).iter().any(|(f, probe)| *f == format && probe())
} }
struct MjpegEncoder { struct MjpegEncoder {
@ -1241,7 +1282,7 @@ pub fn export(project: &Arc<Project>, seq: ItemId, settings: &ExportSettings, so
} }
(total, count) (total, count)
} }
Format::H264 | Format::ProRes | Format::DnxHr | Format::Apv | Format::Mjpeg | Format::MxfOp1a | Format::MxfOpAtom => { Format::H264 | Format::Hevc | Format::ProRes | Format::DnxHr | Format::Apv | Format::Mjpeg | Format::MxfOp1a | Format::MxfOpAtom => {
// Handled by the stepped exporter above; reaching here would be a dispatch bug. // Handled by the stepped exporter above; reaching here would be a dispatch bug.
return Err(ExportError::Unsupported(format!("{:?} must run as a stepped export", settings.format))); return Err(ExportError::Unsupported(format!("{:?} must run as a stepped export", settings.format)));
} }

View File

@ -405,8 +405,8 @@ impl ExportSettings {
pub fn audio_codec(&self) -> AudioCodec { pub fn audio_codec(&self) -> AudioCodec {
match (self.format, self.audio.codec) { match (self.format, self.audio.codec) {
(Format::Wav | Format::Aiff | Format::MxfOp1a | Format::MxfOpAtom, _) => AudioCodec::Pcm, (Format::Wav | Format::Aiff | Format::MxfOp1a | Format::MxfOpAtom, _) => AudioCodec::Pcm,
(Format::H264, _) if self.multiplexer == Multiplexer::Mp4 => AudioCodec::Aac, (f, _) if f.is_h26x() && self.multiplexer == Multiplexer::Mp4 => AudioCodec::Aac,
(_, AudioCodec::Auto) if self.format == Format::H264 => AudioCodec::Aac, (_, AudioCodec::Auto) if self.format.is_h26x() => AudioCodec::Aac,
(_, AudioCodec::Auto) => AudioCodec::Pcm, (_, AudioCodec::Auto) => AudioCodec::Pcm,
(_, c) => c, (_, c) => c,
} }
@ -424,7 +424,7 @@ impl ExportSettings {
/// File extension of the output. /// File extension of the output.
pub fn extension(&self) -> &'static str { pub fn extension(&self) -> &'static str {
if self.format == Format::H264 && self.multiplexer == Multiplexer::Mov { "mov" } else { self.format.extension() } if self.format.is_h26x() && self.multiplexer == Multiplexer::Mov { "mov" } else { self.format.extension() }
} }
/// Estimated output size in bytes for `duration` of a sequence (`seq_w`×`seq_h` at `seq_rate`). /// Estimated output size in bytes for `duration` of a sequence (`seq_w`×`seq_h` at `seq_rate`).
@ -434,7 +434,7 @@ impl ExportSettings {
let fps = r.rate.num as f64 / r.rate.den as f64; let fps = r.rate.num as f64 / r.rate.den as f64;
let px = r.width as f64 * r.height as f64; let px = r.width as f64 * r.height as f64;
let video_bps = match self.video_format() { let video_bps = match self.video_format() {
Format::H264 => r.target_kbps as f64 * 1000.0, Format::H264 | Format::Hevc => r.target_kbps as f64 * 1000.0,
Format::ProRes => crate::prores_profile(&self.prores_profile).nominal_mbps_1080p30() * 1e6 * px / (1920.0 * 1080.0) * fps / 29.97, Format::ProRes => crate::prores_profile(&self.prores_profile).nominal_mbps_1080p30() * 1e6 * px / (1920.0 * 1080.0) * fps / 29.97,
Format::DnxHr => { Format::DnxHr => {
// nominal 1080p29.97 data rates of the DNxHR profiles (Mb/s) // nominal 1080p29.97 data rates of the DNxHR profiles (Mb/s)
@ -507,6 +507,14 @@ impl ExportSettings {
}; };
v += &format!(", keyframe every {} frames", r.keyint); v += &format!(", keyframe every {} frames", r.keyint);
} }
Format::Hevc => {
v += &format!(", HEVC Main (hardware encoder), {}", self.bitrate_mode.label());
v += &match self.bitrate_mode {
BitrateMode::Cbr => format!(", {}", mbps(r.target_kbps)),
_ => format!(", Target {}, Max {}", mbps(r.target_kbps), mbps(r.max_kbps)),
};
v += &format!(", keyframe every {} frames", r.keyint);
}
Format::ProRes => { Format::ProRes => {
use filmcraft_prores::Profile; use filmcraft_prores::Profile;
v += match crate::prores_profile(&self.prores_profile) { v += match crate::prores_profile(&self.prores_profile) {
@ -537,8 +545,8 @@ impl ExportSettings {
"No audio".to_string() "No audio".to_string()
}; };
let container = match self.format { let container = match self.format {
Format::H264 if self.multiplexer == Multiplexer::Mov => "QuickTime", Format::H264 | Format::Hevc if self.multiplexer == Multiplexer::Mov => "QuickTime",
Format::H264 => "MP4", Format::H264 | Format::Hevc => "MP4",
Format::ProRes | Format::DnxHr | Format::Apv | Format::Mjpeg => "QuickTime", Format::ProRes | Format::DnxHr | Format::Apv | Format::Mjpeg => "QuickTime",
Format::MxfOp1a | Format::MxfOpAtom => self.mxf_video_codec.label(), Format::MxfOp1a | Format::MxfOpAtom => self.mxf_video_codec.label(),
Format::PngSequence | Format::TiffSequence | Format::BmpSequence => "Image sequence", Format::PngSequence | Format::TiffSequence | Format::BmpSequence => "Image sequence",

View File

@ -410,3 +410,55 @@ fn aac_bitrate_is_capped_for_low_sample_rates() {
assert!(enc.is_ok(), "{rate} Hz × {ch}: {:?}", enc.err()); assert!(enc.is_ok(), "{rate} Hz × {ch}: {:?}", enc.err());
} }
} }
#[test]
fn h265_is_a_format_that_needs_a_registered_encoder() {
use std::sync::atomic::{AtomicBool, Ordering};
// names, ids, files and serialized settings
for name in ["hevc", "H.265", "h265", "HVC1", "x265"] {
assert_eq!(Format::from_name(name), Some(Format::Hevc), "{name}");
}
assert_eq!(Format::from_name(Format::Hevc.id()), Some(Format::Hevc));
assert_eq!((Format::Hevc.extension(), Format::Hevc.label()), ("mp4", "H.265 (HEVC)"));
assert!(Format::ALL.contains(&Format::Hevc) && Format::ALL.len() == 14);
let s: ExportSettings = serde_json::from_value(serde_json::json!({"format": "hevc"})).unwrap();
assert_eq!(s.format, Format::Hevc);
assert_eq!(serde_json::to_value(&s).unwrap()["format"], "hevc");
// it is H.264's family: MP4 with AAC, or QuickTime; the same size estimate
let h = ExportSettings { format: Format::Hevc, ..Default::default() };
assert_eq!((h.audio_codec(), h.extension()), (AudioCodec::Aac, "mp4"));
assert_eq!(ExportSettings { multiplexer: Multiplexer::Mov, ..h.clone() }.extension(), "mov");
// 10 s of 20 Mbps video + 320 kbps AAC ≈ 25.4 MB, like H.264
let b = h.estimate_bytes(1920, 1080, FrameRate::FPS_25, 48_000, Tick(10 * TICKS_PER_SECOND));
assert!((b as f64 - 25.4e6).abs() < 0.2e6, "{b}");
let summary = h.summary(1920, 1080, FrameRate::FPS_25, 48_000, Tick(10 * TICKS_PER_SECOND));
assert_eq!(summary.format, "H.265 (HEVC) (MP4)");
assert!(summary.video.contains("HEVC Main") && summary.video.contains("Target 20.00 Mbps"), "{}", summary.video);
// a hardware encoder has one pass
let e = ExportSettings { bitrate_mode: BitrateMode::Vbr2Pass, ..h.clone() }.validate().unwrap_err();
assert!(e.to_string().contains("two-pass"), "{e}");
assert!(ExportSettings { bitrate_mode: BitrateMode::Cbr, ..h.clone() }.validate().is_ok());
// formats with a built-in encoder are always available; this one only when a registered probe says so
assert!(Format::ALL.iter().filter(|f| f.has_builtin_encoder()).all(|f| available(*f)));
assert!(!Format::Hevc.has_builtin_encoder());
static HERE: AtomicBool = AtomicBool::new(false);
fn probe() -> bool {
HERE.load(Ordering::SeqCst)
}
assert!(!available(Format::Hevc), "nothing registered an encoder for it");
// and an export says so, honestly
let (p, seq, m) = matte([0.0, 0.0, 1.0, 1.0], 160, 90, None);
let sc = Scratch::new("h265-missing");
let err = export(&p, seq, &ExportSettings { path: sc.path("x.mp4"), include_audio: false, ..h.clone() }, &m, &Progress::default()).unwrap_err().to_string();
assert!(err.contains("H.265") && err.contains("encoder not available"), "{err}");
register_format_probe(Format::Hevc, probe);
register_format_probe(Format::Hevc, probe); // the same probe twice is harmless
assert!(!available(Format::Hevc));
HERE.store(true, Ordering::SeqCst);
assert!(available(Format::Hevc));
HERE.store(false, Ordering::SeqCst);
assert!(!available(Format::Hevc));
}

View File

@ -1,8 +1,8 @@
# filmcraft-platform # filmcraft-platform
OS media integration for FilmCraft (layer L5): hardware video decoding through the operating OS media integration for FilmCraft (layer L5): hardware video decoding through the operating
system's codecs, behind `filmcraft_codecs::VideoDecoder`, and hardware H.264 encoding, behind system's codecs, behind `filmcraft_codecs::VideoDecoder`, and hardware H.264 and H.265 (HEVC)
`filmcraft_export::VideoEncoder`. It holds OS media FFI and nothing else. encoding, behind `filmcraft_export::VideoEncoder`. It holds OS media FFI and nothing else.
It is the one crate of the workspace allowed to contain `unsafe`, under the rules of It is the one crate of the workspace allowed to contain `unsafe`, under the rules of
[ADR 0001](../../docs/adr/0001-platform-ffi.md) and [AGENTS.md](../../AGENTS.md) §0.3. [ADR 0001](../../docs/adr/0001-platform-ffi.md) and [AGENTS.md](../../AGENTS.md) §0.3.
@ -57,6 +57,36 @@ let availability = filmcraft_platform::register(); // Available("VideoToolbox")
- The first frame is completed straight away: the SPS / PPS the container needs come with the first - The first frame is completed straight away: the SPS / PPS the container needs come with the first
compressed frame, and the muxer asks for them after the first group of pictures. compressed frame, and the muxer asks for them after the first group of pictures.
## Hardware H.265 (HEVC) encoding (macOS)
The same session wrapper, created for `kCMVideoCodecType_HEVC` with the HEVC Main profile
(`VtProfile::HevcMain`: the profile also says which codec the session is). `Format::Hevc` has no
built-in encoder, which changes three things from H.264:
- **Choosing the format is the opt-in.** No `hardware_encoding` setting: Export ▸ Format ▸ H.265
(HEVC), or `"format": "hevc"` in `file.exportMedia` (`h265` is accepted too). The format is
listed as available only on a machine with a hardware HEVC encoder: `register()` hands
`hardware_encode::hevc_available` (one small hardware session, created on the first question) to
`filmcraft_export::register_format_probe`.
- **There is no fallback encoder.** What the hardware path does not take is an error, not a
different encoder: two-pass VBR is refused up front (`ExportSettings::validate`), HDR sequences
are exported as SDR (the H.265 path is 8-bit), and odd sizes, non-square pixels or a machine
without the encoder end in "H.265 (HEVC) encoder not available yet".
The sessions are created with a hardware encoder *required*, and `VtEncoder::uses_hardware()` asks
VideoToolbox whether it agrees (`UsingHardwareAcceleratedVideoEncoder`; a test checks it for H.264 and
HEVC), so a requirement dropped by accident could never turn into a silent software encode. Export
mode says "Encoder: Hardware" for H.265, and the summary line "HEVC Main (hardware encoder)".
What it takes is H.264's list: MP4 (`hvc1`, the tag QuickTime reads) or QuickTime, AAC
audio, 8-bit 4:2:0 BT.709 limited range, even sizes up to 8192, constant or one-pass variable
bitrate, the keyframe distance (closed GOPs, IDR pictures), and no B-frames. Keyframes are the
random access points of types 16–21 (BLA, IDR, CRA).
The sample entry's `hvcC` record is the one VideoToolbox wrote for the stream (read from the
format description's sample description extension atoms, with the VPS / SPS / PPS), so profile,
level and flags are the encoder's own. If it is missing the export stops with that reason.
## Guarantees ## Guarantees
- **Never undecodable:** the factory declines (returns `None`, so the software decoder is used) - **Never undecodable:** the factory declines (returns `None`, so the software decoder is used)
@ -78,7 +108,7 @@ let availability = filmcraft_platform::register(); // Available("VideoToolbox")
| `tests/videotoolbox.rs` (macOS) | H.264 High, HEVC Main (open GOP: CRA + RASL) and HEVC Main 10, 640×360 (coded 368: cropping) with B-frames: every picture **bit-exact** with our software decoder, same pts order, count, colour and aspect, also after `reset` + reseek to every later sync sample, mid-stream `flush`, and a full pass after resets; forced mid-stream failures (`VtDecoder::fail_after`) at five points continue with the software decoder's exact output; seeded mutation of samples and parameter sets (bit flips, truncation, corrupt length prefixes) never panics or hangs; HEVC 4:2:2 10-bit is bit-exact with ffmpeg's decode | | `tests/videotoolbox.rs` (macOS) | H.264 High, HEVC Main (open GOP: CRA + RASL) and HEVC Main 10, 640×360 (coded 368: cropping) with B-frames: every picture **bit-exact** with our software decoder, same pts order, count, colour and aspect, also after `reset` + reseek to every later sync sample, mid-stream `flush`, and a full pass after resets; forced mid-stream failures (`VtDecoder::fail_after`) at five points continue with the software decoder's exact output; seeded mutation of samples and parameter sets (bit flips, truncation, corrupt length prefixes) never panics or hangs; HEVC 4:2:2 10-bit is bit-exact with ffmpeg's decode |
| `tests/fallback.rs` (every OS) | `HybridDecoder` with a stand-in hardware decoder failing after N samples (every sync sample ± a few, first / last sample, after a seek): output identical to the software decoder; in-band parameter sets identical to the sample entry's stay in hardware, different ones switch to software | | `tests/fallback.rs` (every OS) | `HybridDecoder` with a stand-in hardware decoder failing after N samples (every sync sample ± a few, first / last sample, after a seek): output identical to the software decoder; in-band parameter sets identical to the sample entry's stay in hardware, different ones switch to software |
| `tests/setting.rs` | Hardware decoding Off gives the software decoder through `make_video_decoder` and the media stack (no hardware frames); Auto gives VideoToolbox where available | | `tests/setting.rs` | Hardware decoding Off gives the software decoder through `make_video_decoder` and the media stack (no hardware frames); Auto gives VideoToolbox where available |
| `tests/hardware_encode.rs` (macOS) | what the hardware path takes and declines; round trip through our software decoder (every picture, in order, luma PSNR above 30 dB, keyframes no further apart than asked, no composition offsets); an export through `filmcraft_export` that decodes in our decoder and in ffmpeg / ffprobe (profile, size, frame count, BT.709); the built-in encoder still exporting everything hardware declines; exact output size at sizes that are not multiples of 16; hostile configurations (zero, huge, odd sizes, frame rates, bitrates, keyframe intervals, wrong planes) give errors and never panic; encoders dropped at any point do not crash or hang | | `tests/hardware_encode.rs` (macOS) | what the hardware path takes and declines; round trip through our software decoder (every picture, in order, luma PSNR above 30 dB, keyframes no further apart than asked, no composition offsets); an export through `filmcraft_export` that decodes in our decoder and in ffmpeg / ffprobe (profile, size, frame count, BT.709); the built-in encoder still exporting everything hardware declines; exact output size at sizes that are not multiples of 16; hostile configurations (zero, huge, odd sizes, frame rates, bitrates, keyframe intervals, wrong planes) give errors and never panic; encoders dropped at any point do not crash or hang. The same for **HEVC**: Main profile, 8-bit 4:2:0, `hvc1` entry with VPS / SPS / PPS and 4-byte lengths, MP4 and QuickTime, AAC audio, two-pass refused, the format list agreeing with the probe, ffprobe reading `codec_name=hevc`, `profile=Main`, `codec_tag_string=hvc1`, `pix_fmt=yuv420p`, BT.709 |
Fixtures are made with ffmpeg into `target/fixtures/platform/` (generator only, never linked); Fixtures are made with ffmpeg into `target/fixtures/platform/` (generator only, never linked);
tests skip without ffmpeg or without a hardware decoder. tests skip without ffmpeg or without a hardware decoder.
@ -91,6 +121,10 @@ overlays, AAC, loudness) exported in 311 s against 793 s with the built-in encod
for its densest 134 s), the outputs at SSIM 0.990 / PSNR 46.5 dB. Details in for its densest 134 s), the outputs at SSIM 0.990 / PSNR 46.5 dB. Details in
[docs/performance.md](../../docs/performance.md). [docs/performance.md](../../docs/performance.md).
Hardware H.265 encoding, same machine: no faster than H.264 (the 9:29 timeline in 195 s against
204 s, both bound by the CPU compositor); same picture quality at 10 Mb/s and above, and 22–31 % less
bitrate for the same PSNR at 3–6 Mb/s ([docs/performance.md](../../docs/performance.md), HW3).
Hardware decoding, M4 Pro: Hardware decoding, M4 Pro:
M4 Pro, load 150–190 (`cargo xtask bench --hw off|auto`): CPU per decoded frame H.264 2160p M4 Pro, load 150–190 (`cargo xtask bench --hw off|auto`): CPU per decoded frame H.264 2160p
@ -100,5 +134,5 @@ playback with no dropped frames at Full, 1/2 and 1/4. Details in
## Not yet ## Not yet
Zero-copy upload of `CVPixelBuffer`s into wgpu textures; hardware HEVC encoding and B-frames; Media Foundation / Zero-copy upload of `CVPixelBuffer`s into wgpu textures; B-frames; 10-bit and HDR HEVC (Main 10);
D3D11 (Windows) and VA-API (Linux) decoders; field-coded H.264. Media Foundation / D3D11 (Windows) and VA-API (Linux) decoders; field-coded H.264.

View File

@ -1,22 +1,30 @@
//! The VideoToolbox H.264 encoder behind `filmcraft_export::VideoEncoder` (macOS; safe code around //! The VideoToolbox H.264 and HEVC encoders behind `filmcraft_export::VideoEncoder` (macOS; safe code
//! [`crate::videotoolbox_encode`]). //! around [`crate::videotoolbox_encode`]).
//! //!
//! Hardware encoding is opt-in per export (`ExportSettings::hardware_encoding`, `Auto`): its output depends //! H.264 hardware encoding is opt-in per export (`ExportSettings::hardware_encoding`, `Auto`): its output
//! on the machine, while the built-in encoder's is byte-identical everywhere. The factory declines, //! depends on the machine, while the built-in encoder's is byte-identical everywhere. The factory
//! so the built-in encoder is used, for everything the hardware path does not take: other formats, //! declines, so the built-in encoder is used, for everything the hardware path does not take: other
//! MXF (Annex B), two-pass VBR, HDR, odd picture sizes, non-square pixels, and any configuration //! formats, MXF (Annex B), two-pass VBR, HDR, odd picture sizes, non-square pixels, and any
//! VideoToolbox cannot create a hardware session for. A hardware encoder that fails in the middle of an export reports //! configuration VideoToolbox cannot create a hardware session for. A hardware encoder that fails in
//! an error (the export stops); it cannot hand the stream over to another encoder. //! the middle of an export reports an error (the export stops); it cannot hand the stream over to
//! another encoder.
//!
//! HEVC (`Format::Hevc`) has no built-in encoder, so choosing the format is the opt-in, the same
//! configurations are declined, and a declined export fails with "encoder not available" instead of
//! falling back; [`hevc_available`] tells the format list whether this machine can do it at all.
use std::sync::OnceLock;
use filmcraft_export::{BitrateMode, EncodedPacket, EncoderFrame, ExportError, ExportSettings, Format, H264Pass, H264Profile, HardwareEncoding, VideoEncoder}; use filmcraft_export::{BitrateMode, EncodedPacket, EncoderFrame, ExportError, ExportSettings, Format, H264Pass, H264Profile, HardwareEncoding, VideoEncoder};
use filmcraft_isobmff::{AvcConfig, SampleEntry}; use filmcraft_isobmff::{AvcConfig, HevcConfig, SampleEntry};
use filmcraft_time::FrameRate; use filmcraft_time::FrameRate;
use crate::videotoolbox_encode::{VtConfig, VtEncoder, VtPacket, VtProfile, VtRate}; use crate::videotoolbox_encode::{ParamSets, VtCodec, VtConfig, VtEncoder, VtPacket, VtProfile, VtRate};
/// The hardware encoder's configuration for an export, or why it is not used for it. /// The hardware encoder's configuration for an export, or why it is not used for it.
pub fn config_for(w: u32, h: u32, rate: FrameRate, s: &ExportSettings) -> Result<VtConfig, String> { pub fn config_for(w: u32, h: u32, rate: FrameRate, s: &ExportSettings) -> Result<VtConfig, String> {
if s.hardware_encoding != HardwareEncoding::Auto { let hevc = s.format == Format::Hevc;
if !hevc && s.hardware_encoding != HardwareEncoding::Auto {
return Err("hardware encoding is off".into()); return Err("hardware encoding is off".into());
} }
if s.format.is_mxf() { if s.format.is_mxf() {
@ -45,10 +53,11 @@ pub fn config_for(w: u32, h: u32, rate: FrameRate, s: &ExportSettings) -> Result
timescale, timescale,
frame_duration, frame_duration,
keyframe_interval: s.keyframe_distance.filter(|k| *k > 0).unwrap_or_else(|| (fps * 2.0).round().max(1.0) as u32), keyframe_interval: s.keyframe_distance.filter(|k| *k > 0).unwrap_or_else(|| (fps * 2.0).round().max(1.0) as u32),
profile: match s.h264_profile { profile: match (hevc, s.h264_profile) {
H264Profile::Baseline => VtProfile::Baseline, (true, _) => VtProfile::HevcMain,
H264Profile::Main => VtProfile::Main, (false, H264Profile::Baseline) => VtProfile::Baseline,
H264Profile::High => VtProfile::High, (false, H264Profile::Main) => VtProfile::Main,
(false, H264Profile::High) => VtProfile::High,
}, },
rate: match s.bitrate_mode { rate: match s.bitrate_mode {
BitrateMode::Cbr => VtRate::Cbr { kbps }, BitrateMode::Cbr => VtRate::Cbr { kbps },
@ -57,8 +66,31 @@ pub fn config_for(w: u32, h: u32, rate: FrameRate, s: &ExportSettings) -> Result
}) })
} }
/// The encoder factory `register` puts in front of the built-in ones: a hardware H.264 encoder when /// Whether this machine has a hardware HEVC encoder (asked once, by creating a small session): what
/// the export asks for one and the OS takes the configuration, `None` (the built-in encoder) otherwise. /// makes `Format::Hevc` available.
pub fn hevc_available() -> bool {
static AVAILABLE: OnceLock<bool> = OnceLock::new();
*AVAILABLE.get_or_init(|| {
let probe = VtConfig {
width: 640,
height: 360,
timescale: 30,
frame_duration: 1,
keyframe_interval: 30,
profile: VtProfile::HevcMain,
rate: VtRate::Vbr { target_kbps: 2000, max_kbps: 3000 },
};
let result = VtEncoder::new(probe);
if let Err(why) = &result {
log::info!("no hardware HEVC encoder: {why}");
}
result.is_ok()
})
}
/// The encoder factory `register` puts in front of the built-in ones: a hardware encoder when the
/// export asks for one (H.264 with `hardware_encoding`; HEVC always) and the OS takes the
/// configuration, `None` (the built-in encoder, if there is one) otherwise.
pub fn videotoolbox_encoder_factory( pub fn videotoolbox_encoder_factory(
format: Format, format: Format,
w: u32, w: u32,
@ -66,27 +98,39 @@ pub fn videotoolbox_encoder_factory(
rate: FrameRate, rate: FrameRate,
s: &ExportSettings, s: &ExportSettings,
) -> Option<filmcraft_export::Result<Box<dyn VideoEncoder>>> { ) -> Option<filmcraft_export::Result<Box<dyn VideoEncoder>>> {
if format != Format::H264 || s.hardware_encoding != HardwareEncoding::Auto { let name = match format {
return None; Format::H264 if s.hardware_encoding == HardwareEncoding::Auto => "H.264",
} Format::Hevc => "HEVC",
_ => return None,
};
let config = match config_for(w, h, rate, s) { let config = match config_for(w, h, rate, s) {
Ok(c) => c, Ok(c) => c,
Err(why) => { Err(why) => {
log::info!("hardware H.264 encoding declined: {why}"); log::info!("hardware {name} encoding declined: {why}");
return None; return None;
} }
}; };
match VtEncoder::new(config) { match VtEncoder::new(config) {
Ok(vt) => Some(Ok(Box::new(HardwareH264Encoder { vt, y: Vec::new(), u: Vec::new(), v: Vec::new(), decoded: 0 }))), Ok(vt) => Some(Ok(Box::new(HardwareEncoder { vt, y: Vec::new(), u: Vec::new(), v: Vec::new(), decoded: 0 }))),
Err(why) => { Err(why) => {
log::info!("hardware H.264 encoding declined: {why}"); log::info!("hardware {name} encoding declined: {why}");
None None
} }
} }
} }
/// The `hvcC` configuration of an HEVC stream: VideoToolbox's own record when it wrote one.
fn hevc_config(p: &ParamSets) -> Result<HevcConfig, String> {
let record = p.hvcc.as_deref().ok_or("VideoToolbox wrote no hvcC record for the stream")?;
let cfg = HevcConfig::parse(record).map_err(|e| format!("unreadable hvcC record: {e}"))?;
if cfg.vps().is_empty() || cfg.sps().is_empty() || cfg.pps().is_empty() || cfg.length_size != 4 {
return Err("the hvcC record lacks a parameter set or does not use 4-byte NAL lengths".into());
}
Ok(cfg)
}
/// [`VtEncoder`] as a `VideoEncoder`: RGBA pictures in, MP4 samples out. /// [`VtEncoder`] as a `VideoEncoder`: RGBA pictures in, MP4 samples out.
pub struct HardwareH264Encoder { pub struct HardwareEncoder {
vt: VtEncoder, vt: VtEncoder,
y: Vec<u8>, y: Vec<u8>,
u: Vec<u8>, u: Vec<u8>,
@ -95,7 +139,7 @@ pub struct HardwareH264Encoder {
decoded: i64, decoded: i64,
} }
impl HardwareH264Encoder { impl HardwareEncoder {
/// MP4 samples from compressed frames. Without frame reordering every frame is presented at /// MP4 samples from compressed frames. Without frame reordering every frame is presented at
/// its decode time, so composition offsets are 0; a frame whose time is not `n × duration` /// its decode time, so composition offsets are 0; a frame whose time is not `n × duration`
/// after the first (a dropped or reordered frame) is an error, never a file with wrong times. /// after the first (a dropped or reordered frame) is an error, never a file with wrong times.
@ -119,11 +163,18 @@ impl HardwareH264Encoder {
} }
} }
impl VideoEncoder for HardwareH264Encoder { impl VideoEncoder for HardwareEncoder {
fn sample_entry(&self) -> SampleEntry { fn sample_entry(&self) -> SampleEntry {
let c = self.vt.config(); let c = self.vt.config();
let (sps, pps) = self.vt.parameter_sets().map(|p| (p.sps, p.pps)).unwrap_or_default(); let (w, h) = (c.width.min(u32::from(u16::MAX)) as u16, c.height.min(u32::from(u16::MAX)) as u16);
SampleEntry::avc(AvcConfig::new(sps, pps, 4), c.width.min(u32::from(u16::MAX)) as u16, c.height.min(u32::from(u16::MAX)) as u16) match (c.profile.codec(), self.vt.parameter_sets()) {
(VtCodec::H264, p) => {
let (sps, pps) = p.map(|p| (p.sps, p.pps)).unwrap_or_default();
SampleEntry::avc(AvcConfig::new(sps, pps, 4), w, h)
}
// `encode` refuses to go on without a usable record, so a stream that got this far has one
(VtCodec::Hevc, p) => SampleEntry::hevc(p.as_ref().and_then(|p| hevc_config(p).ok()).unwrap_or_default(), w, h),
}
} }
fn timescale(&self) -> u32 { fn timescale(&self) -> u32 {
@ -132,8 +183,9 @@ impl VideoEncoder for HardwareH264Encoder {
fn encode(&mut self, f: &EncoderFrame) -> filmcraft_export::Result<Vec<EncodedPacket>> { fn encode(&mut self, f: &EncoderFrame) -> filmcraft_export::Result<Vec<EncodedPacket>> {
let c = self.vt.config(); let c = self.vt.config();
let codec = c.profile.codec();
if f.hdr.is_some() { if f.hdr.is_some() {
return Err(ExportError::Unsupported("hardware H.264 encoding does not take HDR pictures".into())); return Err(ExportError::Unsupported("hardware encoding does not take HDR pictures".into()));
} }
if (f.width, f.height) != (c.width, c.height) { if (f.width, f.height) != (c.width, c.height) {
return Err(ExportError::Encode(format!("picture is {}x{}, the encoder was created for {}x{}", f.width, f.height, c.width, c.height))); return Err(ExportError::Encode(format!("picture is {}x{}, the encoder was created for {}x{}", f.width, f.height, c.width, c.height)));
@ -145,8 +197,11 @@ impl VideoEncoder for HardwareH264Encoder {
filmcraft_export::rgba_to_yuv420_8(f.rgba, w, h, &mut self.y, &mut self.u, &mut self.v); filmcraft_export::rgba_to_yuv420_8(f.rgba, w, h, &mut self.y, &mut self.u, &mut self.v);
let packets = self.vt.encode(f.index, &self.y, &self.u, &self.v).map_err(ExportError::Encode)?; let packets = self.vt.encode(f.index, &self.y, &self.u, &self.v).map_err(ExportError::Encode)?;
// the muxer needs the parameter sets after the first group of pictures // the muxer needs the parameter sets after the first group of pictures
if f.index == 0 && self.vt.parameter_sets().is_none() { if f.index == 0 {
return Err(ExportError::Encode("VideoToolbox returned no parameter sets for the first frame".into())); let params = self.vt.parameter_sets().ok_or_else(|| ExportError::Encode("VideoToolbox returned no parameter sets for the first frame".into()))?;
if codec == VtCodec::Hevc {
hevc_config(&params).map_err(ExportError::Encode)?;
}
} }
self.samples(packets) self.samples(packets)
} }

View File

@ -54,6 +54,7 @@ pub fn register() -> Availability {
{ {
filmcraft_codecs::register_video_decoder(videotoolbox_factory); filmcraft_codecs::register_video_decoder(videotoolbox_factory);
filmcraft_export::register_encoder(hardware_encode::videotoolbox_encoder_factory); filmcraft_export::register_encoder(hardware_encode::videotoolbox_encoder_factory);
filmcraft_export::register_format_probe(filmcraft_export::Format::Hevc, hardware_encode::hevc_available);
Availability::Available("VideoToolbox") Availability::Available("VideoToolbox")
} }
#[cfg(not(target_os = "macos"))] #[cfg(not(target_os = "macos"))]

View File

@ -1,4 +1,4 @@
//! VideoToolbox (macOS) hardware H.264 encoding. //! VideoToolbox (macOS) hardware H.264 and HEVC encoding.
//! //!
//! An FFI module of this crate (docs/adr/0001-platform-ffi.md): every `unsafe` block has a //! An FFI module of this crate (docs/adr/0001-platform-ffi.md): every `unsafe` block has a
//! `// SAFETY:` comment, no panic may unwind into VideoToolbox (the output callback runs under //! `// SAFETY:` comment, no panic may unwind into VideoToolbox (the output callback runs under
@ -15,16 +15,21 @@
//! copies each compressed frame (length-prefixed NAL units) into a queue that [`VtEncoder::encode`] //! copies each compressed frame (length-prefixed NAL units) into a queue that [`VtEncoder::encode`]
//! and [`VtEncoder::flush`] drain. //! and [`VtEncoder::flush`] drain.
//! //!
//! The first frame is completed straight away: the SPS / PPS the container needs come with the //! The first frame is completed straight away: the parameter sets the container needs (SPS / PPS
//! first compressed frame, and the muxer asks for them after the first group of frames. //! for H.264, VPS / SPS / PPS and VideoToolbox's own `hvcC` record for HEVC) come with the first
//! compressed frame, and the muxer asks for them after the first group of frames.
use std::ffi::{c_int, c_void}; use std::ffi::{c_int, c_void};
use std::panic::AssertUnwindSafe; use std::panic::AssertUnwindSafe;
use std::ptr::NonNull; use std::ptr::NonNull;
use std::sync::{Mutex, PoisonError}; use std::sync::{Mutex, PoisonError};
use objc2_core_foundation::{CFArray, CFBoolean, CFDictionary, CFNumber, CFRetained, CFString, CFType}; use objc2_core_foundation::{CFArray, CFBoolean, CFData, CFDictionary, CFNumber, CFRetained, CFString, CFType};
use objc2_core_media::{CMSampleBuffer, CMTime, CMTimeFlags, CMVideoFormatDescriptionGetH264ParameterSetAtIndex, kCMVideoCodecType_H264}; use objc2_core_media::{
CMFormatDescription, CMSampleBuffer, CMTime, CMTimeFlags, CMVideoFormatDescriptionGetH264ParameterSetAtIndex,
CMVideoFormatDescriptionGetHEVCParameterSetAtIndex, kCMFormatDescriptionExtension_SampleDescriptionExtensionAtoms, kCMVideoCodecType_H264,
kCMVideoCodecType_HEVC,
};
use objc2_core_video::{ use objc2_core_video::{
CVImageBuffer, CVPixelBufferCreate, CVPixelBufferGetBaseAddressOfPlane, CVPixelBufferGetBytesPerRowOfPlane, CVPixelBufferGetHeightOfPlane, CVImageBuffer, CVPixelBufferCreate, CVPixelBufferGetBaseAddressOfPlane, CVPixelBufferGetBytesPerRowOfPlane, CVPixelBufferGetHeightOfPlane,
CVPixelBufferGetWidthOfPlane, CVPixelBufferLockBaseAddress, CVPixelBufferLockFlags, CVPixelBufferPool, CVPixelBufferUnlockBaseAddress, CVPixelBufferGetWidthOfPlane, CVPixelBufferLockBaseAddress, CVPixelBufferLockFlags, CVPixelBufferPool, CVPixelBufferUnlockBaseAddress,
@ -32,11 +37,12 @@ use objc2_core_video::{
kCVPixelBufferIOSurfacePropertiesKey, kCVPixelBufferPixelFormatTypeKey, kCVPixelBufferWidthKey, kCVPixelBufferIOSurfacePropertiesKey, kCVPixelBufferPixelFormatTypeKey, kCVPixelBufferWidthKey,
}; };
use objc2_video_toolbox::{ use objc2_video_toolbox::{
VTCompressionSession, VTEncodeInfoFlags, VTSessionSetProperty, kVTCompressionPropertyKey_AllowFrameReordering, kVTCompressionPropertyKey_AllowOpenGOP, VTCompressionSession, VTEncodeInfoFlags, VTSessionCopyProperty, VTSessionSetProperty, kVTCompressionPropertyKey_AllowFrameReordering,
kVTCompressionPropertyKey_AverageBitRate, kVTCompressionPropertyKey_ColorPrimaries, kVTCompressionPropertyKey_ConstantBitRate, kVTCompressionPropertyKey_AllowOpenGOP, kVTCompressionPropertyKey_AverageBitRate, kVTCompressionPropertyKey_ColorPrimaries,
kVTCompressionPropertyKey_DataRateLimits, kVTCompressionPropertyKey_ExpectedFrameRate, kVTCompressionPropertyKey_MaxKeyFrameInterval, kVTCompressionPropertyKey_ConstantBitRate, kVTCompressionPropertyKey_DataRateLimits, kVTCompressionPropertyKey_ExpectedFrameRate,
kVTCompressionPropertyKey_ProfileLevel, kVTCompressionPropertyKey_RealTime, kVTCompressionPropertyKey_TransferFunction, kVTCompressionPropertyKey_MaxKeyFrameInterval, kVTCompressionPropertyKey_ProfileLevel, kVTCompressionPropertyKey_RealTime,
kVTCompressionPropertyKey_YCbCrMatrix, kVTProfileLevel_H264_Baseline_AutoLevel, kVTProfileLevel_H264_High_AutoLevel, kVTProfileLevel_H264_Main_AutoLevel, kVTCompressionPropertyKey_TransferFunction, kVTCompressionPropertyKey_UsingHardwareAcceleratedVideoEncoder, kVTCompressionPropertyKey_YCbCrMatrix,
kVTProfileLevel_H264_Baseline_AutoLevel, kVTProfileLevel_H264_High_AutoLevel, kVTProfileLevel_H264_Main_AutoLevel, kVTProfileLevel_HEVC_Main_AutoLevel,
kVTVideoEncoderSpecification_RequireHardwareAcceleratedVideoEncoder, kVTVideoEncoderSpecification_RequireHardwareAcceleratedVideoEncoder,
}; };
@ -47,12 +53,30 @@ const MAX_FRAME_BYTES: usize = 256 << 20;
/// Largest picture the hardware encoder is asked for. /// Largest picture the hardware encoder is asked for.
const MAX_SIDE: u32 = 8192; const MAX_SIDE: u32 = 8192;
/// H.264 profiles VideoToolbox offers. /// The codec of a session.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum VtCodec {
H264,
Hevc,
}
/// H.264 and HEVC profiles VideoToolbox offers (the profile also picks the codec).
#[derive(Clone, Copy, Debug, PartialEq, Eq)] #[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum VtProfile { pub enum VtProfile {
Baseline, Baseline,
Main, Main,
High, High,
/// HEVC Main: 8-bit 4:2:0.
HevcMain,
}
impl VtProfile {
pub fn codec(self) -> VtCodec {
match self {
VtProfile::Baseline | VtProfile::Main | VtProfile::High => VtCodec::H264,
VtProfile::HevcMain => VtCodec::Hevc,
}
}
} }
/// How the bitrate is controlled (kilobits per second). /// How the bitrate is controlled (kilobits per second).
@ -75,6 +99,7 @@ pub struct VtConfig {
pub frame_duration: u32, pub frame_duration: u32,
/// Frames between keyframes (every keyframe is an IDR picture). /// Frames between keyframes (every keyframe is an IDR picture).
pub keyframe_interval: u32, pub keyframe_interval: u32,
/// The profile, which also says whether this is an H.264 or an HEVC session.
pub profile: VtProfile, pub profile: VtProfile,
pub rate: VtRate, pub rate: VtRate,
} }
@ -82,18 +107,22 @@ pub struct VtConfig {
/// One compressed frame in decoding order, which is presentation order (times in the config's timescale). /// One compressed frame in decoding order, which is presentation order (times in the config's timescale).
#[derive(Clone, Debug)] #[derive(Clone, Debug)]
pub struct VtPacket { pub struct VtPacket {
/// Length-prefixed NAL units (4-byte lengths, as in an `avcC` MP4 sample). /// Length-prefixed NAL units (4-byte lengths, as in an `avcC` / `hvcC` MP4 sample).
pub data: Vec<u8>, pub data: Vec<u8>,
pub key: bool, pub key: bool,
pub pts: i64, pub pts: i64,
pub dts: i64, pub dts: i64,
} }
/// The parameter sets of the stream (NAL units without length prefix), for the `avcC` box. /// The parameter sets of the stream (NAL units without length prefix), for the `avcC` / `hvcC` box.
#[derive(Clone, Debug, PartialEq, Eq)] #[derive(Clone, Debug, PartialEq, Eq)]
pub struct ParamSets { pub struct ParamSets {
/// HEVC only.
pub vps: Vec<Vec<u8>>,
pub sps: Vec<Vec<u8>>, pub sps: Vec<Vec<u8>>,
pub pps: Vec<Vec<u8>>, pub pps: Vec<Vec<u8>>,
/// HEVC only: the `hvcC` record (HEVCDecoderConfigurationRecord) VideoToolbox wrote for the stream.
pub hvcc: Option<Vec<u8>>,
} }
/// One output-callback result. /// One output-callback result.
@ -105,6 +134,7 @@ enum Output {
/// State shared with the output callback (its `outputCallbackRefCon`). Boxed by the encoder and /// State shared with the output callback (its `outputCallbackRefCon`). Boxed by the encoder and
/// kept alive until the session is invalidated. /// kept alive until the session is invalidated.
struct Shared { struct Shared {
codec: VtCodec,
timescale: i64, timescale: i64,
out: Mutex<Vec<Output>>, out: Mutex<Vec<Output>>,
params: Mutex<Option<ParamSets>>, params: Mutex<Option<ParamSets>>,
@ -160,37 +190,60 @@ fn ticks(t: CMTime, timescale: i64) -> Option<i64> {
i64::try_from((2 * num + den).div_euclid(2 * den)).ok() i64::try_from((2 * num + den).div_euclid(2 * den)).ok()
} }
/// Whether length-prefixed NAL units hold an IDR slice (nal_unit_type 5). /// Whether length-prefixed NAL units hold a random access point: an IDR slice (nal_unit_type 5) in
fn has_idr(data: &[u8]) -> Result<bool, String> { /// H.264; a BLA, IDR or CRA slice (types 16 to 21) in HEVC.
fn is_sync(data: &[u8], codec: VtCodec) -> Result<bool, String> {
let mut pos = 0usize; let mut pos = 0usize;
let mut idr = false; let mut key = false;
while pos < data.len() { while pos < data.len() {
let len_bytes = data.get(pos..pos + 4).ok_or("truncated NAL length")?; let len_bytes = data.get(pos..pos + 4).ok_or("truncated NAL length")?;
let len = u32::from_be_bytes([len_bytes[0], len_bytes[1], len_bytes[2], len_bytes[3]]) as usize; let len = u32::from_be_bytes([len_bytes[0], len_bytes[1], len_bytes[2], len_bytes[3]]) as usize;
pos += 4; pos += 4;
let end = pos.checked_add(len).filter(|e| *e <= data.len()).ok_or("NAL unit longer than the frame")?; let end = pos.checked_add(len).filter(|e| *e <= data.len()).ok_or("NAL unit longer than the frame")?;
let header = *data.get(pos).ok_or("empty NAL unit")?; let header = *data.get(pos).ok_or("empty NAL unit")?;
if header & 0x1f == 5 { key |= match codec {
idr = true; VtCodec::H264 => header & 0x1f == 5,
} VtCodec::Hevc => (16..=21).contains(&((header >> 1) & 0x3f)),
};
pos = end; pos = end;
} }
Ok(idr) Ok(key)
} }
/// SPS / PPS of the first compressed frame's format description. /// The `hvcC` record VideoToolbox wrote among the sample description extension atoms of an HEVC
fn read_param_sets(sample: &CMSampleBuffer) -> Result<ParamSets, String> { /// format description (the container's own codec configuration, exactly as QuickTime writes it).
fn hvcc_record(format: &CMFormatDescription) -> Option<Vec<u8>> {
// SAFETY: `format` is valid; the key is an immutable framework constant; the result is a
// retained property list (or none).
let atoms = unsafe { format.extension(kCMFormatDescriptionExtension_SampleDescriptionExtensionAtoms) }?;
let atoms = atoms.downcast::<CFDictionary>().ok()?;
// SAFETY: CoreMedia documents this extension as a dictionary from four-character-code strings
// to property lists; every CoreFoundation object is a `CFType`.
let atoms: &CFDictionary<CFString, CFType> = unsafe { atoms.cast_unchecked() };
let data = atoms.get(&CFString::from_str("hvcC"))?.downcast::<CFData>().ok()?;
let bytes = data.to_vec();
(!bytes.is_empty() && bytes.len() <= 65_535).then_some(bytes)
}
/// The parameter sets of the first compressed frame's format description: SPS / PPS for H.264,
/// VPS / SPS / PPS and the `hvcC` record for HEVC.
fn read_param_sets(sample: &CMSampleBuffer, codec: VtCodec) -> Result<ParamSets, String> {
// SAFETY: `sample` is a valid sample buffer for the duration of the callback. // SAFETY: `sample` is a valid sample buffer for the duration of the callback.
let format = unsafe { sample.format_description() }.ok_or("compressed frame without a format description")?; let format = unsafe { sample.format_description() }.ok_or("compressed frame without a format description")?;
let (mut sps, mut pps) = (Vec::new(), Vec::new()); let (mut vps, mut sps, mut pps) = (Vec::new(), Vec::new(), Vec::new());
let (mut index, mut count) = (0usize, 1usize); let (mut index, mut count) = (0usize, 1usize);
while index < count && index < 16 { while index < count && index < 16 {
let (mut ptr, mut size, mut total, mut nal_len): (*const u8, usize, usize, c_int) = (std::ptr::null(), 0, 0, 0); let (mut ptr, mut size, mut total, mut nal_len): (*const u8, usize, usize, c_int) = (std::ptr::null(), 0, 0, 0);
// SAFETY: the out-pointers are valid for the call; the returned pointer is only read // SAFETY: the out-pointers are valid for the call; the returned pointer is only read
// while `format` (retained above) is alive. // while `format` (retained above) is alive.
let status = unsafe { CMVideoFormatDescriptionGetH264ParameterSetAtIndex(&format, index, &mut ptr, &mut size, &mut total, &mut nal_len) }; let status = unsafe {
match codec {
VtCodec::H264 => CMVideoFormatDescriptionGetH264ParameterSetAtIndex(&format, index, &mut ptr, &mut size, &mut total, &mut nal_len),
VtCodec::Hevc => CMVideoFormatDescriptionGetHEVCParameterSetAtIndex(&format, index, &mut ptr, &mut size, &mut total, &mut nal_len),
}
};
if status != 0 { if status != 0 {
return Err(format!("cannot read the H.264 parameter sets ({status})")); return Err(format!("cannot read the {codec:?} parameter sets ({status})"));
} }
if nal_len != 4 { if nal_len != 4 {
return Err(format!("unexpected NAL length size {nal_len}")); return Err(format!("unexpected NAL length size {nal_len}"));
@ -201,17 +254,28 @@ fn read_param_sets(sample: &CMSampleBuffer) -> Result<ParamSets, String> {
} }
// SAFETY: VideoToolbox returned `size` readable bytes at `ptr`, owned by `format`. // SAFETY: VideoToolbox returned `size` readable bytes at `ptr`, owned by `format`.
let bytes = unsafe { std::slice::from_raw_parts(ptr, size) }.to_vec(); let bytes = unsafe { std::slice::from_raw_parts(ptr, size) }.to_vec();
match bytes.first().map(|b| b & 0x1f) { match (codec, bytes.first()) {
Some(7) => sps.push(bytes), (VtCodec::H264, Some(b)) => match b & 0x1f {
Some(8) => pps.push(bytes), 7 => sps.push(bytes),
8 => pps.push(bytes),
_ => {}
},
(VtCodec::Hevc, Some(b)) => match (b >> 1) & 0x3f {
32 => vps.push(bytes),
33 => sps.push(bytes),
34 => pps.push(bytes),
_ => {}
},
_ => {} _ => {}
} }
index += 1; index += 1;
} }
if sps.is_empty() || pps.is_empty() { let complete = !sps.is_empty() && !pps.is_empty() && (codec == VtCodec::H264 || !vps.is_empty());
return Err("the format description has no SPS / PPS".into()); if !complete {
return Err("the format description lacks parameter sets".into());
} }
Ok(ParamSets { sps, pps }) let hvcc = if codec == VtCodec::Hevc { hvcc_record(&format) } else { None };
Ok(ParamSets { vps, sps, pps, hvcc })
} }
/// One compressed frame: its bytes, key flag and timestamps (and, once, the parameter sets). /// One compressed frame: its bytes, key flag and timestamps (and, once, the parameter sets).
@ -234,11 +298,11 @@ fn read_packet(sample: &CMSampleBuffer, shared: &Shared) -> Result<VtPacket, Str
let (pts, dts) = unsafe { (sample.presentation_time_stamp(), sample.decode_time_stamp()) }; let (pts, dts) = unsafe { (sample.presentation_time_stamp(), sample.decode_time_stamp()) };
let pts = ticks(pts, shared.timescale).ok_or("compressed frame without a presentation time")?; let pts = ticks(pts, shared.timescale).ok_or("compressed frame without a presentation time")?;
let dts = ticks(dts, shared.timescale).unwrap_or(pts); let dts = ticks(dts, shared.timescale).unwrap_or(pts);
let key = has_idr(&data)?; let key = is_sync(&data, shared.codec)?;
{ {
let mut params = shared.params.lock().unwrap_or_else(PoisonError::into_inner); let mut params = shared.params.lock().unwrap_or_else(PoisonError::into_inner);
if params.is_none() { if params.is_none() {
*params = Some(read_param_sets(sample)?); *params = Some(read_param_sets(sample, shared.codec)?);
} }
} }
Ok(VtPacket { data, key, pts, dts }) Ok(VtPacket { data, key, pts, dts })
@ -336,8 +400,8 @@ fn fill(pb: &CVImageBuffer, w: usize, h: usize, y: &[u8], u: &[u8], v: &[u8]) ->
Ok(()) Ok(())
} }
/// A hardware H.264 encoder: feed pictures, collect compressed frames (in presentation order, which /// A hardware H.264 or HEVC encoder: feed pictures, collect compressed frames (in presentation
/// is decoding order since nothing is reordered). /// order, which is decoding order since nothing is reordered).
pub struct VtEncoder { pub struct VtEncoder {
session: Session, session: Session,
config: VtConfig, config: VtConfig,
@ -375,13 +439,14 @@ impl VtEncoder {
return Err(format!("picture size {w}x{h}")); return Err(format!("picture size {w}x{h}"));
} }
if !w.is_multiple_of(2) || !h.is_multiple_of(2) { if !w.is_multiple_of(2) || !h.is_multiple_of(2) {
// 4:2:0 H.264 crops in units of two samples: an odd size would come out one sample smaller // 4:2:0 video crops in units of two samples: an odd size would come out one sample smaller
return Err(format!("odd picture size {w}x{h}")); return Err(format!("odd picture size {w}x{h}"));
} }
if config.timescale == 0 || config.frame_duration == 0 || config.timescale > i32::MAX as u32 { if config.timescale == 0 || config.frame_duration == 0 || config.timescale > i32::MAX as u32 {
return Err("invalid frame rate".into()); return Err("invalid frame rate".into());
} }
let shared = Box::new(Shared { timescale: i64::from(config.timescale), out: Mutex::new(Vec::new()), params: Mutex::new(None) }); let codec = config.profile.codec();
let shared = Box::new(Shared { codec, timescale: i64::from(config.timescale), out: Mutex::new(Vec::new()), params: Mutex::new(None) });
// SAFETY: reading immutable framework constants. // SAFETY: reading immutable framework constants.
let require_key: &CFString = unsafe { kVTVideoEncoderSpecification_RequireHardwareAcceleratedVideoEncoder }; let require_key: &CFString = unsafe { kVTVideoEncoderSpecification_RequireHardwareAcceleratedVideoEncoder };
@ -403,7 +468,10 @@ impl VtEncoder {
None, None,
w as i32, w as i32,
h as i32, h as i32,
kCMVideoCodecType_H264, match codec {
VtCodec::H264 => kCMVideoCodecType_H264,
VtCodec::Hevc => kCMVideoCodecType_HEVC,
},
Some(spec.as_ref()), Some(spec.as_ref()),
Some(attrs.as_ref()), Some(attrs.as_ref()),
None, None,
@ -433,6 +501,7 @@ impl VtEncoder {
VtProfile::Baseline => kVTProfileLevel_H264_Baseline_AutoLevel, VtProfile::Baseline => kVTProfileLevel_H264_Baseline_AutoLevel,
VtProfile::Main => kVTProfileLevel_H264_Main_AutoLevel, VtProfile::Main => kVTProfileLevel_H264_Main_AutoLevel,
VtProfile::High => kVTProfileLevel_H264_High_AutoLevel, VtProfile::High => kVTProfileLevel_H264_High_AutoLevel,
VtProfile::HevcMain => kVTProfileLevel_HEVC_Main_AutoLevel,
}, },
) )
}; };
@ -497,12 +566,28 @@ impl VtEncoder {
Ok(()) Ok(())
} }
/// Whether VideoToolbox says this session runs on a hardware encoder. The session is created
/// with one required, so it should always hold; asking (and testing it) means a requirement
/// dropped by accident could never turn into a silent software encode.
pub fn uses_hardware(&self) -> bool {
// SAFETY: reading an immutable framework constant.
let key: &CFString = unsafe { kVTCompressionPropertyKey_UsingHardwareAcceleratedVideoEncoder };
let mut value: *mut CFType = std::ptr::null_mut();
// SAFETY: the session is valid and `value` is a valid out-pointer for the CFTypeRef the
// property is copied to (returned retained, +1); it stays null when the call fails.
let status = unsafe { VTSessionCopyProperty(self.session.session.as_ref(), key, None, (&raw mut value).cast()) };
let Some(value) = NonNull::new(value).filter(|_| status == 0) else { return false };
// SAFETY: a copied property is returned retained (+1); `CFRetained` takes that reference over.
let value = unsafe { CFRetained::from_raw(value) };
value.downcast_ref::<CFBoolean>().is_some_and(CFBoolean::as_bool)
}
/// The configuration this encoder was created with. /// The configuration this encoder was created with.
pub fn config(&self) -> &VtConfig { pub fn config(&self) -> &VtConfig {
&self.config &self.config
} }
/// The stream's SPS / PPS, known once the first compressed frame came out. /// The stream's parameter sets, known once the first compressed frame came out.
pub fn parameter_sets(&self) -> Option<ParamSets> { pub fn parameter_sets(&self) -> Option<ParamSets> {
self.session.shared.params.lock().unwrap_or_else(PoisonError::into_inner).clone() self.session.shared.params.lock().unwrap_or_else(PoisonError::into_inner).clone()
} }
@ -544,7 +629,7 @@ impl VtEncoder {
} }
self.submitted += 1; self.submitted += 1;
if self.submitted == 1 { if self.submitted == 1 {
// the first compressed frame carries the SPS / PPS the container needs // the first compressed frame carries the parameter sets the container needs
// SAFETY: the session is valid. // SAFETY: the session is valid.
require(unsafe { self.session.session.complete_frames(pts) }, "to finish the first frame")?; require(unsafe { self.session.session.complete_frames(pts) }, "to finish the first frame")?;
} }

View File

@ -1,4 +1,4 @@
//! VideoToolbox H.264 encoding (macOS): what the hardware path takes and declines, a round trip //! VideoToolbox H.264 and HEVC encoding (macOS): what the hardware path takes and declines, a round trip
//! through our own software decoder (picture quality, frame count and order, keyframes, the //! through our own software decoder (picture quality, frame count and order, keyframes, the
//! timestamps the muxer needs), a full export through `filmcraft_export` that decodes, the fallback //! timestamps the muxer needs), a full export through `filmcraft_export` that decodes, the fallback
//! to the built-in encoder for everything hardware does not take, and hostile configurations that //! to the built-in encoder for everything hardware does not take, and hostile configurations that
@ -15,7 +15,7 @@ use filmcraft_frame::PixelData;
use filmcraft_isobmff::{CodecConfig, SampleEntry}; use filmcraft_isobmff::{CodecConfig, SampleEntry};
use filmcraft_media::generators::GeneratorSource; use filmcraft_media::generators::GeneratorSource;
use filmcraft_media::{DemoScene, MediaSource}; use filmcraft_media::{DemoScene, MediaSource};
use filmcraft_platform::hardware_encode::{config_for, videotoolbox_encoder_factory}; use filmcraft_platform::hardware_encode::{config_for, hevc_available, videotoolbox_encoder_factory};
use filmcraft_platform::videotoolbox_encode::{VtConfig, VtEncoder, VtProfile, VtRate}; use filmcraft_platform::videotoolbox_encode::{VtConfig, VtEncoder, VtProfile, VtRate};
use filmcraft_project::{ItemKind, Label, MediaClip, MediaRef, Project, SequenceSettings, TrackKind}; use filmcraft_project::{ItemKind, Label, MediaClip, MediaRef, Project, SequenceSettings, TrackKind};
use filmcraft_render::SourceMap; use filmcraft_render::SourceMap;
@ -46,13 +46,18 @@ fn hardware_settings() -> ExportSettings {
ExportSettings { format: Format::H264, hardware_encoding: HardwareEncoding::Auto, bitrate_kbps: 8_000, keyframe_distance: Some(30), ..Default::default() } ExportSettings { format: Format::H264, hardware_encoding: HardwareEncoding::Auto, bitrate_kbps: 8_000, keyframe_distance: Some(30), ..Default::default() }
} }
/// H.265 needs no opt-in: choosing the format is the opt-in.
fn hevc_settings() -> ExportSettings {
ExportSettings { format: Format::Hevc, bitrate_kbps: 6_000, keyframe_distance: Some(30), ..Default::default() }
}
/// The hardware encoder for the settings, or `None` (skip) when this machine has none. /// The hardware encoder for the settings, or `None` (skip) when this machine has none.
fn hardware(s: &ExportSettings, w: u32, h: u32) -> Option<Box<dyn VideoEncoder>> { fn hardware(s: &ExportSettings, w: u32, h: u32) -> Option<Box<dyn VideoEncoder>> {
match videotoolbox_encoder_factory(Format::H264, w, h, FPS, s) { match videotoolbox_encoder_factory(s.format, w, h, FPS, s) {
Some(Ok(e)) => Some(e), Some(Ok(e)) => Some(e),
Some(Err(e)) => panic!("factory error: {e}"), Some(Err(e)) => panic!("factory error: {e}"),
None => { None => {
eprintln!("SKIPPED: no VideoToolbox hardware H.264 encoder for these settings"); eprintln!("SKIPPED: no VideoToolbox hardware {} encoder for these settings", s.format.label());
None None
} }
} }
@ -208,7 +213,7 @@ fn inspect(path: &str) -> (usize, usize, usize) {
let vt = file.track_of_kind(filmcraft_isobmff::TrackKind::Video).unwrap(); let vt = file.track_of_kind(filmcraft_isobmff::TrackKind::Video).unwrap();
let track = &file.tracks[vt]; let track = &file.tracks[vt];
let entry = track.entries.first().expect("a sample entry"); let entry = track.entries.first().expect("a sample entry");
let CodecConfig::Avc(_) = &entry.codec else { panic!("not AVC") }; assert!(matches!(&entry.codec, CodecConfig::Avc(_) | CodecConfig::Hevc(_)), "neither AVC nor HEVC: {}", entry.codec.name());
let mut dec = filmcraft_codecs::software_video_decoder(entry).expect("software decoder"); let mut dec = filmcraft_codecs::software_video_decoder(entry).expect("software decoder");
let mut pictures = 0; let mut pictures = 0;
for (i, s) in track.samples.iter().enumerate() { for (i, s) in track.samples.iter().enumerate() {
@ -431,3 +436,280 @@ fn an_empty_range_never_leaves_a_broken_file() {
} }
} }
} }
// ---------------------------------------------------------------------------------------------
// HEVC (H.265)
// ---------------------------------------------------------------------------------------------
#[test]
fn hevc_declines_what_the_hardware_path_does_not_take() {
let ok = hevc_settings();
let config = config_for(W, H, FPS, &ok).expect("HEVC needs no hardware flag");
assert_eq!((config.profile, config.keyframe_interval, config.rate), (VtProfile::HevcMain, 30, VtRate::Vbr { target_kbps: 6000, max_kbps: 9000 }));
let declined = |s: &ExportSettings, what: &str| assert!(config_for(W, H, FPS, s).is_err(), "{what} must not use the hardware encoder");
declined(&ExportSettings { bitrate_mode: BitrateMode::Vbr2Pass, ..ok.clone() }, "two-pass VBR");
declined(&ExportSettings { signal: filmcraft_export::ColorSignal::PQ, ..ok.clone() }, "HDR");
declined(&ExportSettings { pixel_aspect: Some((4, 3)), ..ok.clone() }, "non-square pixels");
assert!(config_for(321, 181, FPS, &ok).is_err() && config_for(640, 361, FPS, &ok).is_err(), "odd sizes");
assert!(config_for(W, H, FrameRate { num: 0, den: 1 }, &ok).is_err());
// constant bitrate, and the H.264 profile setting does not matter
let c = config_for(W, H, FPS, &ExportSettings { bitrate_mode: BitrateMode::Cbr, bitrate_kbps: 4000, h264_profile: H264Profile::Baseline, ..ok }).unwrap();
assert_eq!((c.profile, c.rate), (VtProfile::HevcMain, VtRate::Cbr { kbps: 4000 }));
}
#[test]
fn hevc_format_follows_the_machine() {
filmcraft_platform::register();
// the format list asks `available`: it must agree with the probe
assert_eq!(filmcraft_export::available(Format::Hevc), hevc_available());
let s = hevc_settings();
// no hardware flag needed, but nothing else is taken by this factory
assert_eq!(videotoolbox_encoder_factory(Format::Hevc, W, H, FPS, &s).is_some(), hevc_available());
assert!(videotoolbox_encoder_factory(Format::Hevc, 321, 181, FPS, &s).is_none());
assert!(videotoolbox_encoder_factory(Format::H264, W, H, FPS, &s).is_none(), "H.264 still needs the hardware flag");
}
#[test]
fn hevc_round_trip_through_our_software_decoder() {
let s = hevc_settings();
let Some(mut enc) = hardware(&s, W, H) else { return };
let n = 75u64;
let packets = encode_all(&mut enc, n);
let entry = enc.sample_entry();
assert_eq!(&entry.format.0, b"hvc1");
let CodecConfig::Hevc(hevc) = &entry.codec else { panic!("expected an HEVC sample entry, got {}", entry.codec.name()) };
assert!(!hevc.vps().is_empty() && !hevc.sps().is_empty() && !hevc.pps().is_empty());
assert_eq!(hevc.length_size, 4);
// Main profile, 8-bit 4:2:0
assert_eq!((hevc.general_profile_idc, hevc.chroma_format_idc, hevc.bit_depth_luma, hevc.bit_depth_chroma), (1, 1, 8, 8));
assert!(hevc.general_level_idc >= 90, "level {} cannot hold 640x360", hevc.general_level_idc);
assert_eq!(enc.timescale(), 30);
// one compressed frame per picture, the first a keyframe, keyframes no further apart than asked
assert_eq!(packets.len() as u64, n);
assert!(packets[0].key);
assert!(packets.iter().all(|p| p.duration == 1 && p.composition_offset == 0 && !p.data.is_empty()));
let keys: Vec<usize> = packets.iter().enumerate().filter(|(_, p)| p.key).map(|(i, _)| i).collect();
assert!(keys.windows(2).all(|w| w[1] - w[0] <= 30), "keyframes at {keys:?}");
assert_eq!(enc.media_start(), None);
// our decoder reproduces every picture, in order, at a good quality
let decoded = decode(&entry, &packets);
assert_eq!(decoded.len() as u64, n);
assert_eq!(decoded.iter().map(|(p, _)| *p).collect::<Vec<_>>(), (0..n as i64).collect::<Vec<_>>());
let (mut y, mut u, mut v) = (Vec::new(), Vec::new(), Vec::new());
let mut worst = 99.0f64;
for (i, (_, luma)) in decoded.iter().enumerate() {
rgba_to_yuv420_8(&picture(i as u64), W as usize, H as usize, &mut y, &mut u, &mut v);
worst = worst.min(psnr(&y, luma));
}
assert!(worst > 30.0, "worst luma PSNR {worst:.1} dB");
eprintln!("hardware HEVC: {} bytes for {n} frames, worst luma PSNR {worst:.1} dB", packets.iter().map(|p| p.data.len()).sum::<usize>());
}
#[test]
fn hevc_export_through_the_pipeline() {
filmcraft_platform::register();
if !hevc_available() {
eprintln!("SKIPPED: no VideoToolbox hardware HEVC encoder");
return;
}
let frames = 70;
let (p, seq, m) = project(frames);
// MP4 (the default container) and QuickTime
for (name, multiplexer) in [("hevc.mp4", filmcraft_export::Multiplexer::Mp4), ("hevc.mov", filmcraft_export::Multiplexer::Mov)] {
let path = tmp(name);
let s = ExportSettings { path: path.clone(), include_audio: false, multiplexer, ..hevc_settings() };
let report = export(&p, seq, &s, &m, &Progress::default()).unwrap_or_else(|e| panic!("{name}: {e}"));
assert_eq!(report.frames, frames as u64, "{name}");
let (samples, pictures, syncs) = inspect(&path);
assert_eq!((samples, pictures), (frames as usize, frames as usize), "{name}");
assert!(syncs >= 3, "{name}: keyframes every 30 frames: {syncs} sync samples");
}
}
#[test]
fn hevc_export_with_audio_writes_aac_in_mp4() {
filmcraft_platform::register();
if !hevc_available() {
eprintln!("SKIPPED: no VideoToolbox hardware HEVC encoder");
return;
}
let (p, seq, m) = project(60);
let path = tmp("hevc-audio.mp4");
let s = ExportSettings { path: path.clone(), include_audio: true, ..hevc_settings() };
export(&p, seq, &s, &m, &Progress::default()).expect("export");
let bytes = std::fs::read(&path).unwrap();
let file = filmcraft_isobmff::open(bytes.as_slice()).unwrap();
assert!(file.track_of_kind(filmcraft_isobmff::TrackKind::Video).is_some());
let at = file.track_of_kind(filmcraft_isobmff::TrackKind::Audio).expect("an audio track");
assert!(matches!(&file.tracks[at].entries[0].codec, CodecConfig::Aac(_)), "audio is {}", file.tracks[at].entries[0].codec.name());
}
#[test]
fn hevc_has_no_two_pass_mode() {
filmcraft_platform::register();
let (p, seq, m) = project(10);
let s = ExportSettings { path: tmp("hevc-2pass.mp4"), include_audio: false, bitrate_mode: BitrateMode::Vbr2Pass, ..hevc_settings() };
let err = export(&p, seq, &s, &m, &Progress::default()).expect_err("two-pass HEVC must be refused").to_string();
assert!(err.contains("two-pass"), "{err}");
}
#[test]
fn hevc_hostile_configurations_give_errors_not_crashes() {
let base = VtConfig {
width: W,
height: H,
timescale: 30,
frame_duration: 1,
keyframe_interval: 30,
profile: VtProfile::HevcMain,
rate: VtRate::Vbr { target_kbps: 6000, max_kbps: 9000 },
};
let variants: Vec<(&str, VtConfig)> = vec![
("zero width", VtConfig { width: 0, ..base.clone() }),
("zero height", VtConfig { height: 0, ..base.clone() }),
("huge width", VtConfig { width: u32::MAX, ..base.clone() }),
("1x1", VtConfig { width: 1, height: 1, ..base.clone() }),
("2x2", VtConfig { width: 2, height: 2, ..base.clone() }),
("odd size", VtConfig { width: 321, height: 181, ..base.clone() }),
("zero timescale", VtConfig { timescale: 0, ..base.clone() }),
("zero frame duration", VtConfig { frame_duration: 0, ..base.clone() }),
("huge timescale", VtConfig { timescale: u32::MAX, ..base.clone() }),
("zero keyframe interval", VtConfig { keyframe_interval: 0, ..base.clone() }),
("huge keyframe interval", VtConfig { keyframe_interval: u32::MAX, ..base.clone() }),
("zero bitrate", VtConfig { rate: VtRate::Vbr { target_kbps: 0, max_kbps: 0 }, ..base.clone() }),
("huge bitrate", VtConfig { rate: VtRate::Vbr { target_kbps: u32::MAX, max_kbps: u32::MAX }, ..base.clone() }),
("max below target", VtConfig { rate: VtRate::Vbr { target_kbps: 6000, max_kbps: 1 }, ..base.clone() }),
("zero cbr", VtConfig { rate: VtRate::Cbr { kbps: 0 }, ..base.clone() }),
("cbr", VtConfig { rate: VtRate::Cbr { kbps: 4000 }, ..base.clone() }),
];
for (name, config) in variants {
match calm(name, || VtEncoder::new(config)) {
Ok(mut enc) => {
let (w, h) = (enc.config().width as usize, enc.config().height as usize);
let (cw, ch) = (w.div_ceil(2), h.div_ceil(2));
assert!(calm(name, || enc.encode(0, &[], &[], &[])).is_err(), "{name}: empty planes");
assert!(calm(name, || enc.encode(0, &vec![0; w * h - 1], &vec![128; cw * ch], &vec![128; cw * ch])).is_err(), "{name}: short luma");
let _ = calm(name, || enc.encode(0, &vec![16; w * h], &vec![128; cw * ch], &vec![128; cw * ch]));
let _ = calm(name, || enc.flush());
}
Err(why) => eprintln!("{name}: declined ({why})"),
}
}
}
#[test]
fn hevc_encoders_can_be_dropped_at_any_point() {
let s = hevc_settings();
for frames in [0u64, 1, 2, 5] {
let Some(mut enc) = hardware(&s, W, H) else { return };
for i in 0..frames {
let rgba = picture(i);
let _ = calm("encode", || enc.encode(&EncoderFrame { width: W, height: H, rgba: &rgba, hdr: None, index: i }));
}
drop(enc); // pending frames, no flush: must not crash or hang
}
}
#[test]
fn hevc_pictures_keep_their_exact_size() {
let s = hevc_settings();
// even sizes that are not multiples of the coding block size come out at exactly their size
for (w, h) in [(640u32, 362u32), (642, 360), (1280, 720), (1920, 1080), (64, 64)] {
let Some(mut enc) = hardware(&s, w, h) else { continue };
let rgba: Vec<u8> = (0..w * h * 4).map(|i| (i % 251) as u8).collect();
let mut packets = Vec::new();
for i in 0..6u64 {
packets.extend(enc.encode(&EncoderFrame { width: w, height: h, rgba: &rgba, hdr: None, index: i }).expect("encode"));
}
packets.extend(enc.flush().expect("flush"));
assert_eq!(packets.len(), 6, "{w}x{h}");
let entry = enc.sample_entry();
let mut dec = filmcraft_codecs::software_video_decoder(&entry).expect("software decoder");
let mut pictures = Vec::new();
for (i, p) in packets.iter().enumerate() {
pictures.extend(dec.decode(&p.data, i as i64).expect("decode"));
}
pictures.extend(dec.flush());
assert_eq!(pictures.len(), 6, "{w}x{h}");
assert!(
pictures.iter().all(|f| (f.frame.width, f.frame.height) == (w, h)),
"{w}x{h}: decoded {}x{}",
pictures[0].frame.width,
pictures[0].frame.height
);
}
}
#[test]
fn ffmpeg_decodes_what_the_hevc_encoder_writes() {
filmcraft_platform::register();
let Some(ffmpeg) = filmcraft_testkit::ffmpeg_or_skip("hardware HEVC export") else { return };
let Some(ffprobe) = filmcraft_testkit::ffprobe_or_skip("hardware HEVC export") else { return };
if !hevc_available() {
eprintln!("SKIPPED: no VideoToolbox hardware HEVC encoder");
return;
}
let frames = 65;
let (p, seq, m) = project(frames);
for (name, multiplexer) in [("oracle-hevc.mp4", filmcraft_export::Multiplexer::Mp4), ("oracle-hevc.mov", filmcraft_export::Multiplexer::Mov)] {
let path = tmp(name);
let s = ExportSettings { path: path.clone(), include_audio: false, multiplexer, ..hevc_settings() };
export(&p, seq, &s, &m, &Progress::default()).expect("export");
// an independent decoder reads the whole file without a complaint
let out = std::process::Command::new(&ffmpeg).args(["-v", "error", "-i", &path, "-f", "null", "-"]).output().expect("run ffmpeg");
assert!(out.status.success() && out.stderr.is_empty(), "{name}: ffmpeg: {}", String::from_utf8_lossy(&out.stderr));
// and finds the stream we asked for: HEVC Main, tagged hvc1 (what QuickTime takes), 4:2:0 BT.709
let out = std::process::Command::new(&ffprobe)
.args([
"-v",
"error",
"-count_frames",
"-select_streams",
"v:0",
"-show_entries",
"stream=codec_name,profile,codec_tag_string,pix_fmt,width,height,nb_read_frames,has_b_frames,color_space",
"-of",
"default=nw=1",
&path,
])
.output()
.expect("run ffprobe");
let info = String::from_utf8_lossy(&out.stdout).to_string();
for want in [
"codec_name=hevc",
"profile=Main",
"codec_tag_string=hvc1",
"pix_fmt=yuv420p",
&format!("width={W}"),
&format!("height={H}"),
&format!("nb_read_frames={frames}"),
"has_b_frames=0",
"color_space=bt709",
] {
assert!(info.contains(want), "{name}: ffprobe lacks `{want}`:\n{info}");
}
}
}
/// The sessions are created with a hardware encoder required; ask VideoToolbox whether it agrees, so
/// that a requirement dropped by accident could never become a silent software encode.
#[test]
fn the_sessions_run_on_the_hardware_encoder() {
let config = |profile| VtConfig {
width: W,
height: H,
timescale: 30,
frame_duration: 1,
keyframe_interval: 30,
profile,
rate: VtRate::Vbr { target_kbps: 6000, max_kbps: 9000 },
};
for (name, profile) in [("H.264", VtProfile::High), ("HEVC", VtProfile::HevcMain)] {
match VtEncoder::new(config(profile)) {
Ok(enc) => assert!(enc.uses_hardware(), "{name}: VideoToolbox does not report a hardware encoder for the session"),
Err(why) => eprintln!("SKIPPED {name}: no hardware encoder ({why})"),
}
}
}

View File

@ -461,7 +461,7 @@ fn settings_column(app: &mut FilmcraftApp, ui: &mut egui::Ui, rect: Rect) {
} }
if (s.has_audio() if (s.has_audio()
|| !s.has_video() || !s.has_video()
|| s.format == Format::H264 || s.format.is_h26x()
|| matches!(s.format, Format::ProRes | Format::DnxHr | Format::Apv | Format::Mjpeg) || matches!(s.format, Format::ProRes | Format::DnxHr | Format::Apv | Format::Mjpeg)
|| s.format.is_mxf()) || s.format.is_mxf())
&& !s.is_image_sequence() && !s.is_image_sequence()
@ -470,7 +470,7 @@ fn settings_column(app: &mut FilmcraftApp, ui: &mut egui::Ui, rect: Rect) {
{ {
audio_section(ui, &mut reg, s, &t, seq_sr); audio_section(ui, &mut reg, s, &t, seq_sr);
} }
if s.format == Format::H264 && section(ui, &mut reg, &mut ex.open_sections, "multiplexer", "Multiplexer", &t) { if s.format.is_h26x() && section(ui, &mut reg, &mut ex.open_sections, "multiplexer", "Multiplexer", &t) {
row(ui, &t, "Multiplexer", |ui| { row(ui, &t, "Multiplexer", |ui| {
let cur = if s.multiplexer == Multiplexer::Mp4 { "MP4" } else { "QuickTime" }; let cur = if s.multiplexer == Multiplexer::Mp4 { "MP4" } else { "QuickTime" };
if let Some(i) = combo(ui, &mut reg, "export.multiplexer", cur, &opts(&["MP4", "QuickTime"]), 160.0) { if let Some(i) = combo(ui, &mut reg, "export.multiplexer", cur, &opts(&["MP4", "QuickTime"]), 160.0) {
@ -507,7 +507,7 @@ fn settings_column(app: &mut FilmcraftApp, ui: &mut egui::Ui, rect: Rect) {
if section(ui, &mut reg, &mut ex.open_sections, "effects", "Effects", &t) { if section(ui, &mut reg, &mut ex.open_sections, "effects", "Effects", &t) {
pick_overlay = effects_section(ui, &mut reg, s, &t); pick_overlay = effects_section(ui, &mut reg, s, &t);
} }
if matches!(s.format, Format::H264 | Format::ProRes | Format::DnxHr | Format::Apv | Format::Mjpeg) if matches!(s.format, Format::H264 | Format::Hevc | Format::ProRes | Format::DnxHr | Format::Apv | Format::Mjpeg)
&& section(ui, &mut reg, &mut ex.open_sections, "metadata", "Metadata", &t) && section(ui, &mut reg, &mut ex.open_sections, "metadata", "Metadata", &t)
{ {
let m = &mut s.metadata; let m = &mut s.metadata;
@ -593,26 +593,34 @@ fn video_section(ui: &mut egui::Ui, reg: &mut Reg, s: &mut ExportSettings, t: &T
}); });
} }
match s.video_format() { match s.video_format() {
Format::H264 => { f @ (Format::H264 | Format::Hevc) => {
row(ui, t, "Profile", |ui| { let hevc = f == Format::Hevc;
let o = [H264Profile::Baseline, H264Profile::Main, H264Profile::High]; if hevc {
let labels: Vec<(String, bool)> = o.iter().map(|p| (p.label().to_string(), true)).collect(); // the hardware encoder writes one profile: Main, 8-bit 4:2:0, level chosen by the encoder
if let Some(i) = combo(ui, reg, "export.video.profile", s.h264_profile.label(), &labels, 120.0) { row(ui, t, "Profile", |ui| ui.label("Main (8-bit)"));
s.h264_profile = o[i]; row(ui, t, "Encoder", |ui| ui.label("Hardware (H.265 has no software encoder)"));
} } else {
}); row(ui, t, "Profile", |ui| {
row(ui, t, "Level", |ui| { let o = [H264Profile::Baseline, H264Profile::Main, H264Profile::High];
let levels: [Option<u8>; 13] = let labels: Vec<(String, bool)> = o.iter().map(|p| (p.label().to_string(), true)).collect();
[None, Some(30), Some(31), Some(32), Some(40), Some(41), Some(42), Some(50), Some(51), Some(52), Some(60), Some(61), Some(62)]; if let Some(i) = combo(ui, reg, "export.video.profile", s.h264_profile.label(), &labels, 120.0) {
let lab = |l: &Option<u8>| l.map(|l| format!("{}.{}", l / 10, l % 10)).unwrap_or_else(|| "Auto".into()); s.h264_profile = o[i];
let labels: Vec<(String, bool)> = levels.iter().map(|l| (lab(l), true)).collect(); }
if let Some(i) = combo(ui, reg, "export.video.level", &lab(&s.h264_level), &labels, 120.0) { });
s.h264_level = levels[i]; row(ui, t, "Level", |ui| {
} let levels: [Option<u8>; 13] =
}); [None, Some(30), Some(31), Some(32), Some(40), Some(41), Some(42), Some(50), Some(51), Some(52), Some(60), Some(61), Some(62)];
let lab = |l: &Option<u8>| l.map(|l| format!("{}.{}", l / 10, l % 10)).unwrap_or_else(|| "Auto".into());
let labels: Vec<(String, bool)> = levels.iter().map(|l| (lab(l), true)).collect();
if let Some(i) = combo(ui, reg, "export.video.level", &lab(&s.h264_level), &labels, 120.0) {
s.h264_level = levels[i];
}
});
}
row(ui, t, "Bitrate Encoding", |ui| { row(ui, t, "Bitrate Encoding", |ui| {
// a hardware encoder has one pass: H.265 offers no two-pass mode
let o = [BitrateMode::Cbr, BitrateMode::Vbr1Pass, BitrateMode::Vbr2Pass]; let o = [BitrateMode::Cbr, BitrateMode::Vbr1Pass, BitrateMode::Vbr2Pass];
let labels: Vec<(String, bool)> = o.iter().map(|m| (m.label().to_string(), true)).collect(); let labels: Vec<(String, bool)> = o.iter().map(|m| (m.label().to_string(), !(hevc && *m == BitrateMode::Vbr2Pass))).collect();
if let Some(i) = combo(ui, reg, "export.video.bitrateMode", s.bitrate_mode.label(), &labels, 140.0) { if let Some(i) = combo(ui, reg, "export.video.bitrateMode", s.bitrate_mode.label(), &labels, 140.0) {
s.bitrate_mode = o[i]; s.bitrate_mode = o[i];
} }
@ -655,12 +663,15 @@ fn video_section(ui: &mut egui::Ui, reg: &mut Reg, s: &mut ExportSettings, t: &T
}); });
// The operating system's hardware encoder where there is one (macOS); everything it does // The operating system's hardware encoder where there is one (macOS); everything it does
// not take (two-pass, HDR, MXF) and every machine without one keeps the built-in encoder. // not take (two-pass, HDR, MXF) and every machine without one keeps the built-in encoder.
row(ui, t, "Hardware Encoding", |ui| { // H.265 has only the hardware encoder: choosing the format is the opt-in.
let mut on = s.hardware_encoding == HardwareEncoding::Auto; if !hevc {
if check(ui, reg, "export.video.hardwareEncoding", &mut on, "Use the hardware encoder when available") { row(ui, t, "Hardware Encoding", |ui| {
s.hardware_encoding = if on { HardwareEncoding::Auto } else { HardwareEncoding::Off }; let mut on = s.hardware_encoding == HardwareEncoding::Auto;
} if check(ui, reg, "export.video.hardwareEncoding", &mut on, "Use the hardware encoder when available") {
}); s.hardware_encoding = if on { HardwareEncoding::Auto } else { HardwareEncoding::Off };
}
});
}
} }
Format::ProRes => { Format::ProRes => {
row(ui, t, "Profile", |ui| { row(ui, t, "Profile", |ui| {
@ -714,13 +725,12 @@ fn audio_section(ui: &mut egui::Ui, reg: &mut Reg, s: &mut ExportSettings, t: &T
} }
} }
row(ui, t, "Audio Format", |ui| { row(ui, t, "Audio Format", |ui| {
let fixed = s.format == Format::H264 && s.multiplexer == Multiplexer::Mp4 || audio_only || s.format.is_mxf(); let fixed = s.format.is_h26x() && s.multiplexer == Multiplexer::Mp4 || audio_only || s.format.is_mxf();
let cur = match s.audio_codec() { let cur = match s.audio_codec() {
AudioCodec::Aac => "AAC", AudioCodec::Aac => "AAC",
_ => "Uncompressed (PCM)", _ => "Uncompressed (PCM)",
}; };
let labels = let labels = vec![("AAC".to_string(), !audio_only), ("Uncompressed (PCM)".to_string(), !(s.format.is_h26x() && s.multiplexer == Multiplexer::Mp4))];
vec![("AAC".to_string(), !audio_only), ("Uncompressed (PCM)".to_string(), !(s.format == Format::H264 && s.multiplexer == Multiplexer::Mp4))];
if fixed { if fixed {
ui.label(cur); ui.label(cur);
} else if let Some(i) = combo(ui, reg, "export.audio.codec", cur, &labels, 180.0) { } else if let Some(i) = combo(ui, reg, "export.audio.codec", cur, &labels, 180.0) {

View File

@ -179,6 +179,38 @@ fn export_mode_settings_and_summary() {
d.snapshot("export-mode"); d.snapshot("export-mode");
} }
#[test]
fn h265_shows_the_h264_family_controls_without_the_h264_only_ones() {
use filmcraft_engine::export::{ExportSettings, Format};
let mut d = Driver::new("h265");
d.ok("ui.set", json!({"mode": "export"}));
d.frames(4);
// H.264, the default: profile, level and the hardware checkbox
for id in ["export.video.profile", "export.video.level", "export.video.hardwareEncoding", "export.video.bitrateMode"] {
assert!(d.has(id), "H.264: {id}");
}
d.app().ui.export.settings = ExportSettings { format: Format::Hevc, ..Default::default() };
d.frames(4);
for id in [
"export.video.bitrateMode",
"export.video.target",
"export.video.max",
"export.video.keyframeOn",
"export.section.multiplexer",
"export.section.audio",
"export.audio.sampleRate",
] {
assert!(d.has(id), "H.265: {id}");
}
// Main is its only profile, the level is the encoder's choice, and hardware is the only encoder
for id in ["export.video.profile", "export.video.level", "export.video.hardwareEncoding"] {
assert!(!d.has(id), "H.265 has no {id}");
}
let sum = d.ok("ui.elements", json!({"prefix": "export.summary"}))[0]["label"].as_str().unwrap().to_string();
assert!(sum.contains("HEVC Main") && sum.contains("Target 20.00 Mbps"), "{sum}");
d.snapshot("export-h265");
}
#[test] #[test]
fn preset_manager_search_favourite_save_apply() { fn preset_manager_search_favourite_save_apply() {
let mut d = Driver::new("manager"); let mut d = Driver::new("manager");

View File

@ -83,3 +83,12 @@ deliberate:
Everything else in this record applies unchanged: `unsafe` stays in `crates/platform`, every block Everything else in this record applies unchanged: `unsafe` stays in `crates/platform`, every block
has a `// SAFETY:` comment, no panic crosses the FFI boundary, the public API is safe, and the crate has a `// SAFETY:` comment, no panic crosses the FFI boundary, the public API is safe, and the crate
compiles everywhere (`register()` is a no-op off macOS). compiles everywhere (`register()` is a no-op off macOS).
## Addendum (2026-10-07): hardware H.265 (HEVC) encoding
The same VideoToolbox session wrapper also creates HEVC sessions (`VtProfile::HevcMain`). The rules
above hold; HEVC adds one difference. The crate has no software HEVC encoder to fall back to, and
there will not be one (pure Rust, clean-room: x265 is GPL), so `Format::Hevc` exists only where the
OS has a hardware encoder. Choosing the format is the opt-in, `filmcraft_export::available` asks a
probe the platform crate registers, and what the hardware path does not take (two-pass, odd sizes,
a machine without the encoder) is an error naming the reason instead of a different encoder.

View File

@ -72,7 +72,7 @@ and `filmcraft-cli`.
| `engine` | L4 | `Session`, command registry, undo history, media pool, jobs, interchange glue | | `engine` | L4 | `Session`, command registry, undo history, media pool, jobs, interchange glue |
| `ui-egui` | L5 | the egui frontend: docking, panels, timeline, monitors, playback, control-channel handlers | | `ui-egui` | L5 | the egui frontend: docking, panels, timeline, monitors, playback, control-channel handlers |
| `automation` | L5 | MCP server (`rmcp`, stdio), headless or bridged to the running app | | `automation` | L5 | MCP server (`rmcp`, stdio), headless or bridged to the running app |
| `platform` | L5 | OS media FFI only: hardware video decoding (VideoToolbox H.264 / HEVC on macOS; a no-op elsewhere) behind `codecs::VideoDecoder`, with transparent fallback to our decoders, and opt-in hardware H.264 encoding (VideoToolbox) behind `export::VideoEncoder`, which declines to the built-in encoder for what it does not take. The one crate allowed `unsafe` ([ADR 0001](adr/0001-platform-ffi.md), [README](../crates/platform/README.md)) | | `platform` | L5 | OS media FFI only: hardware video decoding (VideoToolbox H.264 / HEVC on macOS; a no-op elsewhere) behind `codecs::VideoDecoder`, with transparent fallback to our decoders, and hardware H.264 encoding (VideoToolbox, opt-in) and H.265 encoding (VideoToolbox, the only H.265 encoder; the format exists only where a hardware encoder does) behind `export::VideoEncoder`; H.264 declines to the built-in encoder for what the hardware does not take. The one crate allowed `unsafe` ([ADR 0001](adr/0001-platform-ffi.md), [README](../crates/platform/README.md)) |
| `filmcraft` | L6 | desktop binary: eframe/wgpu window, cpal audio output, file dialogs, native macOS menu, TCP control server | | `filmcraft` | L6 | desktop binary: eframe/wgpu window, cpal audio output, file dialogs, native macOS menu, TCP control server |
| `filmcraft-cli` | L6 | headless CLI: `exec`, `run`, `inspect`, `describe`, `commands`, `import`, `export`, `render`, `probe`, `mcp`; `--bridge` targets the running app | | `filmcraft-cli` | L6 | headless CLI: `exec`, `run`, `inspect`, `describe`, `commands`, `import`, `export`, `render`, `probe`, `mcp`; `--bridge` targets the running app |
| `filmcraft-web` | L6 | the browser app (wasm32): eframe web runner on WebGPU/WebGL2, Blob-backed services, OPFS recovery, WebAudio, WebCodecs, `window.filmcraft` API ([web.md](web.md)) | | `filmcraft-web` | L6 | the browser app (wasm32): eframe web runner on WebGPU/WebGL2, Blob-backed services, OPFS recovery, WebAudio, WebCodecs, `window.filmcraft` API ([web.md](web.md)) |

View File

@ -181,6 +181,43 @@ the media engine removes most of the first and lets the next batch render while
encodes (the exporter renders a batch of frames in parallel, then encodes it). What is left is the encodes (the exporter renders a batch of frames in parallel, then encodes it). What is left is the
CPU compositor, the next target of #30 ("GPU export"). CPU compositor, the next target of #30 ("GPU export").
## Results (HW3: VideoToolbox hardware H.265 (HEVC) encoding, against hardware H.264)
Apple M1 (8 cores, 16 GB), 2026-10-07, single runs on an otherwise idle machine, release build. Both
encoders are the M1's media engine. H.265 is `Format::Hevc` (Main, 8-bit 4:2:0, one-pass VBR, no
B-frames); H.264 is the opt-in hardware encoder of HW2 (High).
### Speed
The raw encoders (ffmpeg's `hevc_videotoolbox` and `h264_videotoolbox` on a 1080p25 test pattern, 500
frames): 166 and 189 frames/s. H.265 is the codec that compresses further, not the faster one to encode.
The 9:29 timeline of HW2 (camera clip, ProRes 4444 overlays, AAC, loudness normalised to −14 LUFS), 20 Mb/s
H.264 and 10 Mb/s H.265: **204 s and 195 s** (CPU 1042 s and 1029 s), files of 1417 MB and 719 MB. The
same time: the same H.264 export took 193 s and 200 s on other runs. The export is bound by the CPU
compositor and the decoders, not by the encoder, so a different hardware codec does not change it.
### Quality per bit
30 s with a banner (15–45 s of that timeline), exported by FilmCraft as ProRes 422 HQ (the reference)
and by each encoder at several bitrates; ffmpeg's `psnr` and `ssim` of the decoded 4:2:0 pictures against
the reference. Average PSNR over the three planes, and SSIM (higher is better):
| Mb/s | H.264 | H.265 | difference | H.264 Mb/s for H.265's PSNR |
|---|---|---|---|---|
| 20 | 46.88 dB, 0.9902 | 46.91 dB, 0.9903 | +0.03 dB | beyond the range measured |
| 10 | 44.96 dB, 0.9863 | 45.09 dB, 0.9863 | +0.13 dB | 10.5 |
| 6 | 43.72 dB, 0.9833 | 44.31 dB, 0.9843 | +0.59 dB | 7.7 |
| 4 | 42.91 dB, 0.9807 | 43.64 dB, 0.9825 | +0.73 dB | 5.8 |
| 3 | 42.37 dB, 0.9786 | 43.05 dB, 0.9807 | +0.68 dB | 4.3 |
At 10 Mb/s and above the two give the same picture. At 3–6 Mb/s H.265 reaches the same PSNR with 22–31 %
less bitrate. Its worst frame is better at every bitrate (lowest per-frame PSNR 45.9 against 42.1 dB at
20 Mb/s, 43.1 against 41.5 dB at 10, 39.5 against 37.8 dB at 3).
So there is no case for a lower default bitrate: the two formats start at the same 20 Mb/s. H.265
pays off where the file size matters, at 3–6 Mb/s; that 9:29 timeline is 428 MB at 6 Mb/s.
## Results (GPU1: blend modes on the GPU compositor, #30, before → after) ## Results (GPU1: blend modes on the GPU compositor, #30, before → after)
Before = this change with the old whole-frame CPU fallback for non-Normal blend modes put back Before = this change with the old whole-frame CPU fallback for non-Normal blend modes put back