Merge M12 (multicam) into M5.5–7 (masks/presets): schema v7 = masks, v8 = multicam (WIP: CI not yet run)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Brandon Thomas 2026-10-01 11:55:13 -04:00
commit a9278c81cc
35 changed files with 3859 additions and 28 deletions

View File

@ -31,7 +31,7 @@ Status: ✅ done · 🟡 in progress · ⬜ not started. Estimates are remaining
| M0 | Skeleton + visual shell | ✅ | Workspace, 20 crates, dock/workspaces, Premiere 26 look, native menus, control channel, MCP, xtask gates (layers, wasm) | — | — | | M0 | Skeleton + visual shell | ✅ | Workspace, 20 crates, dock/workspaces, Premiere 26 look, native menus, control channel, MCP, xtask gates (layers, wasm) | — | — |
| M1 | Media I/O | ✅ | MP4/MOV demux+mux, WAV, stills, MJPEG, symphonia audio (MP3/FLAC/ALAC/Vorbis), GOP seek + frame cache, import | Media Browser polish | 2 | | M1 | Media I/O | ✅ | MP4/MOV demux+mux, WAV, stills, MJPEG, symphonia audio (MP3/FLAC/ALAC/Vorbis), GOP seek + frame cache, import | Media Browser polish | 2 |
| M2 | H.264 decoder | ✅ | Own decoder, bit-exact on 37+ streams, 500–600 fps 1080p | — | — | | M2 | H.264 decoder | ✅ | Own decoder, bit-exact on 37+ streams, 500–600 fps 1080p | — | — |
| M3 | Editing core | 🟡 | Edit algebra (insert/overwrite/razor/lift/extract/ripple/roll/slip/slide/rate-stretch/nest/paste), tools, markers, trim mode + Trim Monitor + dynamic J/K/L trimming, Keyboard Shortcuts editor with FilmCraft/Premiere/FCP/Avid presets | Multicam, subclip polish, ~220 Premiere default shortcuts whose commands don't exist yet | 6–10 | | M3 | Editing core | 🟡 | Edit algebra (insert/overwrite/razor/lift/extract/ripple/roll/slip/slide/rate-stretch/nest/paste), tools, markers, trim mode + Trim Monitor + dynamic J/K/L trimming, Keyboard Shortcuts editor with FilmCraft/Premiere/FCP/Avid presets; multicam (Create Multi-Camera Source Sequence, Multi-Camera view with live switching on 1–9, angle switching, Enable/Flatten, Edit Cameras) and sync (Synchronize, Merge Clips; In/Out/timecode/marker/audio — GCC-PHAT, sample-accurate) | Multicam paging >16 angles and grid thumbnails, subclip polish, ~220 Premiere default shortcuts whose commands don't exist yet | 6–10 |
| M4 | Playback | 🟡 | Audio-clock master, prefetch, J/K/L, dropped-frame stats, playback resolution | Render previews, render bar, 4K/8K tuning | 8–12 | | M4 | Playback | 🟡 | Audio-clock master, prefetch, J/K/L, dropped-frame stats, playback resolution | Render previews, render bar, 4K/8K tuning | 8–12 |
| M5 | Effects, keyframes, GPU | 🟡 | ~60 CPU effects, 30 transitions, keyframes + value/velocity graphs, wgpu compositor | Full ~150-effect catalogue, WGSL parity for all, masks + tracking, adjustment layers, presets, Warp Stabilizer, Morph Cut | 30–45 | | M5 | Effects, keyframes, GPU | 🟡 | ~60 CPU effects, 30 transitions, keyframes + value/velocity graphs, wgpu compositor | Full ~150-effect catalogue, WGSL parity for all, masks + tracking, adjustment layers, presets, Warp Stabilizer, Morph Cut | 30–45 |
| M6 | Export | ✅ | Own H.264 encoder (High/Main/Baseline, B-frames, VBR/CBR/2-pass) → MP4 + own AAC; ProRes, MJPEG, PNG, GIF, WAV; background jobs | Preset library, queue UI, smart render | 6–8 | | M6 | Export | ✅ | Own H.264 encoder (High/Main/Baseline, B-frames, VBR/CBR/2-pass) → MP4 + own AAC; ProRes, MJPEG, PNG, GIF, WAV; background jobs | Preset library, queue UI, smart render | 6–8 |
@ -48,6 +48,7 @@ Status: ✅ done · 🟡 in progress · ⬜ not started. Estimates are remaining
## Log ## Log
- **2026-10-01 (M12 multicam):** Synchronize / Merge Clips / Create Multi-Camera Source Sequence (sync by In, Out, timecode ± hours, clip markers, or audio: GCC-PHAT cross-correlation, offsets recovered to the sample on noisy multi-mic recordings), multi-camera clips (angle render, Switch Audio), Program monitor Multi-Camera view with live switching (keys 1–9, one undo step per pass), Ctrl+1–9 cuts, Enable / Flatten / Edit Cameras, project schema v7. Fixed: nested sequence audio was silent.
- **2026-10-01 (evening):** M11 done: offline media with our own slate and the Link Media dialog (fingerprint-checked relink, folder remap, search), proxies (create in background / attach / toggle; export stays full-res) with ingest settings, Project Manager (collect, consolidate + transcode). Fixed: GOP cache deadlock when a rayon decoder inside a parallel export re-entered the same source. - **2026-10-01 (evening):** M11 done: offline media with our own slate and the Link Media dialog (fingerprint-checked relink, folder remap, search), proxies (create in background / attach / toggle; export stays full-res) with ingest settings, Project Manager (collect, consolidate + transcode). Fixed: GOP cache deadlock when a rayon decoder inside a parallel export re-entered the same source.
- **2026-10-01 (M7.4):** Essential Sound panel: Dialogue/Music/SFX/Ambience types, Loudness auto-match (BS.1770, exact to the target), Repair (noise, rumble, hum, new DeEsser and spectral DeReverb), Clarity (dynamics, EQ presets, Enhance Speech DSP chain), Creative reverb / stereo width, ducking that writes Volume keyframes, presets; all `essentialSound.*` commands, effects visible and keyframable in Effect Controls. - **2026-10-01 (M7.4):** Essential Sound panel: Dialogue/Music/SFX/Ambience types, Loudness auto-match (BS.1770, exact to the target), Repair (noise, rumble, hum, new DeEsser and spectral DeReverb), Clarity (dynamics, EQ presets, Enhance Speech DSP chain), Creative reverb / stereo width, ducking that writes Volume keyframes, presets; all `essentialSound.*` commands, effects visible and keyframable in Effect Controls.
- **2026-10-01 (night):** M8 colour: camera log curves and gamuts from published specs, colour-managed pipeline (PQ/HLG working spaces, Interpret Footage, BT.2390 tone mapping), HDR export signalling verified with ffprobe, LUTs (.cube/.3dl, tetrahedral CPU + WGSL, project library, Lumetri Input LUT / Look), Colour Match, Lumetri section bypass, HDR scopes. - **2026-10-01 (night):** M8 colour: camera log curves and gamuts from published specs, colour-managed pipeline (PQ/HLG working spaces, Interpret Footage, BT.2390 tone mapping), HDR export signalling verified with ffprobe, LUTs (.cube/.3dl, tetrahedral CPU + WGSL, project library, Lumetri Input LUT / Look), Colour Match, Lumetri section bypass, HDR scopes.

View File

@ -89,6 +89,22 @@ Pure functions for Essential Sound auto-ducking:
merge; clamped to the clip (a ramp cut by an edge gets the ramp's value at the edge); strictly merge; clamped to the clip (a ramp cut by an edge gets the ramp's value at the edge); strictly
increasing times. increasing times.
## Audio synchronisation (`sync`)
`find_offset(a, b, sample_rate, &SyncOptions)` returns the lag `L` with `b[n] ≈ g·a[n + L]`, a
sub-sample estimate, and two trust figures (`confidence` = peak / RMS of the correlation,
`distinct` = peak / next peak > 50 ms away; `reliable()` = ≥ 8 and ≥ 1.5).
- Coarse: DC removal + normalisation, Blackman-windowed-sinc decimation to ≤ 8 kHz (further for
long recordings, so the transform stays ≤ 4 M points), GCC-PHAT-β (`R = A·B*/|A·B*|^0.75`,
40 Hz – 0.4·rate band) with both signals packed into one complex FFT.
- Refine: the same correlation at the full rate on the loudest common window (≤ 2¹⁷ samples), ±2
coarse steps around the coarse peak, parabolic interpolation.
- Tested within one sample on speech-like multi-microphone signals (gains −20…+10 dB, SNR down to
3 dB, coloured microphones, fractional delays: worst 0.4 samples integer / 0.11 samples
sub-sample), a 0 dB SNR room with a strong echo, inverted polarity, unrelated signals
(unreliable) and `max_lag`. Two 10-minute recordings: 2.4 s (release, one core).
## Known limits ## Known limits
- Effects assume ≤ the channel count they were built with; extra channels pass through - Effects assume ≤ the channel count they were built with; extra channels pass through

View File

@ -6,6 +6,7 @@
//! registry ([`effects()`]) describing every effect's parameters (range, unit, default) so the //! registry ([`effects()`]) describing every effect's parameters (range, unit, default) so the
//! engine and UI can build controls generically. //! engine and UI can build controls generically.
//! * [`biquad`] — RBJ-cookbook biquads (TDF-II) with analytic magnitude response. //! * [`biquad`] — RBJ-cookbook biquads (TDF-II) with analytic magnitude response.
//! * [`sync`] — offset between two recordings of one event (GCC-PHAT, sample-accurate).
//! //!
//! Conventions: audio is **planar f32** (`&mut [&mut [f32]]`, one slice per channel, all the same //! Conventions: audio is **planar f32** (`&mut [&mut [f32]]`, one slice per channel, all the same
//! length). `process` never allocates; all buffers are sized at construction. Parameters may be //! length). `process` never allocates; all buffers are sized at construction. Parameters may be
@ -21,6 +22,7 @@ pub mod fft;
pub mod loudness; pub mod loudness;
pub mod oversample; pub mod oversample;
mod smooth; mod smooth;
pub mod sync;
pub use effects::{Category, EffectInfo, create_effect, effect_info, effects}; pub use effects::{Category, EffectInfo, create_effect, effect_info, effects};
pub use loudness::{LoudnessMeter, normalize_gain_db}; pub use loudness::{LoudnessMeter, normalize_gain_db};

View File

@ -0,0 +1,505 @@
//! Synchronising recordings of the same event by their audio (Synchronize, Merge Clips and
//! Create Multi-Camera Source Sequence "Audio").
//!
//! [`find_offset`] returns the lag `L` (in samples) for which `b[n] ≈ g · a[n + L]`: when `b`
//! started recording two seconds after `a`, `L = 2 · sample_rate`. It works in two passes:
//!
//! 1. **Coarse.** Both signals are DC-removed, normalised, low-pass filtered (windowed sinc) and
//! decimated to ≤ 8 kHz (further for very long recordings, so the transform stays ≤ 4 M points).
//! The generalised cross-correlation with partial phase transform weighting (GCC-PHAT-β:
//! `R = A·B* / |A·B*|^β`) is computed with one complex FFT of both signals packed as `a + i·b`.
//! PHAT weighting whitens the spectra, so the peak is sharp and independent of the microphones'
//! gains and frequency responses; β < 1 keeps noise-only bins from dominating.
//! 2. **Refine.** Around the coarse peak, the same correlation is computed at the full sample rate
//! on the loudest common window (≤ 2.7 s at 48 kHz), and the peak is refined by parabolic
//! interpolation to a fraction of a sample.
//!
//! Two figures say how much to trust the result: `confidence`, the coarse peak's height over the
//! RMS of the correlation, and `distinct`, its height over the highest peak more than 50 ms away
//! (periodic material such as music or a steady hum can match at several lags).
//!
//! References: Knapp & Carter, "The generalized correlation method for estimation of time delay",
//! IEEE Trans. ASSP 24(4), 1976; the β-weighted PHAT variant is standard textbook practice.
use crate::fft::Fft;
/// Tuning of [`find_offset`].
#[derive(Clone, Copy, Debug, PartialEq)]
pub struct SyncOptions {
/// Largest |lag| considered, in samples (None = anything with enough overlap).
pub max_lag: Option<i64>,
/// Upper bound of the coarse search's sample rate (Hz).
pub coarse_rate: u32,
/// Most samples (both signals together) the coarse transform may hold; longer recordings are
/// decimated further.
pub max_coarse_samples: usize,
/// PHAT weighting exponent (1 = full whitening, 0 = plain cross-correlation).
pub phat_beta: f32,
/// Length of the full-rate refinement window (samples).
pub refine_window: usize,
}
impl Default for SyncOptions {
fn default() -> Self {
Self { max_lag: None, coarse_rate: 8000, max_coarse_samples: 1 << 22, phat_beta: 0.75, refine_window: 1 << 17 }
}
}
/// The offset found between two recordings.
#[derive(Clone, Copy, Debug, PartialEq)]
pub struct SyncResult {
/// `b[n]` lines up with `a[n + lag]` (samples at the input rate).
pub lag: i64,
/// The same with sub-sample precision (parabolic interpolation of the refined peak).
pub lag_fine: f64,
/// Coarse peak height over the RMS of the correlation.
pub confidence: f32,
/// Coarse peak height over the next-highest peak more than 50 ms away.
pub distinct: f32,
/// True when the signals correlate negatively (one recording has inverted polarity).
pub inverted: bool,
}
impl SyncResult {
/// Whether the match is clear enough to trust.
pub fn reliable(&self) -> bool {
self.confidence >= 8.0 && self.distinct >= 1.5
}
}
/// Average interleaved or planar channels into one mono signal.
pub fn mixdown(channels: &[&[f32]]) -> Vec<f32> {
let n = channels.iter().map(|c| c.len()).min().unwrap_or(0);
let k = channels.len().max(1) as f32;
(0..n).map(|i| channels.iter().map(|c| c[i]).sum::<f32>() / k).collect()
}
/// Low-pass filter (Blackman-windowed sinc, cutoff at 0.42 × the new sample rate) and keep every
/// `d`-th sample.
pub fn decimate(x: &[f32], d: usize) -> Vec<f32> {
if d <= 1 {
return x.to_vec();
}
let half = 8 * d;
let fc = 0.42 / d as f64; // cycles per input sample
let taps: Vec<f32> = {
let n = 2 * half + 1;
let mut h: Vec<f64> = (0..n)
.map(|k| {
let t = k as f64 - half as f64;
let sinc = if t == 0.0 { 2.0 * fc } else { (2.0 * std::f64::consts::PI * fc * t).sin() / (std::f64::consts::PI * t) };
let w = 0.42 - 0.5 * (2.0 * std::f64::consts::PI * k as f64 / (n - 1) as f64).cos()
+ 0.08 * (4.0 * std::f64::consts::PI * k as f64 / (n - 1) as f64).cos();
sinc * w
})
.collect();
let s: f64 = h.iter().sum();
h.iter_mut().for_each(|v| *v /= s);
h.into_iter().map(|v| v as f32).collect()
};
let out_len = x.len().div_ceil(d);
let mut out = Vec::with_capacity(out_len);
for i in 0..out_len {
let c = (i * d) as isize;
let lo = (c - half as isize).max(0) as usize;
let hi = ((c + half as isize + 1) as usize).min(x.len());
let k0 = (lo as isize - (c - half as isize)) as usize;
let mut acc = 0.0f32;
for (j, &v) in x[lo..hi].iter().enumerate() {
acc += v * taps[k0 + j];
}
out.push(acc);
}
out
}
/// Remove the mean and scale to unit RMS (silence stays silence).
fn normalise(x: &mut [f32]) {
if x.is_empty() {
return;
}
let mean = (x.iter().map(|&v| v as f64).sum::<f64>() / x.len() as f64) as f32;
x.iter_mut().for_each(|v| *v -= mean);
let rms = (x.iter().map(|&v| (v as f64) * (v as f64)).sum::<f64>() / x.len() as f64).sqrt();
if rms > 1e-12 {
let k = (1.0 / rms) as f32;
x.iter_mut().for_each(|v| *v *= k);
}
}
/// GCC-PHAT-β of `a` and `b` (zero-padded to `n`, a power of two ≥ `a.len() + b.len()`):
/// `r[L mod n] = Σ a[i + L]·b[i]` with whitened spectra. Only bins whose frequency (cycles per
/// sample) lies in `band` contribute.
fn gcc(a: &[f32], b: &[f32], n: usize, beta: f32, band: (f32, f32)) -> Vec<f32> {
let fft = Fft::new(n);
let mut re = vec![0.0f32; n];
let mut im = vec![0.0f32; n];
re[..a.len()].copy_from_slice(a);
im[..b.len()].copy_from_slice(b);
fft.forward(&mut re, &mut im);
// X = A + iB for real a, b: A(k) = (X(k) + X*(n-k)) / 2, B(k) = (X(k) - X*(n-k)) / 2i.
let weigh = |k: usize, xr: f32, xi: f32, yr: f32, yi: f32| -> (f32, f32) {
// y = X*(n-k) given as (yr, yi) already conjugated
let (ar, ai) = (0.5 * (xr + yr), 0.5 * (xi + yi));
let (dr, di) = (xr - yr, xi - yi);
let (br, bi) = (0.5 * di, -0.5 * dr);
// R = A·conj(B)
let rr = ar * br + ai * bi;
let ri = ai * br - ar * bi;
let f = k.min(n - k) as f32 / n as f32;
if f < band.0 || f > band.1 {
return (0.0, 0.0);
}
let mag = (rr * rr + ri * ri).sqrt();
if mag <= 1e-30 {
return (0.0, 0.0);
}
let w = mag.powf(-beta);
(rr * w, ri * w)
};
for k in 0..=n / 2 {
let j = (n - k) % n;
let (xkr, xki, xjr, xji) = (re[k], im[k], re[j], im[j]);
let rk = weigh(k, xkr, xki, xjr, -xji);
let rj = weigh(j, xjr, xji, xkr, -xki);
re[k] = rk.0;
im[k] = rk.1;
re[j] = rj.0;
im[j] = rj.1;
}
fft.inverse(&mut re, &mut im);
re
}
fn corr_at(r: &[f32], lag: i64) -> f32 {
let n = r.len() as i64;
r[lag.rem_euclid(n) as usize]
}
/// Find the lag `L` with `b[n] ≈ g·a[n + L]` (see the module docs). `None` when either signal is
/// too short or silent, or no lag leaves enough overlap.
pub fn find_offset(a: &[f32], b: &[f32], sample_rate: u32, opts: &SyncOptions) -> Option<SyncResult> {
let sr = sample_rate.max(1) as usize;
let min_len = (sr / 10).max(64);
if a.len() < min_len || b.len() < min_len {
return None;
}
let energy = |x: &[f32]| x.iter().map(|&v| (v as f64) * (v as f64)).sum::<f64>();
if energy(a) < 1e-12 || energy(b) < 1e-12 {
return None;
}
// ---- coarse
let d = (sr.div_ceil(opts.coarse_rate.max(500) as usize)).max((a.len() + b.len()).div_ceil(opts.max_coarse_samples.max(1 << 12))).max(1);
let mut ca = a.to_vec();
let mut cb = b.to_vec();
normalise(&mut ca);
normalise(&mut cb);
let (ca, cb) = (decimate(&ca, d), decimate(&cb, d));
let (la, lb) = (ca.len() as i64, cb.len() as i64);
let n = ((la + lb) as usize).next_power_of_two();
let rate_c = sr as f32 / d as f32;
let band = ((40.0 / rate_c).min(0.05), if d > 1 { 0.40 } else { 0.5 });
let r = gcc(&ca, &cb, n, opts.phat_beta, band);
// valid lags: b[0..lb) against a[L..L+lb) must overlap by a minimum amount
let min_overlap = ((rate_c as i64) / 2).min(la.min(lb) / 4).max(8);
let max_lag_c = opts.max_lag.map(|m| m.abs() / d as i64 + 1);
let lo = -(lb - min_overlap);
let hi = la - min_overlap;
if hi < lo {
return None;
}
let mut best = (0i64, 0.0f32);
let mut sum2 = 0.0f64;
let mut count = 0usize;
for l in lo..=hi {
if max_lag_c.is_some_and(|m| l.abs() > m) {
continue;
}
let v = corr_at(&r, l);
sum2 += (v as f64) * (v as f64);
count += 1;
if v.abs() > best.1.abs() {
best = (l, v);
}
}
if count == 0 || best.1 == 0.0 {
return None;
}
let rms = (sum2 / count as f64).sqrt().max(1e-30) as f32;
let confidence = best.1.abs() / rms;
let guard = ((rate_c * 0.05) as i64).max(2);
let mut second = 0.0f32;
for l in lo..=hi {
if (l - best.0).abs() <= guard || max_lag_c.is_some_and(|m| l.abs() > m) {
continue;
}
second = second.max(corr_at(&r, l).abs());
}
let distinct = if second > 0.0 { best.1.abs() / second } else { f32::INFINITY };
let inverted = best.1 < 0.0;
let coarse = best.0 * d as i64;
// ---- refine at the full rate
let margin = (2 * d + 4) as i64;
let (fa, fb) = (a.len() as i64, b.len() as i64);
let ov0 = (-coarse).max(0);
let ov1 = fb.min(fa - coarse);
if ov1 - ov0 < 16 {
return Some(SyncResult { lag: coarse, lag_fine: coarse as f64, confidence, distinct, inverted });
}
let w = ((ov1 - ov0) as usize).min(opts.refine_window.max(256)) as i64;
// the window of b with the most energy (both sides: a lag-shifted)
let start = {
let hop = (w / 4).max(1);
let pre = |x: &[f32]| {
let mut p = Vec::with_capacity(x.len() + 1);
p.push(0.0f64);
let mut s = 0.0;
for &v in x {
s += (v as f64) * (v as f64);
p.push(s);
}
p
};
let (pa, pb) = (pre(a), pre(b));
let mut s = ov0;
let mut best_s = ov0;
let mut best_e = -1.0f64;
while s + w <= ov1 {
let eb = pb[(s + w) as usize] - pb[s as usize];
let ea = pa[(s + coarse + w) as usize] - pa[(s + coarse) as usize];
let e = eb.min(ea);
if e > best_e {
best_e = e;
best_s = s;
}
s += hop;
}
best_s
};
let bw: Vec<f32> = b[start as usize..(start + w) as usize].to_vec();
let aw: Vec<f32> = (start + coarse - margin..start + coarse + w + margin).map(|i| if i >= 0 && i < fa { a[i as usize] } else { 0.0 }).collect();
let mut aw = aw;
let mut bw = bw;
normalise(&mut aw);
normalise(&mut bw);
let n2 = (aw.len() + bw.len()).next_power_of_two();
let r2 = gcc(&aw, &bw, n2, opts.phat_beta, (20.0 / sr as f32, 0.5));
// r2[L'] = Σ aw[i + L']·bw[i]; L' = margin + δ
let sign = if inverted { -1.0 } else { 1.0 };
let mut bl = 0i64;
let mut bv = f32::MIN;
for l in 0..=2 * margin {
let v = corr_at(&r2, l) * sign;
if v > bv {
bv = v;
bl = l;
}
}
let (y0, y1, y2) = (corr_at(&r2, bl - 1) * sign, bv, corr_at(&r2, bl + 1) * sign);
let den = y0 - 2.0 * y1 + y2;
let frac = if den.abs() > 1e-20 { (0.5 * (y0 - y2) / den).clamp(-0.5, 0.5) } else { 0.0 };
let delta = bl - margin;
let lag = coarse + delta;
Some(SyncResult { lag, lag_fine: lag as f64 + frac as f64, confidence, distinct, inverted })
}
#[cfg(test)]
mod tests {
use super::*;
use crate::testutil::Rng;
/// Speech-like test signal: voiced syllables (harmonics of a gliding pitch under an envelope),
/// noise bursts (consonants) and pauses.
fn speechy(sr: f64, secs: f64, seed: u64) -> Vec<f32> {
let n = (sr * secs) as usize;
let mut rng = Rng::new(seed);
let mut out = vec![0.0f32; n];
let mut i = 0usize;
while i < n {
let len = ((0.08 + 0.25 * (rng.uniform() * 0.5 + 0.5) as f64) * sr) as usize;
let kind = rng.uniform();
let f0 = 110.0 + 120.0 * (rng.uniform() as f64 * 0.5 + 0.5);
let glide = 1.0 + 0.3 * rng.uniform() as f64;
let amp = 0.2 + 0.3 * (rng.uniform() * 0.5 + 0.5);
for k in 0..len.min(n - i) {
let t = k as f64 / sr;
let env = (std::f64::consts::PI * k as f64 / len as f64).sin() as f32;
let v = if kind > 0.3 {
let f = f0 * (1.0 + (glide - 1.0) * k as f64 / len as f64);
(1..8).map(|h| ((2.0 * std::f64::consts::PI * f * h as f64 * t).sin() / h as f64) as f32).sum::<f32>()
} else if kind > -0.4 {
rng.uniform()
} else {
0.0
};
out[i + k] = v * env * amp;
}
i += len + ((rng.uniform() * 0.5 + 0.5) * 0.15 * sr as f32) as usize;
}
out
}
/// `x` delayed by `delay` samples (fractional: windowed-sinc interpolation), scaled by `gain`,
/// optionally coloured by a zero-phase low-pass (`[k, 1-2k, k]`, so the delay stays exact), with
/// white noise `snr_db` below the signal.
fn mic(x: &[f32], delay: f64, gain: f32, lowpass: Option<f32>, snr_db: f32, len: usize, seed: u64) -> Vec<f32> {
let mut out = vec![0.0f32; len];
let di = delay.floor() as i64;
let fr = delay - di as f64;
for (n, o) in out.iter_mut().enumerate() {
let src = n as i64 - di;
if fr == 0.0 {
if src >= 0 && (src as usize) < x.len() {
*o = x[src as usize];
}
} else {
let mut acc = 0.0f64;
for k in -16i64..=16 {
let j = src - k;
if j < 0 || j as usize >= x.len() {
continue;
}
let t = k as f64 - fr;
let s = (std::f64::consts::PI * t).sin() / (std::f64::consts::PI * t);
let w = 0.5 + 0.5 * (std::f64::consts::PI * t / 17.0).cos();
acc += x[j as usize] as f64 * s * w;
}
*o = acc as f32;
}
}
if let Some(k) = lowpass {
let src = out.clone();
for i in 1..len.saturating_sub(1) {
out[i] = k * src[i - 1] + (1.0 - 2.0 * k) * src[i] + k * src[i + 1];
}
}
let rms = crate::testutil::rms(&out) as f32;
let nrms = rms * 10f32.powf(-snr_db / 20.0) * 3f32.sqrt();
let mut rng = Rng::new(seed);
for v in &mut out {
*v = *v * gain + rng.uniform() * nrms * gain;
}
out
}
#[test]
fn decimate_keeps_low_frequencies_and_removes_high_ones() {
let sr = 48_000.0;
let low = crate::testutil::sine(300.0, 1.0, sr, 48_000, 0.0);
let high = crate::testutil::sine(7_000.0, 1.0, sr, 48_000, 0.0);
let dl = decimate(&low, 6);
let dh = decimate(&high, 6);
assert_eq!(dl.len(), 8000);
let rl = crate::testutil::rms(&dl[100..7900]);
let rh = crate::testutil::rms(&dh[100..7900]);
assert!((rl - std::f64::consts::FRAC_1_SQRT_2).abs() < 0.01, "{rl}");
assert!(rh < 0.01, "{rh}");
}
#[test]
fn pure_delay_exact_both_signs() {
let sr = 48_000;
let s = speechy(sr as f64, 12.0, 1);
for delay in [0i64, 1, 7, 12_345, 96_001, -4_321, -50_000] {
// b = s shifted: b[n] = s[n + delay]
let b: Vec<f32> = (0..400_000i64).map(|n| if n + delay >= 0 && ((n + delay) as usize) < s.len() { s[(n + delay) as usize] } else { 0.0 }).collect();
let r = find_offset(&s, &b, sr, &SyncOptions::default()).unwrap();
assert_eq!(r.lag, delay, "{r:?}");
assert!(r.reliable(), "{r:?}");
assert!(!r.inverted);
}
}
/// Three microphones recording one talker: different gains (−20…+10 dB), noise (SNR 25, 10 and
/// 3 dB), one coloured by a low-pass, fractional delays. Every offset within one sample.
#[test]
fn multi_mic_offsets_within_one_sample() {
let sr = 48_000u32;
let src = speechy(sr as f64, 30.0, 7);
let len = 30 * sr as usize;
// mic k records src delayed by d_k: mic_k[n] = src[n - d_k]
let cases: [(f64, f32, Option<f32>, f32); 4] =
[(0.0, 1.0, None, 40.0), (2_345.0, 0.1, None, 25.0), (-31_337.4, 3.0, Some(0.3), 10.0), (150_000.6, 0.5, None, 3.0)];
let mics: Vec<Vec<f32>> = cases.iter().enumerate().map(|(i, c)| mic(&src, c.0, c.1, c.2, c.3, len, 100 + i as u64)).collect();
let mut worst = 0.0f64;
let mut worst_fine = 0.0f64;
for (k, c) in cases.iter().enumerate().skip(1) {
// b = mic_k: b[n] = src[n - d_k] = a[n - d_k + d_0] → lag = d_0 - d_k
let expect = cases[0].0 - c.0;
let r = find_offset(&mics[0], &mics[k], sr, &SyncOptions::default()).unwrap();
let err = (r.lag as f64 - expect).abs();
let err_fine = (r.lag_fine - expect).abs();
worst = worst.max(err);
worst_fine = worst_fine.max(err_fine);
assert!(err <= 1.0, "mic {k}: lag {} vs {expect} ({r:?})", r.lag);
assert!(err_fine < 0.75, "mic {k}: fine {} vs {expect}", r.lag_fine);
assert!(r.reliable(), "{r:?}");
}
eprintln!("multi-mic worst error: {worst} samples (integer lag), {worst_fine:.3} samples (sub-sample estimate)");
}
/// A room: the far microphone hears the direct sound plus a strong reflection 23 ms later, at
/// 0 dB SNR. The direct path wins.
#[test]
fn echo_and_heavy_noise() {
let sr = 48_000u32;
let src = speechy(sr as f64, 20.0, 21);
let len = src.len();
let near = mic(&src, 0.0, 1.0, None, 35.0, len, 1);
let direct = mic(&src, 777.0, 1.0, None, 60.0, len, 2);
let echo = mic(&src, 777.0 + 0.023 * sr as f64, 0.6, None, 60.0, len, 3);
let mut rng = Rng::new(9);
let room: Vec<f32> = direct.iter().zip(&echo).map(|(a, b)| a + b).collect();
let nrms = crate::testutil::rms(&room) as f32 * 3f32.sqrt();
let far: Vec<f32> = room.iter().map(|v| (v + rng.uniform() * nrms) * 0.3).collect();
let r = find_offset(&near, &far, sr, &SyncOptions::default()).unwrap();
assert!((r.lag + 777).abs() <= 1, "{r:?}");
assert!(r.reliable(), "{r:?}");
}
/// Two 10-minute recordings (run with `--ignored --nocapture` in release for the timing).
#[test]
#[ignore]
fn long_recordings_timing() {
let sr = 48_000u32;
let src = speechy(sr as f64, 600.0, 31);
let a = mic(&src, 0.0, 1.0, None, 20.0, src.len(), 1);
let b = mic(&src, -1_234_567.0, 0.5, None, 10.0, src.len() - 2_000_000, 2);
let t = std::time::Instant::now();
let r = find_offset(&a, &b, sr, &SyncOptions::default()).unwrap();
eprintln!("10 min + 9.3 min at 48 kHz: {:?} in {:.2} s", r, t.elapsed().as_secs_f64());
assert!((r.lag - 1_234_567).abs() <= 1);
}
#[test]
fn inverted_polarity_is_found() {
let sr = 48_000;
let s = speechy(sr as f64, 8.0, 3);
let b: Vec<f32> = (0..300_000usize).map(|n| if n + 1000 < s.len() { -0.5 * s[n + 1000] } else { 0.0 }).collect();
let r = find_offset(&s, &b, sr, &SyncOptions::default()).unwrap();
assert_eq!(r.lag, 1000);
assert!(r.inverted);
}
#[test]
fn unrelated_signals_are_not_reliable() {
let sr = 48_000;
let a = speechy(sr as f64, 10.0, 11);
let b = speechy(sr as f64, 10.0, 12);
let r = find_offset(&a, &b, sr, &SyncOptions::default()).unwrap();
assert!(!r.reliable(), "{r:?}");
assert!(find_offset(&a, &[0.0; 48_000], sr, &SyncOptions::default()).is_none());
assert!(find_offset(&a[..100], &a, sr, &SyncOptions::default()).is_none());
}
#[test]
fn max_lag_limits_the_search() {
let sr = 48_000;
let s = speechy(sr as f64, 10.0, 5);
let b: Vec<f32> = s[48_000..].to_vec();
let r = find_offset(&s, &b, sr, &SyncOptions { max_lag: Some(60_000), ..Default::default() }).unwrap();
assert_eq!(r.lag, 48_000);
let r = find_offset(&s, &b, sr, &SyncOptions { max_lag: Some(10_000), ..Default::default() }).unwrap();
assert!(!r.reliable() || r.lag != 48_000);
}
}

View File

@ -16,6 +16,7 @@
//! Keyframes are stored in media time, so trims and splits never need to move them. //! Keyframes are stored in media time, so trims and splits never need to move them.
pub mod captions; pub mod captions;
pub mod multicam;
use std::collections::HashMap; use std::collections::HashMap;

213
crates/edit/src/multicam.rs Normal file
View File

@ -0,0 +1,213 @@
//! Multi-camera editing: switching angles, recording cuts live, healing through edits and
//! flattening multi-camera clips into the clips they show.
//!
//! A multi-camera clip is a track item whose [`TrackItem::multicam`] is set; it refers to a
//! multi-camera source sequence. Its angle is a camera index of that source.
use std::collections::HashMap;
use filmcraft_project::{ClipId, ItemId, MulticamSel, Sequence, Track, TrackId, TrackItem};
use filmcraft_time::{Tick, TimeRange};
use crate::{EditCtx, EditError, Result, remove_orphan_transitions, split_track_at};
/// Whether `it` is a multi-camera clip (enabled or not).
pub fn is_multicam(it: &TrackItem) -> bool {
it.multicam.is_some()
}
/// Set the angle of multi-camera clips. Returns how many changed.
pub fn set_angle(seq: &mut Sequence, clips: &[ClipId], angle: u32) -> usize {
let mut n = 0;
for t in seq.all_tracks_mut() {
if t.locked {
continue;
}
for it in t.items.iter_mut().filter(|i| clips.contains(&i.id)) {
if let Some(m) = &mut it.multicam
&& m.angle != angle
{
m.angle = angle;
n += 1;
}
}
}
n
}
/// Enable or disable multi-camera behaviour of clips (Multi-Camera ▸ Enable). Clips that are not
/// multi-camera clips are left alone. Returns how many changed.
pub fn set_enabled(seq: &mut Sequence, clips: &[ClipId], enabled: bool) -> usize {
let mut n = 0;
for t in seq.all_tracks_mut() {
for it in t.items.iter_mut().filter(|i| clips.contains(&i.id)) {
if let Some(m) = &mut it.multicam
&& m.enabled != enabled
{
m.enabled = enabled;
n += 1;
}
}
}
n
}
/// Two adjacent pieces of one multi-camera clip that a cut between them would not change.
fn continuous(a: &TrackItem, b: &TrackItem) -> bool {
a.end() == b.start
&& a.item == b.item
&& a.multicam.is_some()
&& a.multicam == b.multicam
&& a.speed == b.speed
&& !a.reverse
&& !b.reverse
&& a.frame_hold.is_none()
&& b.frame_hold.is_none()
&& a.source_out() == b.source_in
&& a.enabled == b.enabled
&& a.gain_db == b.gain_db
&& a.effects == b.effects
}
/// Remove through edits between pieces of the same multi-camera clip showing the same angle,
/// for edit points in `range` (inclusive of its ends). Returns the ids of the removed pieces.
pub fn heal_track(track: &mut Track, range: TimeRange) -> Vec<ClipId> {
let mut removed = Vec::new();
let mut i = 0;
while i + 1 < track.items.len() {
let (a, b) = (&track.items[i], &track.items[i + 1]);
let at = b.start;
if at >= range.start && at <= range.end() && continuous(a, b) {
let (aid, bid) = (a.id, b.id);
let dur = b.duration;
track.items[i].duration += dur;
track.items.remove(i + 1);
track.transitions.retain(|tr| !(tr.from == Some(aid) && tr.to == Some(bid)));
for tr in &mut track.transitions {
if tr.from == Some(bid) {
tr.from = Some(aid);
}
if tr.to == Some(bid) {
tr.to = Some(aid);
}
}
removed.push(bid);
} else {
i += 1;
}
}
remove_orphan_transitions(track);
removed
}
/// One live switch: from `time` on, show `angle`.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct Cut {
pub time: Tick,
pub angle: u32,
}
/// Apply a recording pass: on `tracks`, the multi-camera clips of `source` get angle `cuts[i].angle` from
/// `cuts[i].time` until the next cut, the last one until `end`. Clips are split at each cut and at
/// `end`; afterwards through edits inside the recorded range are healed, so pressing the angle that
/// is already showing adds no edit. Locked tracks are skipped. Returns the number of edit points
/// in the recorded range after the pass.
pub fn record(seq: &mut Sequence, tracks: &[TrackId], source: ItemId, cuts: &[Cut], end: Tick, ctx: &mut EditCtx) -> usize {
let mut cuts: Vec<Cut> = cuts.iter().copied().filter(|c| c.time < end).collect();
cuts.sort_by_key(|c| c.time);
// two presses at one time: the later wins
let mut dedup: Vec<Cut> = Vec::new();
for c in cuts {
match dedup.last_mut() {
Some(l) if l.time == c.time => *l = c,
_ => dedup.push(c),
}
}
let Some(first) = dedup.first().map(|c| c.time) else { return 0 };
let mut links = HashMap::new();
let mut points = 0;
for tr in seq.all_tracks_mut() {
if tr.locked || !tracks.contains(&tr.id) {
continue;
}
let mc_at = |tr: &Track, t: Tick| tr.items.iter().any(|i| i.start < t && t < i.end() && is_multicam(i) && i.item == source);
for t in dedup.iter().map(|c| c.time).chain([end]) {
if mc_at(tr, t) {
split_track_at(tr, t, ctx, &mut links);
}
}
for (k, c) in dedup.iter().enumerate() {
let r_end = dedup.get(k + 1).map_or(end, |n| n.time);
for it in tr.items.iter_mut().filter(|i| i.start >= c.time && i.end() <= r_end && i.item == source) {
if let Some(m) = &mut it.multicam {
m.angle = c.angle;
}
}
}
heal_track(tr, TimeRange::from_bounds(first, end));
points += tr.items.iter().filter(|i| i.start > first && i.start < end).count();
}
points
}
/// The clips a multi-camera (or any nested) clip shows, as track items of the outer sequence:
/// the parts of `source_track`'s items inside the clip's source range, moved to timeline time.
/// Inner links are remapped through `links` (so a flattened video/audio pair stays linked).
/// Fails for clips with a speed change, reverse or frame hold (no exact flattening).
pub fn flatten_items(outer: &TrackItem, source_track: &Track, ctx: &mut EditCtx, links: &mut HashMap<u64, u64>) -> Result<Vec<TrackItem>> {
if (outer.speed - 1.0).abs() > 1e-9 || outer.reverse || outer.frame_hold.is_some() {
return Err(EditError::Other("can't flatten a clip with a speed change, reverse or frame hold".into()));
}
let (s0, s1) = (outer.source_in, outer.source_out());
let mut out = Vec::new();
for ni in source_track.items.iter().filter(|i| i.start < s1 && i.end() > s0) {
let mut it = ni.clone();
if it.start < s0 {
let cut = s0 - it.start;
it.source_in += Tick((cut.0 as f64 * it.speed.abs()).round() as i64);
it.duration -= cut;
it.start = s0;
}
if it.end() > s1 {
it.duration = s1 - it.start;
}
if it.duration <= Tick::ZERO {
continue;
}
it.start = outer.start + (it.start - s0);
it.id = ClipId(ctx.alloc());
it.link = it.link.map(|l| *links.entry(l).or_insert_with(|| ctx.alloc()));
it.group = None;
// the outer clip's look carries over: its standard effects follow the inner ones, and its
// Motion/Opacity/Volume replace the inner ones when only the outer clip changed them
let take_intrinsics = outer.has_modified_intrinsics() && !it.has_modified_intrinsics();
for e in &outer.effects {
let intrinsic = e.def().is_some_and(|d| d.intrinsic);
if !intrinsic {
it.effects.push(e.clone());
} else if take_intrinsics && let Some(slot) = it.effects.iter_mut().find(|x| x.effect == e.effect) {
*slot = e.clone();
}
}
out.push(it);
}
Ok(out)
}
/// Replace a multi-camera clip on `track` by `pieces` (from [`flatten_items`]); pieces whose
/// track is not `track` go to the tracks given with them. Returns the new ids.
pub fn replace_with(seq: &mut Sequence, clip: ClipId, pieces: Vec<(TrackId, TrackItem)>, ctx: &mut EditCtx) -> Result<Vec<ClipId>> {
let (tid, _) = seq.find_item(clip).ok_or(EditError::NoItem(clip))?;
let t = seq.track_mut(tid).ok_or(EditError::NoTrack(tid))?;
if t.locked {
return Err(EditError::Locked);
}
t.items.retain(|i| i.id != clip);
remove_orphan_transitions(t);
crate::overwrite(seq, pieces, ctx)
}
/// The multi-camera selection a freshly edited multi-camera clip gets.
pub fn default_sel(angle: u32) -> MulticamSel {
MulticamSel { enabled: true, angle }
}

View File

@ -55,6 +55,7 @@ impl Fx {
frame_hold: None, frame_hold: None,
scale_to_frame: false, scale_to_frame: false,
essential: None, essential: None,
multicam: None,
} }
} }
fn put(&mut self, track: TrackId, start: i64, dur: i64, src_in: i64) -> ClipId { fn put(&mut self, track: TrackId, start: i64, dur: i64, src_in: i64) -> ClipId {

View File

@ -75,7 +75,7 @@ fn has_recovery(s: &Session) -> std::result::Result<(), String> {
pub(crate) fn has_seq(s: &Session) -> std::result::Result<(), String> { pub(crate) fn has_seq(s: &Session) -> std::result::Result<(), String> {
s.active_sequence().map(|_| ()).ok_or_else(|| "no sequence is open".into()) s.active_sequence().map(|_| ()).ok_or_else(|| "no sequence is open".into())
} }
fn has_selection(s: &Session) -> std::result::Result<(), String> { pub(crate) fn has_selection(s: &Session) -> std::result::Result<(), String> {
has_seq(s)?; has_seq(s)?;
if s.state.selection.is_empty() { Err("no clips selected".into()) } else { Ok(()) } if s.state.selection.is_empty() { Err("no clips selected".into()) } else { Ok(()) }
} }
@ -102,7 +102,7 @@ fn has_previews(s: &Session) -> std::result::Result<(), String> {
has_seq(s)?; has_seq(s)?;
if s.previews.count() == 0 { Err("there are no render files".into()) } else { Ok(()) } if s.previews.count() == 0 { Err("there are no render files".into()) } else { Ok(()) }
} }
fn has_project_selection(s: &Session) -> std::result::Result<(), String> { pub(crate) fn has_project_selection(s: &Session) -> std::result::Result<(), String> {
if s.state.project_selection.is_empty() { Err("select an item in the Project panel".into()) } else { Ok(()) } if s.state.project_selection.is_empty() { Err("select an item in the Project panel".into()) } else { Ok(()) }
} }
fn has_clipboard(s: &Session) -> std::result::Result<(), String> { fn has_clipboard(s: &Session) -> std::result::Result<(), String> {
@ -152,13 +152,13 @@ pub(crate) fn time_p(s: &Session, p: &Value, prefix: &str) -> Option<Tick> {
pub(crate) fn clip_p(p: &Value, k: &str) -> Option<ClipId> { pub(crate) fn clip_p(p: &Value, k: &str) -> Option<ClipId> {
u64_p(p, k).map(ClipId) u64_p(p, k).map(ClipId)
} }
fn clips_p(s: &Session, p: &Value) -> Vec<ClipId> { pub(crate) fn clips_p(s: &Session, p: &Value) -> Vec<ClipId> {
match p.get("clips").and_then(Value::as_array) { match p.get("clips").and_then(Value::as_array) {
Some(a) => a.iter().filter_map(|v| v.as_u64().map(ClipId)).collect(), Some(a) => a.iter().filter_map(|v| v.as_u64().map(ClipId)).collect(),
None => clip_p(p, "clip").map(|c| vec![c]).unwrap_or_else(|| s.state.selection.clone()), None => clip_p(p, "clip").map(|c| vec![c]).unwrap_or_else(|| s.state.selection.clone()),
} }
} }
fn track_p(s: &Session, p: &Value, k: &str) -> Option<TrackId> { pub(crate) fn track_p(s: &Session, p: &Value, k: &str) -> Option<TrackId> {
let v = p.get(k)?; let v = p.get(k)?;
if let Some(id) = v.as_u64() { if let Some(id) = v.as_u64() {
return Some(TrackId(id)); return Some(TrackId(id));
@ -192,7 +192,7 @@ pub fn with_links(s: &Session, clips: &[ClipId]) -> Vec<ClipId> {
out out
} }
fn default_seq_settings_for(info: &filmcraft_media::MediaInfo) -> SequenceSettings { pub(crate) fn default_seq_settings_for(info: &filmcraft_media::MediaInfo) -> SequenceSettings {
let mut st = SequenceSettings::default(); let mut st = SequenceSettings::default();
if let Some(v) = &info.video { if let Some(v) = &info.video {
st.width = v.width; st.width = v.width;
@ -1876,6 +1876,7 @@ fn build() -> Vec<CommandSpec> {
]; ];
v.extend(crate::captions::commands()); v.extend(crate::captions::commands());
v.extend(crate::mixer::commands()); v.extend(crate::mixer::commands());
v.extend(crate::multicam::commands());
v.extend(crate::essential_sound::commands()); v.extend(crate::essential_sound::commands());
v.extend(crate::color::commands()); v.extend(crate::color::commands());
v.extend(crate::graphics::commands()); v.extend(crate::graphics::commands());

View File

@ -20,12 +20,14 @@ pub mod masks;
pub mod media_pool; pub mod media_pool;
pub mod mixer; pub mod mixer;
pub mod presets; pub mod presets;
pub mod multicam;
pub mod previews; pub mod previews;
pub mod project_manager; pub mod project_manager;
pub mod proxies; pub mod proxies;
pub mod relink; pub mod relink;
pub mod shortcut_presets; pub mod shortcut_presets;
pub mod shortcuts; pub mod shortcuts;
pub mod sync;
pub mod trim; pub mod trim;
use std::sync::Arc; use std::sync::Arc;
@ -173,6 +175,10 @@ pub struct EditorState {
/// The mask being edited on the Program monitor (Effect Controls selection). /// The mask being edited on the Program monitor (Effect Controls selection).
#[serde(default)] #[serde(default)]
pub selected_mask: Option<masks::MaskSel>, pub selected_mask: Option<masks::MaskSel>,
/// Multi-Camera Audio Follows Video: switching a multi-camera clip's angle switches its linked
/// audio clips too.
#[serde(default)]
pub multicam_audio_follows_video: bool,
} }
/// Events for frontends (drained each frame). /// Events for frontends (drained each frame).
@ -210,6 +216,8 @@ pub struct Session {
pub previews: Arc<previews::PreviewStore>, pub previews: Arc<previews::PreviewStore>,
/// Audio Track Mixer automation pass in progress. /// Audio Track Mixer automation pass in progress.
pub mixrec: mixer::Recorder, pub mixrec: mixer::Recorder,
/// Multi-camera live switching pass in progress.
pub mcrec: multicam::Recorder,
/// Dynamic (J/K/L) trimming and trim-mode loop playback in progress. /// Dynamic (J/K/L) trimming and trim-mode loop playback in progress.
pub trim_play: trim::TrimPlayback, pub trim_play: trim::TrimPlayback,
/// Keyboard shortcuts (active bindings, presets; `shortcuts.*` commands). /// Keyboard shortcuts (active bindings, presets; `shortcuts.*` commands).
@ -284,6 +292,7 @@ impl Session {
loaded_schema: filmcraft_format::SCHEMA_VERSION, loaded_schema: filmcraft_format::SCHEMA_VERSION,
previews: Arc::new(previews::PreviewStore::temp()), previews: Arc::new(previews::PreviewStore::temp()),
mixrec: Default::default(), mixrec: Default::default(),
mcrec: Default::default(),
trim_play: Default::default(), trim_play: Default::default(),
shortcuts: shortcuts::Shortcuts::new(), shortcuts: shortcuts::Shortcuts::new(),
offline: Default::default(), offline: Default::default(),
@ -649,6 +658,8 @@ mod mixer_tests;
#[cfg(test)] #[cfg(test)]
mod presets_tests; mod presets_tests;
#[cfg(test)] #[cfg(test)]
mod multicam_tests;
#[cfg(test)]
mod previews_tests; mod previews_tests;
#[cfg(test)] #[cfg(test)]
mod project_manager_tests; mod project_manager_tests;

View File

@ -0,0 +1,968 @@
//! Multi-camera editing and clip synchronisation commands.
//!
//! | Id | Menu | |
//! |---|---|---|
//! | `clip.synchronize` | Clip ▸ Synchronize… | align selected timeline clips to a reference by In, Out, timecode, clip marker or audio |
//! | `clip.mergeClips` | Clip ▸ Merge Clips… | one video + audio clips → a merged clip |
//! | `clip.createMulticam` | Clip ▸ Create Multi-Camera Source Sequence… | synchronised cameras → a multi-camera source sequence |
//! | `clip.multicamEnable` | Clip ▸ Multi-Camera ▸ Enable | toggle multi-camera behaviour of nested clips |
//! | `clip.multicamFlatten` | Clip ▸ Multi-Camera ▸ Flatten | replace multi-camera clips by the clips their angle shows |
//! | `multicam.switchAngle` | | set the angle of clips (keys 1–9 when stopped, clicking an angle) |
//! | `multicam.recordStart` / `multicam.cut` / `multicam.recordStop` | | live switching while playing: one undo step per pass |
//! | `multicam.selectCamera1`…`9` | keys 1–9 | while recording: cut to camera N; stopped: switch the clip at the playhead |
//! | `multicam.cutToCamera1`…`9` | Ctrl+1–9 | add an edit at the playhead and show camera N from there |
//! | `multicam.audioFollowsVideo` | | switching video switches the linked audio clips too |
//! | `multicam.editCameras` | | rename / show / hide cameras, change the audio mode |
//! | `multicam.inspect` | | the multi-camera clip at the playhead, its cameras and the recorder state |
//!
//! Sync methods are described in [`crate::sync`].
use std::collections::HashMap;
use std::sync::Arc;
use filmcraft_edit as edit;
use filmcraft_edit::EditCtx;
use filmcraft_edit::multicam::Cut;
use filmcraft_project::{
BinEntry, Camera, ClipId, ItemId, ItemKind, Label, MergedClip, MulticamAudio, MulticamSel, MulticamSource, Project, Sequence, SequenceSettings, Track,
TrackId, TrackItem, TrackKind, resolve_auto_points,
};
use filmcraft_time::{FrameRate, Tick, TimeRange};
use serde_json::{Value, json};
use crate::commands::{CommandSpec, always, bad, bool_p, clips_p, has_selection, has_seq, item_p, str_p, time_p, track_p, u64_p, with_links};
use crate::sync::{self, Method, SyncClip};
use crate::{EngineError, MediaPool, Result, Session};
type Run = fn(&mut Session, &Value) -> Result<Value>;
type Enabled = fn(&Session) -> std::result::Result<(), String>;
#[allow(clippy::too_many_arguments)]
fn spec(id: &'static str, label: &'static str, menu: &'static [&'static str], params: &'static str, enabled: Enabled, run: Run, journal: bool) -> CommandSpec {
CommandSpec { id, label, menu, shortcut: None, params, enabled, run, journal }
}
macro_rules! camera_keys {
($($n:literal),*) => {
[$(
CommandSpec {
id: concat!("multicam.selectCamera", $n),
label: concat!("Select Camera ", $n),
menu: &[],
shortcut: Some(concat!($n)),
params: r#"{"videoOnly":bool?}"#,
enabled: has_seq,
run: |s, p| cut(s, &with_camera(p, $n)),
journal: true,
},
CommandSpec {
id: concat!("multicam.cutToCamera", $n),
label: concat!("Cut to Camera ", $n),
menu: &[],
shortcut: Some(concat!("Ctrl+", $n)),
params: r#"{"videoOnly":bool?}"#,
enabled: has_seq,
run: |s, p| cut_to_camera(s, &with_camera(p, $n)),
journal: true,
},
)*]
};
}
fn with_camera(p: &Value, n: u64) -> Value {
let mut v = if p.is_object() { p.clone() } else { json!({}) };
v["camera"] = json!(n);
v
}
pub fn commands() -> Vec<CommandSpec> {
let mut v = base_commands();
v.extend(camera_keys!(1, 2, 3, 4, 5, 6, 7, 8, 9));
v
}
fn base_commands() -> Vec<CommandSpec> {
vec![
spec(
"clip.synchronize",
"Synchronize…",
&["Clip"],
r#"{"method":"in|out|timecode|marker|audio","clips":[id]?,"reference":clip?,"track":"V1"|"A1"?,"ignoreHours":bool?,"marker":str?,"offset":frames?}"#,
can_synchronize,
synchronize,
true,
),
spec(
"clip.mergeClips",
"Merge Clips…",
&["Clip"],
r#"{"items":[id]?,"method":"in|out|timecode|marker|audio","name":str?,"removeVideoAudio":bool?,"ignoreHours":bool?,"marker":str?,"offset":frames?}"#,
has_two_items,
merge_clips,
true,
),
spec(
"clip.createMulticam",
"Create Multi-Camera Source Sequence…",
&["Clip"],
r#"{"items":[id]?,"name":str?,"method":"in|out|timecode|marker|audio","ignoreHours":bool?,"marker":str?,"offset":frames?,"audio":"camera1|all|switch","cameraNames":"clip|track|metadata","processedBin":bool?,"reference":item?}"#,
has_items,
create_multicam,
true,
),
spec("clip.multicamEnable", "Enable", &["Clip", "Multi-Camera"], r#"{"clips":[id]?,"enabled":bool?}"#, has_selection, enable, true),
spec("clip.multicamFlatten", "Flatten", &["Clip", "Multi-Camera"], r#"{"clips":[id]?}"#, has_selection, flatten, true),
spec(
"multicam.switchAngle",
"Switch Multi-Camera Angle",
&[],
r#"{"camera":1..16 (shown order)|"angle":0-based,"clips":[id]?,"time":ticks?,"videoOnly":bool?,"audioOnly":bool?}"#,
has_seq,
switch_angle,
true,
),
spec("multicam.recordStart", "Start Multi-Camera Recording", &[], r#"{"time":ticks?}"#, has_seq, record_start, false),
spec("multicam.cut", "Cut to Camera", &[], r#"{"camera":1..16|"angle":0-based,"time":ticks?,"videoOnly":bool?}"#, has_seq, cut, true),
spec("multicam.recordStop", "Stop Multi-Camera Recording", &[], r#"{"time":ticks?}"#, always, record_stop, true),
spec("multicam.audioFollowsVideo", "Multi-Camera Audio Follows Video", &[], r#"{"enabled":bool?}"#, always, audio_follows, true),
spec(
"multicam.editCameras",
"Edit Cameras…",
&[],
r#"{"sequence":id?,"cameras":[{"angle":0-based,"name":str?,"enabled":bool?}]?,"audio":"camera1|all|switch"?}"#,
always,
edit_cameras,
true,
),
spec("multicam.cutToCamera", "Cut to Camera", &[], r#"{"camera":1..16|"angle":0-based,"time":ticks?,"videoOnly":bool?}"#, has_seq, cut_to_camera, true),
CommandSpec {
id: "multicam.inspect",
label: "Inspect Multi-Camera",
menu: &[],
shortcut: None,
params: r#"{"time":ticks?}"#,
enabled: always,
run: inspect,
journal: false,
},
]
}
// ---------------------------------------------------------------- enablement
fn can_synchronize(s: &Session) -> std::result::Result<(), String> {
has_seq(s)?;
let sel = with_links(s, &s.state.selection);
if sel.len() < 2 { Err("select at least two clips to synchronize".into()) } else { Ok(()) }
}
fn has_items(s: &Session) -> std::result::Result<(), String> {
if s.state.project_selection.iter().any(|i| sync::project_clip(&s.project, *i).is_some()) {
Ok(())
} else {
Err("select clips in the Project panel".into())
}
}
fn has_two_items(s: &Session) -> std::result::Result<(), String> {
if s.state.project_selection.iter().filter(|i| sync::project_clip(&s.project, **i).is_some()).count() >= 2 {
Ok(())
} else {
Err("select a video clip and audio clips in the Project panel".into())
}
}
// ---------------------------------------------------------------- helpers
fn items_p(s: &Session, p: &Value) -> Vec<ItemId> {
match p.get("items").and_then(Value::as_array) {
Some(a) => a.iter().filter_map(|v| v.as_u64().map(ItemId)).collect(),
None => s.state.project_selection.clone(),
}
}
fn method_p(p: &Value, cmd: &str) -> Result<Method> {
Method::from_params(p).map_err(|e| bad(cmd, e))
}
fn offset_p(p: &Value, rate: FrameRate) -> Tick {
p.get("offset").and_then(Value::as_i64).map(|f| rate.tick_of(f)).unwrap_or(Tick::ZERO)
}
/// Run `f` on sequence `seq_id` of `p` with an edit context (ids, media durations), then check it.
fn seq_edit<R>(p: &mut Project, seq_id: ItemId, media: &Arc<MediaPool>, f: impl FnOnce(&mut Sequence, &mut EditCtx) -> Result<R>) -> Result<R> {
let snapshot = p.clone();
let media = media.clone();
let durations = move |id: ItemId| crate::media_duration(&snapshot, &media, id);
let min = p.sequence(seq_id).map(|s| s.settings.frame_rate.frame_duration()).unwrap_or(Tick(1));
let mut next = p.next_id;
let r = {
let seq = p.sequence_mut(seq_id).ok_or(EngineError::NoSequence)?;
let mut ctx = EditCtx { next_id: &mut next, media_duration: &durations, min_duration: min };
f(seq, &mut ctx)?
};
p.next_id = next;
if let Some(q) = p.sequence(seq_id) {
q.check().map_err(EngineError::Other)?;
}
Ok(r)
}
/// Frame-aligned placement that keeps sub-frame (sample) accuracy: a clip whose media range should
/// start at the exact timeline time `exact` starts at the next frame boundary instead, with its
/// source In moved by the difference. Returns (start, source In shift).
fn frame_align(exact: Tick, rate: FrameRate) -> (Tick, Tick) {
let snapped = rate.snap(exact);
let start = if snapped < exact { snapped + rate.frame_duration() } else { snapped };
(start, start - exact)
}
/// A track item showing `range` of `item` whose media time `range.start` sits at timeline `exact`.
fn placed_item(p: &mut Project, item: ItemId, kind: TrackKind, exact: Tick, range: TimeRange, settings: &SequenceSettings) -> Option<TrackItem> {
let rate = settings.frame_rate;
let (start, shift) = frame_align(exact, rate);
let src = TimeRange::from_bounds(range.start + shift, range.end());
let mut ti = p.make_track_item(item, kind, start, src, rate)?;
// whole frames that fit in the media
let frames = rate.frame_at(src.duration).max(1);
ti.duration = rate.tick_of(frames);
if kind == TrackKind::Video {
let size = sync_source_size(p, item).unwrap_or((settings.width, settings.height));
for e in &mut ti.effects {
resolve_auto_points(e, (settings.width, settings.height), size);
}
}
Some(ti)
}
fn sync_source_size(p: &Project, item: ItemId) -> Option<(u32, u32)> {
match &p.item(item)?.kind {
ItemKind::Media(m) => m.info.video.as_ref().map(|v| (v.width, v.height)),
ItemKind::Subclip { parent, .. } => sync_source_size(p, *parent),
_ => None,
}
}
// ---------------------------------------------------------------- Synchronize
/// A set of linked timeline clips that move together, with the clip used to sync it.
struct Unit {
members: Vec<ClipId>,
rep: TrackItem,
rep_track: (TrackKind, usize),
}
fn synchronize(s: &mut Session, p: &Value) -> Result<Value> {
let cmd = "clip.synchronize";
let method = method_p(p, cmd)?;
let seq_id = s.state.active_sequence.ok_or(EngineError::NoSequence)?;
let q = s.active_sequence().ok_or(EngineError::NoSequence)?.clone();
let rate = q.settings.frame_rate;
let sel = with_links(s, &clips_p(s, p));
// group into link units
let mut units: Vec<Unit> = Vec::new();
let mut seen: Vec<ClipId> = Vec::new();
let pos = |c: ClipId| -> Option<(TrackKind, usize, TrackItem)> {
for (k, ts) in [(TrackKind::Video, &q.video_tracks), (TrackKind::Audio, &q.audio_tracks)] {
for (ti, t) in ts.iter().enumerate() {
if let Some(it) = t.item(c) {
return Some((k, ti, it.clone()));
}
}
}
None
};
for &c in &sel {
if seen.contains(&c) {
continue;
}
let Some((_, _, it)) = pos(c) else { continue };
let members: Vec<ClipId> = match it.link {
Some(l) => sel.iter().copied().filter(|o| pos(*o).is_some_and(|x| x.2.link == Some(l))).collect(),
None => vec![c],
};
seen.extend(&members);
let mut cands: Vec<(TrackKind, usize, TrackItem)> = members.iter().filter_map(|m| pos(*m)).collect();
// the representative: audio for audio sync, else video; lowest track first
let audio_first = method == Method::Audio;
cands.sort_by_key(|(k, ti, _)| (if (*k == TrackKind::Audio) == audio_first { 0 } else { 1 }, *ti));
let (k, ti, rep) = cands.remove(0);
units.push(Unit { members, rep, rep_track: (k, ti) });
}
if units.len() < 2 {
return Err(bad(cmd, "select clips from at least two different sources"));
}
// reference unit
let reference = if let Some(c) = u64_p(p, "reference").map(ClipId) {
units.iter().position(|u| u.members.contains(&c)).ok_or_else(|| bad(cmd, "`reference` is not among the clips"))?
} else if let Some(t) = track_p(s, p, "track") {
units
.iter()
.position(|u| u.members.iter().any(|m| q.track(t).is_some_and(|tr| tr.item(*m).is_some())))
.ok_or_else(|| bad(cmd, "no selected clip on the reference track"))?
} else {
(0..units.len()).min_by_key(|&i| (if units[i].rep_track.0 == TrackKind::Video { 0 } else { 1 }, units[i].rep_track.1)).unwrap_or(0)
};
let clips: Vec<SyncClip> = units.iter().map(|u| sync::timeline_clip(&s.project, &u.rep)).collect();
let anchors = sync::anchors(s, &clips, reference, &method, offset_p(p, rate))?;
// timeline time of each unit's anchor must equal the reference's
let at = |u: &Unit, a: Tick| u.rep.start + Tick(((a - u.rep.source_in).0 as f64 / u.rep.speed.abs().max(1e-9)).round() as i64);
let t_ref = at(&units[reference], anchors[reference].anchor);
let mut deltas: Vec<Tick> = units.iter().zip(&anchors).map(|(u, a)| t_ref - at(u, a.anchor)).collect();
// nothing may move before the sequence start: shift everything (reference too) right
let min_start = units
.iter()
.zip(&deltas)
.flat_map(|(u, d)| u.members.iter().filter_map(|m| pos(*m)).map(move |x| x.2.start + *d).collect::<Vec<_>>())
.min()
.unwrap_or(Tick::ZERO);
if min_start < Tick::ZERO {
deltas.iter_mut().for_each(|d| *d -= min_start);
}
let mut moved = 0usize;
let mut placements: Vec<(TrackId, TrackItem)> = Vec::new();
let mut remove: Vec<ClipId> = Vec::new();
for (u, d) in units.iter().zip(&deltas) {
if *d == Tick::ZERO {
continue;
}
moved += 1;
// frame-aligned start, sub-frame remainder taken from the source In
let (start, shift) = frame_align(u.rep.start + *d, rate);
let dd = start - u.rep.start;
for m in &u.members {
let Some((tid, it)) = q.find_item(*m) else { continue };
let mut it = it.clone();
it.start += dd;
if shift > Tick::ZERO {
it.source_in += Tick((shift.0 as f64 * it.speed.abs()).round() as i64);
it.duration = (it.duration - rate.frame_duration()).max(rate.frame_duration());
}
remove.push(*m);
placements.push((tid, it));
}
}
let starts: Vec<Tick> = units.iter().zip(&deltas).map(|(u, d)| u.rep.start + *d).collect();
let rep = sync::report(&clips, &anchors, &starts, rate);
if moved > 0 {
let media = s.media.clone();
s.edit(&format!("Synchronize ({})", method.name()), |pr, st| {
seq_edit(pr, seq_id, &media, |seq, ctx| {
edit::delete_items(seq, &remove);
let ids = edit::overwrite(seq, placements, ctx)?;
Ok(ids)
})
.map(|ids| {
st.selection = ids;
})
})?;
}
Ok(json!({"moved": moved, "reference": reference, "clips": rep}))
}
// ---------------------------------------------------------------- Merge Clips / Create Multi-Camera
fn camera_name(p: &Project, item: ItemId, mode: &str, n: usize) -> String {
let it = p.item(item);
let clip = || it.map(|i| i.name.clone()).unwrap_or_else(|| format!("Camera {n}"));
match mode {
"track" | "tracks" | "trackname" | "trackNames" | "enumerate" => format!("Camera {n}"),
"metadata" | "cameraAngle" | "label" => it
.and_then(|i| {
["Camera Angle", "Camera Label", "Camera", "camera", "cameraAngle"].iter().find_map(|k| i.metadata.get(*k).filter(|v| !v.is_empty()).cloned())
})
.unwrap_or_else(clip),
_ => clip(),
}
}
/// Resolve and order the clips for Merge Clips / Create Multi-Camera.
fn sync_items(s: &Session, items: &[ItemId], cmd: &str) -> Result<Vec<SyncClip>> {
let mut out = Vec::new();
for i in items {
let c = sync::project_clip(&s.project, *i).ok_or_else(|| bad(cmd, format!("item {} is not a clip", i.0)))?;
if !out.iter().any(|o: &SyncClip| o.item == c.item) {
out.push(c);
}
}
if out.is_empty() {
return Err(bad(cmd, "no clips"));
}
Ok(out)
}
fn settings_for(s: &Session, item: ItemId) -> SequenceSettings {
let info = match s.project.item(item).map(|i| &i.kind) {
Some(ItemKind::Media(m)) => Some(m.info.clone()),
Some(ItemKind::Subclip { parent, .. }) => s.project.item(*parent).and_then(|i| i.as_media()).map(|m| m.info.clone()),
_ => None,
};
let mut st = info.as_ref().map(crate::commands::default_seq_settings_for).unwrap_or_default();
if let Some(m) = s.project.item(item).and_then(|i| i.as_media())
&& let Some(r) = m.interpret.frame_rate
{
st.frame_rate = r;
}
st
}
fn processed_bin(p: &mut Project) -> filmcraft_project::BinId {
if let Some(b) = p.root.children.iter().find_map(|c| match c {
BinEntry::Bin(b) if b.name == "Processed Clips" => Some(b.id),
_ => None,
}) {
return b;
}
p.add_bin("Processed Clips", None)
}
fn merge_clips(s: &mut Session, p: &Value) -> Result<Value> {
let cmd = "clip.mergeClips";
let method = method_p(p, cmd)?;
let clips = sync_items(s, &items_p(s, p), cmd)?;
let has_v = |c: &SyncClip| s.project.item(c.item).is_some_and(|i| i.has_video());
let has_a = |c: &SyncClip| s.project.item(c.item).is_some_and(|i| i.has_audio());
let videos: Vec<usize> = (0..clips.len()).filter(|&i| has_v(&clips[i])).collect();
if videos.len() > 1 {
return Err(bad(cmd, "Merge Clips takes one video clip plus audio clips"));
}
let audio_n = clips.iter().filter(|c| !has_v(c)).count();
if audio_n == 0 || audio_n > 16 {
return Err(bad(cmd, "Merge Clips takes 1–16 audio clips"));
}
// the video clip (or the first audio clip) is the reference
let reference = videos.first().copied().unwrap_or(0);
let mut order: Vec<usize> = vec![reference];
order.extend((0..clips.len()).filter(|&i| i != reference));
let clips: Vec<SyncClip> = order.iter().map(|&i| clips[i].clone()).collect();
let settings = settings_for(s, clips[0].item);
let rate = settings.frame_rate;
let anchors = sync::anchors(s, &clips, 0, &method, offset_p(p, rate))?;
let starts = sync::placements(&clips, &anchors, rate, false);
let remove_video_audio = bool_p(p, "removeVideoAudio").unwrap_or(false);
let name = str_p(p, "name").map(str::to_string).unwrap_or_else(|| format!("{} - Merged", clips[0].name));
let video = videos.first().map(|_| clips[0].item);
let vid_audio = video.is_some() && has_a(&clips[0]) && !remove_video_audio;
let audio_items: Vec<usize> = (1..clips.len()).collect();
let n_audio = audio_items.len() + vid_audio as usize;
let rep = sync::report(&clips, &anchors, &starts, rate);
let bin = s.project.root.parent_of(clips[0].item).filter(|b| *b != s.project.root.id);
let id = s.edit("Merge Clips", |pr, st| {
let sid = pr.new_sequence(&name, settings.clone(), video.is_some() as usize, n_audio, bin);
let mut v_items = Vec::new();
let mut a_items = Vec::new();
let link = pr.alloc_id();
if video.is_some() {
let mut v = placed_item(pr, clips[0].item, TrackKind::Video, starts[0], clips[0].range, &settings).ok_or_else(|| bad(cmd, "bad video clip"))?;
v.link = Some(link);
v_items.push(v);
if vid_audio {
let mut a = placed_item(pr, clips[0].item, TrackKind::Audio, starts[0], clips[0].range, &settings).ok_or_else(|| bad(cmd, "bad clip"))?;
a.link = Some(link);
a_items.push(a);
}
}
for &i in &audio_items {
let mut a = placed_item(pr, clips[i].item, TrackKind::Audio, starts[i], clips[i].range, &settings).ok_or_else(|| bad(cmd, "bad audio clip"))?;
a.link = Some(link);
a_items.push(a);
}
let q = pr.sequence_mut(sid).expect("new sequence");
for (t, it) in q.video_tracks.iter_mut().zip(v_items) {
t.items.push(it);
}
for (t, it) in q.audio_tracks.iter_mut().zip(a_items) {
t.items.push(it);
}
q.merged = Some(MergedClip { video, audio: audio_items.iter().map(|&i| clips[i].item).collect(), sync: method.name().into() });
q.check().map_err(EngineError::Other)?;
if let Some(it) = pr.item_mut(sid) {
it.label = Label::Iris;
}
st.project_selection = vec![sid];
Ok(sid)
})?;
Ok(json!({"item": id.0, "clips": rep}))
}
fn create_multicam(s: &mut Session, p: &Value) -> Result<Value> {
let cmd = "clip.createMulticam";
let method = method_p(p, cmd)?;
let mut clips = sync_items(s, &items_p(s, p), cmd)?;
let has_v = |s: &Session, c: &SyncClip| s.project.item(c.item).is_some_and(|i| i.has_video());
let has_a = |s: &Session, c: &SyncClip| s.project.item(c.item).is_some_and(|i| i.has_audio());
// cameras with video first (selection order), then audio-only sources
clips.sort_by_key(|c| !has_v(s, c));
if !clips.iter().any(|c| has_v(s, c)) {
return Err(bad(cmd, "select at least one clip with video"));
}
let names_mode = str_p(p, "cameraNames").unwrap_or("clip").to_string();
if names_mode == "metadata" {
// cameras ordered by their Camera Angle metadata when present
let key = |c: &SyncClip| s.project.item(c.item).and_then(|i| i.metadata.get("Camera Angle").cloned());
let mut v: Vec<(bool, Option<String>, SyncClip)> = clips.iter().map(|c| (!has_v(s, c), key(c), c.clone())).collect();
v.sort_by_key(|a| (a.0, a.1.is_none(), a.1.clone()));
clips = v.into_iter().map(|x| x.2).collect();
}
let reference = match item_p(p, "reference") {
Some(r) => clips.iter().position(|c| c.item == r).ok_or_else(|| bad(cmd, "`reference` is not among the items"))?,
None => 0,
};
let audio = match str_p(p, "audio") {
Some(a) => MulticamAudio::from_name(a).ok_or_else(|| bad(cmd, format!("unknown audio mode `{a}` (camera1, all, switch)")))?,
None => MulticamAudio::Camera1,
};
let settings = settings_for(s, clips[reference].item);
let rate = settings.frame_rate;
let anchors = sync::anchors(s, &clips, reference, &method, offset_p(p, rate))?;
let starts = sync::placements(&clips, &anchors, rate, false);
let rep = sync::report(&clips, &anchors, &starts, rate);
let name = str_p(p, "name").map(str::to_string).unwrap_or_else(|| format!("{} Multicam", clips[reference].name));
let move_processed = bool_p(p, "processedBin").unwrap_or(false);
let flags: Vec<(bool, bool)> = clips.iter().map(|c| (has_v(s, c), has_a(s, c))).collect();
let names: Vec<String> = clips.iter().enumerate().map(|(i, c)| camera_name(&s.project, c.item, &names_mode, i + 1)).collect();
let nv = flags.iter().filter(|f| f.0).count();
let na = flags.iter().filter(|f| f.1).count();
let id = s.edit("Create Multi-Camera Source Sequence", |pr, st| {
let sid = pr.new_sequence(&name, settings.clone(), nv, na.max(1), None);
let mut cameras = Vec::new();
let (mut vi, mut ai) = (0usize, 0usize);
let mut placed: Vec<(TrackKind, usize, TrackItem)> = Vec::new();
let (vtracks, atracks): (Vec<TrackId>, Vec<TrackId>) = {
let q = pr.sequence(sid).expect("new sequence");
(q.video_tracks.iter().map(|t| t.id).collect(), q.audio_tracks.iter().map(|t| t.id).collect())
};
for (k, c) in clips.iter().enumerate() {
let (hv, ha) = flags[k];
let link = (hv && ha).then(|| pr.alloc_id());
let mut cam = Camera { name: names[k].clone(), video_track: None, audio_tracks: Vec::new(), enabled: true, source: Some(c.item) };
if hv {
let mut v = placed_item(pr, c.item, TrackKind::Video, starts[k], c.range, &settings).ok_or_else(|| bad(cmd, "bad clip"))?;
v.link = link;
placed.push((TrackKind::Video, vi, v));
cam.video_track = Some(vtracks[vi]);
vi += 1;
}
if ha {
let mut a = placed_item(pr, c.item, TrackKind::Audio, starts[k], c.range, &settings).ok_or_else(|| bad(cmd, "bad clip"))?;
a.link = link;
placed.push((TrackKind::Audio, ai, a));
cam.audio_tracks.push(atracks[ai]);
ai += 1;
}
cameras.push(cam);
}
{
let q = pr.sequence_mut(sid).expect("new sequence");
for (kind, idx, it) in placed {
q.tracks_mut(kind)[idx].items.push(it);
}
for cam in &cameras {
if let Some(vt) = cam.video_track.and_then(|id| q.video_tracks.iter_mut().find(|x| x.id == id)) {
vt.name = cam.name.clone();
}
for at in &cam.audio_tracks {
if let Some(x) = q.audio_tracks.iter_mut().find(|x| x.id == *at) {
x.name = cam.name.clone();
}
}
}
q.multicam = Some(MulticamSource { cameras, audio, sync: method.name().into() });
q.check().map_err(EngineError::Other)?;
}
if let Some(it) = pr.item_mut(sid) {
it.label = Label::Mango;
}
if move_processed {
let bin = processed_bin(pr);
for c in &clips {
pr.root.remove_item(c.item);
if let Some(b) = pr.root.find_bin_mut(bin) {
b.children.push(BinEntry::Item(c.item));
}
}
}
st.project_selection = vec![sid];
Ok(sid)
})?;
Ok(json!({"sequence": id.0, "cameras": names, "clips": rep}))
}
// ---------------------------------------------------------------- multi-camera clips
/// The topmost enabled multi-camera video clip at `t` (targeted tracks first) in the active
/// sequence: (track, clip).
pub fn multicam_clip_at(s: &Session, t: Tick) -> Option<(TrackId, TrackItem)> {
let q = s.active_sequence()?;
let tg = s.targeting().targeted;
let at = |tr: &Track| tr.item_at(t).filter(|i| i.multicam.is_some_and(|m| m.enabled) && s.project.sequence(i.item).is_some()).cloned();
let targeted = q.video_tracks.iter().rev().filter(|tr| tg.contains(&tr.id)).find_map(|tr| at(tr).map(|i| (tr.id, i)));
targeted.or_else(|| q.video_tracks.iter().rev().find_map(|tr| at(tr).map(|i| (tr.id, i))))
}
/// Resolve `camera` (1-based, shown order) or `angle` (0-based) for the source of `clip`.
fn angle_p(s: &Session, p: &Value, source: ItemId, cmd: &str) -> Result<u32> {
let q = s.project.sequence(source).ok_or_else(|| bad(cmd, "not a multi-camera clip"))?;
let cams = q.cameras();
if let Some(a) = u64_p(p, "angle") {
if (a as usize) < cams.cameras.len() {
return Ok(a as u32);
}
return Err(bad(cmd, format!("angle {a} out of range (0–{})", cams.cameras.len().saturating_sub(1))));
}
let n = u64_p(p, "camera").ok_or_else(|| bad(cmd, "need `camera` (1-based) or `angle`"))? as usize;
let shown = cams.shown_angles();
shown.get(n.wrapping_sub(1)).map(|a| *a as u32).ok_or_else(|| bad(cmd, format!("there is no camera {n} ({} shown)", shown.len())))
}
/// Linked audio clips of `video` that are multi-camera clips.
fn audio_partners(q: &Sequence, video: &TrackItem) -> Vec<ClipId> {
let Some(l) = video.link else { return Vec::new() };
q.audio_tracks.iter().flat_map(|t| t.items.iter()).filter(|i| i.link == Some(l) && i.multicam.is_some()).map(|i| i.id).collect()
}
fn switch_angle(s: &mut Session, p: &Value) -> Result<Value> {
let cmd = "multicam.switchAngle";
let q = s.active_sequence().ok_or(EngineError::NoSequence)?.clone();
let t = time_p(s, p, "").unwrap_or(s.playhead());
let video_only = bool_p(p, "videoOnly").unwrap_or(false);
let audio_only = bool_p(p, "audioOnly").unwrap_or(false);
let follows = s.state.multicam_audio_follows_video;
let videos: Vec<TrackItem> = if p.get("clips").is_some() || p.get("clip").is_some() {
clips_p(s, p).iter().filter_map(|c| q.find_item(*c).map(|x| x.1.clone())).filter(|i| i.multicam.is_some()).collect()
} else {
multicam_clip_at(s, t).map(|x| vec![x.1]).unwrap_or_default()
};
if videos.is_empty() {
return Err(EngineError::Other("no multi-camera clip at the playhead".into()));
}
let angle = angle_p(s, p, videos[0].item, cmd)?;
let mut targets: Vec<ClipId> = Vec::new();
for v in &videos {
let is_audio = q.audio_tracks.iter().any(|t| t.item(v.id).is_some());
if is_audio {
targets.push(v.id);
continue;
}
if !audio_only {
targets.push(v.id);
}
if audio_only || (follows && !video_only) {
targets.extend(audio_partners(&q, v));
}
}
let n = s.edit_sequence("Switch Multi-Camera Angle", |seq, _, _| Ok(edit::multicam::set_angle(seq, &targets, angle)))?;
Ok(json!({"angle": angle, "changed": n}))
}
fn enable(s: &mut Session, p: &Value) -> Result<Value> {
let q = s.active_sequence().ok_or(EngineError::NoSequence)?.clone();
let clips = with_links(s, &clips_p(s, p));
let nested: Vec<TrackItem> = clips.iter().filter_map(|c| q.find_item(*c).map(|x| x.1.clone())).filter(|i| s.project.sequence(i.item).is_some()).collect();
if nested.is_empty() {
return Err(EngineError::Other("select nested sequence or multi-camera clips".into()));
}
let on = bool_p(p, "enabled").unwrap_or_else(|| nested.iter().any(|i| !i.multicam.is_some_and(|m| m.enabled)));
let project = s.project.clone();
let ids: Vec<ClipId> = nested.iter().map(|i| i.id).collect();
let n = s.edit_sequence(if on { "Enable Multi-Camera" } else { "Disable Multi-Camera" }, |seq, _, _| {
let mut n = 0;
for t in seq.all_tracks_mut() {
for it in t.items.iter_mut().filter(|i| ids.contains(&i.id)) {
let first = project.sequence(it.item).and_then(|q| q.cameras().first_video_angle()).unwrap_or(0) as u32;
let sel = it.multicam.get_or_insert(MulticamSel { enabled: false, angle: first });
if sel.enabled != on {
sel.enabled = on;
n += 1;
}
}
}
Ok(n)
})?;
Ok(json!({"enabled": on, "changed": n}))
}
fn flatten(s: &mut Session, p: &Value) -> Result<Value> {
let seq_id = s.state.active_sequence.ok_or(EngineError::NoSequence)?;
let q = s.active_sequence().ok_or(EngineError::NoSequence)?.clone();
let clips = with_links(s, &clips_p(s, p));
let mut targets: Vec<(TrackKind, usize, TrackItem)> = Vec::new();
for (k, ts) in [(TrackKind::Video, &q.video_tracks), (TrackKind::Audio, &q.audio_tracks)] {
for (ti, t) in ts.iter().enumerate() {
for it in t.items.iter().filter(|i| clips.contains(&i.id) && i.multicam.is_some_and(|m| m.enabled)) {
targets.push((k, ti, it.clone()));
}
}
}
if targets.is_empty() {
return Err(EngineError::Other("select multi-camera clips to flatten".into()));
}
let project = s.project.clone();
let media = s.media.clone();
let n = targets.len();
let ids = s.edit("Flatten Multi-Camera", |pr, st| {
// extra audio tracks needed for sources whose audio spans several tracks
let need_a = targets
.iter()
.filter(|t| t.0 == TrackKind::Audio)
.map(|(_, ti, it)| {
let nq = project.sequence(it.item).map(|nq| audible(nq, it).len()).unwrap_or(1);
ti + nq.max(1)
})
.max()
.unwrap_or(0);
let have = pr.sequence(seq_id).map(|q| q.audio_tracks.len()).unwrap_or(0);
for k in have..need_a {
let id = TrackId(pr.alloc_id());
if let Some(q) = pr.sequence_mut(seq_id) {
q.audio_tracks.push(Track::new(id, TrackKind::Audio, format!("Audio {}", k + 1)));
}
}
let new_ids = seq_edit(pr, seq_id, &media, |seq, ctx| {
let mut links = HashMap::new();
let mut out = Vec::new();
for (kind, ti, it) in &targets {
let Some(nested) = project.sequence(it.item) else { continue };
let mut pieces: Vec<(TrackId, TrackItem)> = Vec::new();
match kind {
TrackKind::Video => {
let angle = it.multicam.map(|m| m.angle as usize).unwrap_or(0);
if let Some(nt) = nested.angle_video_track_index(angle).and_then(|i| nested.video_tracks.get(i)) {
let dest = seq.video_tracks[*ti].id;
pieces.extend(edit::multicam::flatten_items(it, nt, ctx, &mut links)?.into_iter().map(|x| (dest, x)));
}
}
TrackKind::Audio => {
for (k, at) in audible(nested, it).iter().enumerate() {
let Some(nt) = nested.audio_tracks.iter().find(|x| x.id == *at) else { continue };
let Some(dest) = seq.audio_tracks.get(ti + k).map(|t| t.id) else { continue };
pieces.extend(edit::multicam::flatten_items(it, nt, ctx, &mut links)?.into_iter().map(|x| (dest, x)));
}
}
}
out.extend(edit::multicam::replace_with(seq, it.id, pieces, ctx)?);
}
Ok(out)
})?;
st.selection = new_ids.clone();
Ok(new_ids)
})?;
Ok(json!({"flattened": n, "clips": ids.iter().map(|c| c.0).collect::<Vec<_>>()}))
}
/// Audio tracks of `nested` a multi-camera audio clip plays.
fn audible(nested: &Sequence, it: &TrackItem) -> Vec<TrackId> {
let angle = it.multicam_angle(nested);
match &nested.multicam {
Some(mc) => mc.audible_tracks(if mc.audio == MulticamAudio::SwitchAudio { angle } else { None }),
None => nested.audio_tracks.iter().filter(|t| !t.muted).map(|t| t.id).collect(),
}
}
// ---------------------------------------------------------------- live switching
/// A live switching pass (Multi-Camera view, playing): every cut is applied at once, and the whole
/// pass is one undo step.
#[derive(Clone, Debug, Default)]
pub struct Recorder {
/// Sequence being recorded (None = no pass running).
pub seq: Option<ItemId>,
pub start: Tick,
/// Tracks the pass switches (fixed by the first cut).
pub tracks: Vec<TrackId>,
pub cuts: Vec<Cut>,
/// The multi-camera source being switched (fixed by the first cut).
pub source: Option<ItemId>,
/// The project before the pass's first cut.
base: Option<Arc<Project>>,
key: String,
pass: u64,
}
impl Recorder {
pub fn active(&self) -> bool {
self.seq.is_some()
}
}
const RECORD_LABEL: &str = "Record Multi-Camera";
fn record_start(s: &mut Session, p: &Value) -> Result<Value> {
let t = time_p(s, p, "").unwrap_or(s.playhead());
let pass = s.mcrec.pass + 1;
s.mcrec = Recorder { seq: s.state.active_sequence, start: t, pass, key: format!("multicam-record-{pass}"), ..Default::default() };
Ok(json!({"recording": true, "start": t.0}))
}
/// Re-apply the pass from its base project up to `end`.
fn apply_pass(s: &mut Session, end: Tick) -> Result<usize> {
let Some(seq_id) = s.mcrec.seq else { return Ok(0) };
let merging = s.history.merge_key.as_deref() == Some(s.mcrec.key.as_str()) && s.history.undo.last().is_some_and(|u| u.0 == RECORD_LABEL);
if !merging || s.mcrec.base.is_none() {
s.mcrec.base = Some(s.project.clone());
}
let base = s.mcrec.base.clone().expect("set above");
let (tracks, cuts, key) = (s.mcrec.tracks.clone(), s.mcrec.cuts.clone(), s.mcrec.key.clone());
let Some(source) = s.mcrec.source else { return Ok(0) };
let media = s.media.clone();
s.edit_merged(RECORD_LABEL, &key, |pr, _| {
*pr = (*base).clone();
seq_edit(pr, seq_id, &media, |seq, ctx| Ok(edit::multicam::record(seq, &tracks, source, &cuts, end, ctx)))
})
}
fn cut(s: &mut Session, p: &Value) -> Result<Value> {
let cmd = "multicam.cut";
let t = time_p(s, p, "").unwrap_or(s.playhead());
if !s.mcrec.active() || s.mcrec.seq != s.state.active_sequence {
// stopped: switch the clip at the playhead
return switch_angle(s, p);
}
let (_, clip) = multicam_clip_at(s, t).ok_or_else(|| EngineError::Other("no multi-camera clip at the playhead".into()))?;
let angle = angle_p(s, p, clip.item, cmd)?;
if s.mcrec.tracks.is_empty() {
let q = s.active_sequence().ok_or(EngineError::NoSequence)?;
let mut tracks: Vec<TrackId> =
q.video_tracks.iter().filter(|tr| tr.item_at(t).is_some_and(|i| i.item == clip.item && i.multicam.is_some())).map(|tr| tr.id).collect();
tracks.truncate(1);
if s.state.multicam_audio_follows_video && !bool_p(p, "videoOnly").unwrap_or(false) {
let partners = audio_partners(q, &clip);
tracks.extend(q.audio_tracks.iter().filter(|tr| tr.items.iter().any(|i| partners.contains(&i.id))).map(|tr| tr.id));
}
s.mcrec.tracks = tracks;
}
s.mcrec.source = Some(clip.item);
s.mcrec.cuts.push(Cut { time: t, angle });
// provisional: the new angle runs to the end of this multi-camera clip (its contiguous pieces)
// until the pass stops
let end = s
.active_sequence()
.and_then(|q| q.track(s.mcrec.tracks[0]))
.map(|tr| {
let mut end = clip.end();
for it in tr.items.iter().filter(|i| i.start >= clip.end()) {
if it.start != end || it.item != clip.item {
break;
}
end = it.end();
}
end
})
.unwrap_or(clip.end());
apply_pass(s, end.max(t))?;
Ok(json!({"angle": angle, "time": t.0, "cuts": s.mcrec.cuts.len()}))
}
/// Ctrl+N: an edit at the playhead on the multi-camera clip (and, when audio follows video, its
/// linked audio), and camera N from there to the clip's end.
fn cut_to_camera(s: &mut Session, p: &Value) -> Result<Value> {
let cmd = "multicam.cutToCamera";
if s.mcrec.active() && s.mcrec.seq == s.state.active_sequence {
return cut(s, p);
}
let t = time_p(s, p, "").unwrap_or(s.playhead());
let (_, clip) = multicam_clip_at(s, t).ok_or_else(|| EngineError::Other("no multi-camera clip at the playhead".into()))?;
let angle = angle_p(s, p, clip.item, cmd)?;
let q = s.active_sequence().ok_or(EngineError::NoSequence)?;
let mut clips = vec![clip.id];
if s.state.multicam_audio_follows_video && !bool_p(p, "videoOnly").unwrap_or(false) {
clips.extend(audio_partners(q, &clip));
}
let n = s.edit_sequence("Cut to Camera", |seq, ctx, _| {
let new = edit::razor_items(seq, &clips, t, ctx);
// the pieces starting at t (new right halves, or the clips themselves at an edit point)
let right: Vec<ClipId> =
seq.all_tracks().flat_map(|tr| tr.items.iter()).filter(|i| i.start == t && (new.contains(&i.id) || clips.contains(&i.id))).map(|i| i.id).collect();
Ok(edit::multicam::set_angle(seq, &right, angle))
})?;
Ok(json!({"angle": angle, "changed": n}))
}
fn record_stop(s: &mut Session, p: &Value) -> Result<Value> {
if !s.mcrec.active() {
return Ok(json!({"recording": false, "cuts": 0}));
}
let t = time_p(s, p, "").unwrap_or(s.playhead());
let n = s.mcrec.cuts.len();
let mut points = 0;
if n > 0 {
points = apply_pass(s, t)?;
}
let pass = s.mcrec.pass;
s.mcrec = Recorder { pass, ..Default::default() };
s.history.merge_key = None;
Ok(json!({"recording": false, "cuts": n, "editPoints": points}))
}
fn audio_follows(s: &mut Session, p: &Value) -> Result<Value> {
let on = bool_p(p, "enabled").unwrap_or(!s.state.multicam_audio_follows_video);
s.state.multicam_audio_follows_video = on;
Ok(json!({"enabled": on}))
}
fn edit_cameras(s: &mut Session, p: &Value) -> Result<Value> {
let cmd = "multicam.editCameras";
let id = match item_p(p, "sequence") {
Some(i) => i,
None => multicam_clip_at(s, s.playhead()).map(|c| c.1.item).ok_or_else(|| bad(cmd, "need `sequence` (or a multi-camera clip at the playhead)"))?,
};
let q = s.project.sequence(id).ok_or_else(|| bad(cmd, "no such sequence"))?;
if q.multicam.is_none() {
return Err(bad(cmd, "not a multi-camera source sequence"));
}
let changes: Vec<Value> = p.get("cameras").and_then(Value::as_array).cloned().unwrap_or_default();
let audio = match str_p(p, "audio") {
Some(a) => Some(MulticamAudio::from_name(a).ok_or_else(|| bad(cmd, format!("unknown audio mode `{a}`")))?),
None => None,
};
s.edit("Edit Cameras", |pr, _| {
let q = pr.sequence_mut(id).ok_or_else(|| bad(cmd, "no such sequence"))?;
let mc = q.multicam.as_mut().ok_or_else(|| bad(cmd, "not a multi-camera source"))?;
for c in &changes {
let a = c.get("angle").and_then(Value::as_u64).ok_or_else(|| bad(cmd, "each camera needs `angle`"))? as usize;
let cam = mc.cameras.get_mut(a).ok_or_else(|| bad(cmd, format!("no angle {a}")))?;
if let Some(n) = c.get("name").and_then(Value::as_str) {
cam.name = n.to_string();
}
if let Some(e) = c.get("enabled").and_then(Value::as_bool) {
cam.enabled = e;
}
}
if let Some(a) = audio {
mc.audio = a;
}
Ok(())
})?;
Ok(json!({"sequence": id.0}))
}
/// The multi-camera clip at the playhead (or `time`), for the Multi-Camera view and agents.
pub fn inspect_at(s: &Session, t: Tick) -> Value {
let rec = json!({"recording": s.mcrec.active(), "cuts": s.mcrec.cuts.iter().map(|c| json!({"time": c.time.0, "angle": c.angle})).collect::<Vec<_>>()});
let Some((track, clip)) = multicam_clip_at(s, t) else {
return json!({"clip": null, "recorder": rec, "audioFollowsVideo": s.state.multicam_audio_follows_video});
};
let q = s.project.sequence(clip.item);
let cams = q.map(|q| q.cameras()).unwrap_or_default();
let audio_angle = s
.active_sequence()
.map(|seq| audio_partners(seq, &clip))
.and_then(|ps| ps.first().and_then(|c| s.active_sequence().and_then(|seq| seq.find_item(*c)).and_then(|x| x.1.multicam.map(|m| m.angle))));
json!({
"clip": clip.id.0,
"track": track.0,
"source": clip.item.0,
"sourceName": s.project.item(clip.item).map(|i| i.name.clone()),
"sourceTime": clip.source_time_at(t).0,
"angle": clip.multicam.map(|m| m.angle),
"audioAngle": audio_angle,
"audio": cams.audio.name(),
"shown": cams.shown_angles(),
"cameras": cams.cameras.iter().enumerate().map(|(i, c)| json!({"angle": i, "name": c.name, "enabled": c.enabled, "video": c.video_track.is_some()})).collect::<Vec<_>>(),
"recorder": rec,
"audioFollowsVideo": s.state.multicam_audio_follows_video,
})
}
fn inspect(s: &mut Session, p: &Value) -> Result<Value> {
let t = time_p(s, p, "").unwrap_or(s.playhead());
Ok(inspect_at(s, t))
}

View File

@ -0,0 +1,677 @@
//! Synchronisation and multi-camera editing end to end.
//!
//! Fixtures (ffmpeg, `target/fixtures/multicam/`): one "event" (pink noise gated into bursts, so
//! it is not periodic) recorded by three devices that started at different times — two cameras
//! (MJPEG + PCM `.mov` with QuickTime timecode) and an external recorder (`.wav`) — each with its
//! own gain and independent noise. The true offsets are known to the sample.
//!
//! The editing tests use movies made with our own encoders (no ffmpeg).
use std::path::{Path, PathBuf};
use std::process::Command;
use filmcraft_media::DemoScene;
use filmcraft_project::{ItemId, Marker, MarkerId, MarkerKind, MulticamAudio, Sequence, TrackItem};
use filmcraft_time::{FrameRate, TICKS_PER_SECOND, Tick};
use serde_json::{Value, json};
use crate::Session;
use crate::media_test_util::{make_movie, psnr, tmp_dir};
const SR: i64 = 48_000;
/// Event samples at which each recording starts (camera A, camera B, recorder).
const START_A: i64 = 96_000;
const START_B: i64 = 96_000 + 121_234;
const START_REC: i64 = 48_000;
const SAMPLE: Tick = Tick(TICKS_PER_SECOND / SR);
fn ff(ffmpeg: &Path, args: &[&str]) -> bool {
Command::new(ffmpeg).args(["-v", "error", "-y"]).args(args).status().map(|s| s.success()).unwrap_or(false)
}
/// The event as heard by one device: starts at event sample `start`, `secs` long, `gain`, white
/// noise of amplitude `noise` (seed `seed`).
fn mic_filter(start: i64, secs: f64, gain: f64, noise: f64, seed: u32) -> String {
let len = (secs * SR as f64) as i64;
format!(
"anoisesrc=d=60:c=pink:r=48000:a=0.35:s=7,volume='if(lt(mod(t*2.7\\,1)\\,0.55)\\,1\\,0.08)':eval=frame,atrim=start_sample={start},asetpts=N/SR/TB,volume={gain}[s];\
anoisesrc=d=60:c=white:r=48000:a={noise}:s={seed}[n];[s][n]amix=inputs=2:normalize=0:duration=first,atrim=end_sample={len}[a]"
)
}
/// (camera A .mov, camera B .mov, recorder .wav), or None without ffmpeg.
fn fixtures() -> Option<(PathBuf, PathBuf, PathBuf)> {
let ffmpeg = filmcraft_testkit::ffmpeg_or_skip("multicam fixtures")?;
let dir = filmcraft_testkit::fixtures_dir("multicam");
let cam = |name: &str, src: &str, start: i64, secs: f64, gain: f64, noise: f64, seed: u32, tc: &str| {
filmcraft_testkit::fixtures::generate(&dir.join(name), |tmp| {
let t = tmp.to_string_lossy();
let video = format!("{src}=s=320x180:r=24:d={secs}");
ff(
&ffmpeg,
&[
"-f",
"lavfi",
"-i",
&video,
"-filter_complex",
&mic_filter(start, secs, gain, noise, seed),
"-map",
"0:v",
"-map",
"[a]",
"-c:v",
"mjpeg",
"-q:v",
"8",
"-c:a",
"pcm_s16le",
"-ar",
"48000",
"-ac",
"1",
"-timecode",
tc,
"-shortest",
&t,
],
)
})
};
let a = cam("camA.mov", "testsrc2", START_A, 12.0, 1.0, 0.02, 11, "01:00:00:00")?;
let b = cam("camB.mov", "smptebars", START_B, 10.0, 0.25, 0.05, 12, "01:00:02:12")?;
let rec = filmcraft_testkit::fixtures::generate(&dir.join("recorder.wav"), |tmp| {
let t = tmp.to_string_lossy();
ff(
&ffmpeg,
&[
"-f",
"lavfi",
"-i",
"anullsrc=r=48000",
"-filter_complex",
&mic_filter(START_REC, 16.0, 1.5, 0.003, 13),
"-map",
"[a]",
"-c:a",
"pcm_s16le",
"-ac",
"1",
&t,
],
)
})?;
Some((a, b, rec))
}
fn import(s: &mut Session, files: &[&Path]) -> Vec<ItemId> {
let paths: Vec<String> = files.iter().map(|p| p.to_string_lossy().into_owned()).collect();
let r = s.execute("file.import", json!({"paths": paths})).unwrap();
r["items"].as_array().unwrap().iter().map(|v| ItemId(v.as_u64().unwrap())).collect()
}
fn ids(v: &[ItemId]) -> Value {
json!(v.iter().map(|i| i.0).collect::<Vec<_>>())
}
/// Run a Project-panel command on `items` (selected first, as in the UI).
fn on_items(s: &mut Session, cmd: &str, items: &[ItemId], mut p: Value) -> crate::Result<Value> {
s.execute("project.select", json!({"items": ids(items)})).unwrap();
p["items"] = ids(items);
s.execute(cmd, p)
}
/// Timeline time at which event sample `m` plays in clip `ti` (whose media starts at event sample
/// `start`).
fn event_time(ti: &TrackItem, start: i64, m: i64) -> Tick {
let media = Tick::from_units(m - start, SR);
ti.start + (media - ti.source_in)
}
fn find_clip(q: &Sequence, item: ItemId, video: bool) -> &TrackItem {
let tracks = if video { &q.video_tracks } else { &q.audio_tracks };
tracks.iter().flat_map(|t| t.items.iter()).find(|i| i.item == item).unwrap_or_else(|| panic!("no clip of {item:?}"))
}
#[test]
fn audio_sync_multicam_sample_accurate() {
let Some((a, b, rec)) = fixtures() else { return };
let mut s = Session::default();
let items = import(&mut s, &[&a, &b, &rec]);
let t0 = std::time::Instant::now();
let r =
on_items(&mut s, "clip.createMulticam", &items, json!({"method": "audio", "audio": "switch", "name": "Interview MC", "processedBin": true})).unwrap();
let ms = t0.elapsed().as_secs_f64() * 1000.0;
let seq_id = ItemId(r["sequence"].as_u64().unwrap());
let q = s.project.sequence(seq_id).unwrap().clone();
// structure: two cameras with video, an audio-only recorder, switching audio
let mc = q.multicam.as_ref().unwrap();
assert_eq!(mc.cameras.len(), 3);
assert_eq!(q.video_tracks.len(), 2);
assert_eq!(q.audio_tracks.len(), 3);
assert_eq!(mc.audio, MulticamAudio::SwitchAudio);
assert_eq!(mc.cameras[0].name, "camA.mov");
assert!(mc.cameras[2].video_track.is_none() && mc.cameras[2].audio_tracks.len() == 1, "the recorder is an audio-only source");
assert_eq!(q.video_tracks[0].name, "camA.mov");
assert_eq!(s.project.item(seq_id).unwrap().type_label(), "Multi-Camera Source Sequence");
// processed clips moved into their bin
let bin = s.project.root.children.iter().find_map(|c| match c {
filmcraft_project::BinEntry::Bin(b) if b.name == "Processed Clips" => Some(b.clone()),
_ => None,
});
assert_eq!(bin.map(|b| b.children.len()), Some(3));
// every recording plays event sample m at the same timeline time (±1 sample)
let clips = [(find_clip(&q, items[0], false), START_A), (find_clip(&q, items[1], false), START_B), (find_clip(&q, items[2], false), START_REC)];
let m = 300_000;
let t_ref = event_time(clips[0].0, clips[0].1, m);
let mut worst = Tick::ZERO;
for (c, start) in &clips[1..] {
let d = (event_time(c, *start, m) - t_ref).abs();
worst = worst.max(d);
}
// video clips sit on frame boundaries and keep the same alignment as their audio
for (k, item) in items[..2].iter().enumerate() {
let v = find_clip(&q, *item, true);
assert_eq!(v.start, q.settings.frame_rate.snap(v.start), "frame-aligned");
assert_eq!((v.start, v.source_in), (clips[k].0.start, clips[k].0.source_in));
}
let report = &r["clips"];
eprintln!(
"audio sync: worst error {:.2} samples over 3 recordings; B lag {} (truth {}), recorder lag {} (truth {}); {ms:.0} ms",
worst.0 as f64 / SAMPLE.0 as f64,
report[1]["audio"]["lagSamples"],
START_B - START_A,
report[2]["audio"]["lagSamples"],
START_REC - START_A
);
assert!(worst <= SAMPLE, "worst {} samples", worst.0 as f64 / SAMPLE.0 as f64);
assert!(report[1]["audio"]["reliable"].as_bool().unwrap() && report[2]["audio"]["reliable"].as_bool().unwrap(), "{report}");
// undo removes the sequence and restores the bins
s.execute("edit.undo", json!({})).unwrap();
assert!(s.project.sequence(seq_id).is_none());
}
#[test]
fn timecode_marker_and_in_point_sync_within_one_frame() {
let Some((a, b, _)) = fixtures() else { return };
let mut s = Session::default();
let items = import(&mut s, &[&a, &b]);
let truth = Tick::from_units(START_B - START_A, SR);
let rate = FrameRate::FPS_24;
let offset_of = |s: &Session, r: &Value| -> Tick {
let q = s.project.sequence(ItemId(r["sequence"].as_u64().unwrap())).unwrap();
let (va, vb) = (find_clip(q, items[0], true), find_clip(q, items[1], true));
// timeline time of media 0 of each camera; B starts `truth` after A in the event
(vb.start - vb.source_in) - (va.start - va.source_in)
};
// timecode: A 01:00:00:00, B 01:00:02:12 = 2.5 s later (the true offset is 2.5257 s)
let r = on_items(&mut s, "clip.createMulticam", &items, json!({"method": "timecode"})).unwrap();
let off = offset_of(&s, &r);
assert_eq!(off, rate.tick_of(60), "timecode offset is exactly 60 frames");
assert!((off - truth).abs() <= rate.frame_duration(), "within a frame of the truth");
// ignoring hours gives the same result here
let r = on_items(&mut s, "clip.createMulticam", &items, json!({"method": "timecode", "ignoreHours": true})).unwrap();
assert_eq!(offset_of(&s, &r), rate.tick_of(60));
// clip markers: the same event moment marked in both clips (frame accuracy)
let event_frame_a = 72; // 3 s into camera A
let at_b = rate.tick_of(event_frame_a) - truth;
let mut p = (*s.project).clone();
for (it, t) in [(items[0], rate.tick_of(event_frame_a)), (items[1], rate.snap(at_b))] {
let m = p.item_mut(it).unwrap().as_media_mut().unwrap();
m.markers.push(Marker {
id: MarkerId(900 + it.0),
start: t,
duration: Tick::ZERO,
name: "clap".into(),
comment: String::new(),
kind: MarkerKind::Comment,
color: filmcraft_project::Label::Rose,
});
}
s.project = std::sync::Arc::new(p);
let r = on_items(&mut s, "clip.createMulticam", &items, json!({"method": "marker", "marker": "clap"})).unwrap();
let off = offset_of(&s, &r);
assert!((off - truth).abs() <= rate.frame_duration(), "marker sync {} vs {}", off.0, truth.0);
// In points: A's In at 2.5 s, B's at 0 → B starts 2.5 s after A
let mut p = (*s.project).clone();
p.item_mut(items[0]).unwrap().as_media_mut().unwrap().mark_in = Some(rate.tick_of(60));
s.project = std::sync::Arc::new(p);
let r = on_items(&mut s, "clip.createMulticam", &items, json!({"method": "in"})).unwrap();
assert_eq!(offset_of(&s, &r), rate.tick_of(60));
// Out points: A ends at 12 s, B at 10 s, both Outs unset → ends aligned: B media 0 at 2 s
let r = on_items(&mut s, "clip.createMulticam", &items, json!({"method": "out"})).unwrap();
assert_eq!(offset_of(&s, &r), rate.tick_of(48));
// a missing marker is an error, not a guess
assert!(on_items(&mut s, "clip.createMulticam", &items, json!({"method": "marker", "marker": "nope"})).is_err());
}
#[test]
fn merge_clips_by_audio() {
let Some((a, _, rec)) = fixtures() else { return };
let mut s = Session::default();
let items = import(&mut s, &[&a, &rec]);
let r = on_items(&mut s, "clip.mergeClips", &items, json!({"method": "audio", "removeVideoAudio": true})).unwrap();
let id = ItemId(r["item"].as_u64().unwrap());
let q = s.project.sequence(id).unwrap();
assert_eq!(s.project.item(id).unwrap().type_label(), "Merged Clip");
assert_eq!(s.project.item(id).unwrap().name, "camA.mov - Merged");
assert_eq!((q.video_tracks.len(), q.audio_tracks.len()), (1, 1), "camera audio removed");
let v = find_clip(q, items[0], true);
let ar = find_clip(q, items[1], false);
assert_eq!(v.link, ar.link, "merged parts are linked");
// the recorder's event sample m plays with the camera's
let m = 400_000;
let d = (event_time(ar, START_REC, m) - (v.start + (Tick::from_units(m - START_A, SR) - v.source_in))).abs();
assert!(d <= SAMPLE, "merge offset error {} samples", d.0 as f64 / SAMPLE.0 as f64);
assert!(on_items(&mut s, "clip.mergeClips", &items[..1], json!({})).is_err(), "needs audio clips");
}
#[test]
fn timeline_synchronize_by_audio_and_timecode() {
let Some((a, b, rec)) = fixtures() else { return };
let mut s = Session::default();
let items = import(&mut s, &[&a, &b, &rec]);
let mut p = (*s.project).clone();
let rate = FrameRate::FPS_24;
let seq = p.new_sequence("Edit", filmcraft_project::SequenceSettings { width: 320, height: 180, frame_rate: rate, ..Default::default() }, 2, 3, None);
// camera B on V2/A2 at 20 s, recorder on A3 at 1 s, camera A on V1/A1 at 5 s
let place = |p: &mut filmcraft_project::Project, it: ItemId, at: i64, v: Option<usize>, a: usize| {
let d = p.item(it).unwrap().duration();
let range = filmcraft_time::TimeRange::new(Tick::ZERO, d);
let link = p.alloc_id();
if let Some(vt) = v {
let mut vi = p.make_track_item(it, filmcraft_project::TrackKind::Video, rate.tick_of(at), range, rate).unwrap();
vi.link = Some(link);
p.sequence_mut(seq).unwrap().video_tracks[vt].items.push(vi);
}
let mut ai = p.make_track_item(it, filmcraft_project::TrackKind::Audio, rate.tick_of(at), range, rate).unwrap();
ai.link = Some(link);
p.sequence_mut(seq).unwrap().audio_tracks[a].items.push(ai);
};
place(&mut p, items[0], 120, Some(0), 0);
place(&mut p, items[1], 480, Some(1), 1);
place(&mut p, items[2], 24, None, 2);
s.project = std::sync::Arc::new(p);
s.state.active_sequence = Some(seq);
let all: Vec<u64> = s.project.sequence(seq).unwrap().all_tracks().flat_map(|t| t.items.iter().map(|i| i.id.0)).collect();
s.execute("timeline.select", json!({"clips": all})).unwrap();
let r = s.execute("clip.synchronize", json!({"method": "audio"})).unwrap();
assert_eq!(r["moved"], 2, "{r}");
let q = s.active_sequence().unwrap().clone();
let ca = find_clip(&q, items[0], false);
assert_eq!(ca.start, rate.tick_of(120), "the reference (V1) stays put");
let m = 350_000;
for (it, start) in [(items[1], START_B), (items[2], START_REC)] {
let c = find_clip(&q, it, false);
let d = (event_time(c, start, m) - event_time(ca, START_A, m)).abs();
assert!(d <= SAMPLE, "timeline sync error {} samples", d.0 as f64 / SAMPLE.0 as f64);
assert_eq!(c.start, rate.snap(c.start), "moved clips stay on frames");
}
let vb = find_clip(&q, items[1], true);
let ab = find_clip(&q, items[1], false);
assert_eq!((vb.start, vb.source_in), (ab.start, ab.source_in), "linked partners moved together");
s.execute("edit.undo", json!({})).unwrap();
assert_eq!(find_clip(s.active_sequence().unwrap(), items[1], false).start, rate.tick_of(480));
// timecode: B lands exactly 60 frames after A
s.execute("timeline.select", json!({"clips": all})).unwrap();
let r = s.execute("clip.synchronize", json!({"method": "timecode", "track": "V1"})).unwrap();
assert!(r["moved"].as_u64().unwrap() >= 1);
let q = s.active_sequence().unwrap();
assert_eq!(find_clip(q, items[1], true).start - find_clip(q, items[0], true).start, rate.tick_of(60));
}
// ------------------------------------------------------------------ editing (our own media)
const W: u32 = 160;
const H: u32 = 90;
/// Three 4-second cameras (different scenes, own audio), a multi-camera source sequence by In
/// points, and a sequence holding one multi-camera clip on V1/A1. Returns (session, camera items,
/// source sequence, edit sequence).
fn mc_session(audio: &str) -> (Session, Vec<ItemId>, ItemId, ItemId) {
let dir = tmp_dir("mc-edit");
let files: Vec<PathBuf> = ["a.mov", "b.mov", "c.mov"].iter().map(|f| dir.join(f)).collect();
for (f, sc) in files.iter().zip([DemoScene::OceanSunset, DemoScene::CityNight, DemoScene::Aurora]) {
make_movie(f, sc, W, H, 96);
}
let mut s = Session::default();
let refs: Vec<&Path> = files.iter().map(|p| p.as_path()).collect();
let items = import(&mut s, &refs);
let r = on_items(&mut s, "clip.createMulticam", &items, json!({"method": "in", "audio": audio, "cameraNames": "track"})).unwrap();
let src = ItemId(r["sequence"].as_u64().unwrap());
let mut p = (*s.project).clone();
let edit =
p.new_sequence("Cut", filmcraft_project::SequenceSettings { width: W, height: H, frame_rate: FrameRate::FPS_24, ..Default::default() }, 2, 2, None);
s.project = std::sync::Arc::new(p);
s.state.active_sequence = Some(edit);
s.execute("source.open", json!({"item": src.0})).unwrap();
s.execute("source.overwrite", json!({})).unwrap();
s.set_playhead(Tick::ZERO);
(s, items, src, edit)
}
fn v1(s: &Session) -> Vec<(i64, u32)> {
let q = s.active_sequence().unwrap();
let rate = q.settings.frame_rate;
q.video_tracks[0].items.iter().map(|i| (rate.frame_at(i.start), i.multicam.unwrap().angle)).collect()
}
fn a1(s: &Session) -> Vec<(i64, u32)> {
let q = s.active_sequence().unwrap();
let rate = q.settings.frame_rate;
q.audio_tracks[0].items.iter().map(|i| (rate.frame_at(i.start), i.multicam.unwrap().angle)).collect()
}
fn frame(s: &mut Session, f: i64) -> Vec<u8> {
let t = s.sequence_rate().tick_of(f);
s.set_playhead(t);
s.render_program(1.0).unwrap().over_black_rgba8()
}
/// The frame of camera `item` at its media frame `f`, rendered alone.
fn camera_frame(s: &Session, item: ItemId, f: i64) -> Vec<u8> {
let provider = s.media.provider(s.project.clone(), s.services.clone());
filmcraft_render::render_item(&s.project, item, FrameRate::FPS_24.tick_of(f), 1.0, &provider).unwrap().over_black_rgba8()
}
#[test]
fn multicam_clip_structure_and_render() {
let (mut s, items, src, _) = mc_session("camera1");
let q = s.project.sequence(src).unwrap();
let mc = q.multicam.as_ref().unwrap();
assert_eq!(mc.cameras.iter().map(|c| c.name.as_str()).collect::<Vec<_>>(), ["Camera 1", "Camera 2", "Camera 3"]);
assert_eq!(mc.audio, MulticamAudio::Camera1);
assert_eq!((q.video_tracks.len(), q.audio_tracks.len()), (3, 3));
// edited in as an enabled multi-camera clip on angle 0 (video and audio)
assert_eq!(v1(&s), [(0, 0)]);
assert_eq!(a1(&s), [(0, 0)]);
let inspect = s.execute("multicam.inspect", json!({"frame": 10})).unwrap();
assert_eq!(inspect["source"], src.0);
assert_eq!(inspect["shown"], json!([0, 1, 2]));
// the clip shows angle 0 only (not the composite, whose top track is camera 3)
assert!(psnr(&frame(&mut s, 10), &camera_frame(&s, items[0], 10)) > 40.0);
s.execute("multicam.switchAngle", json!({"camera": 3})).unwrap();
assert_eq!(v1(&s), [(0, 2)]);
assert!(psnr(&frame(&mut s, 10), &camera_frame(&s, items[2], 10)) > 40.0);
// disabling multi-camera shows the plain nest (top track = camera 3 here, so pick camera 2)
s.execute("multicam.switchAngle", json!({"camera": 2})).unwrap();
let clip = s.active_sequence().unwrap().video_tracks[0].items[0].id;
s.execute("timeline.select", json!({"clips": [clip.0]})).unwrap();
s.execute("clip.multicamEnable", json!({"enabled": false})).unwrap();
assert!(psnr(&frame(&mut s, 10), &camera_frame(&s, items[2], 10)) > 40.0, "plain nest: the top track");
s.execute("clip.multicamEnable", json!({})).unwrap();
assert!(psnr(&frame(&mut s, 10), &camera_frame(&s, items[1], 10)) > 40.0, "re-enabled, angle remembered");
// the GPU plan draws the angle's media frame directly (one layer, no CPU pass) and matches
let provider = s.media.provider(s.project.clone(), s.services.clone());
let edit_id = s.state.active_sequence.unwrap();
let t10 = FrameRate::FPS_24.tick_of(10);
let plan = filmcraft_render::plan::plan_frame(&s.project, edit_id, t10, filmcraft_render::RenderOptions::default(), &provider);
match &plan {
filmcraft_render::plan::FramePlan::Layers { layers, .. } => {
assert_eq!(layers.len(), 1);
assert!(!matches!(layers[0].frame.data, filmcraft_frame::PixelData::RgbaF32(_)), "a decoded frame, not a CPU render");
}
_ => panic!("expected layers"),
}
let via_plan = filmcraft_render::plan::execute_cpu(&plan);
s.set_playhead(t10);
let reference = s.render_program(1.0).unwrap();
assert!(psnr(&via_plan.over_black_rgba8(), &reference.over_black_rgba8()) > 40.0);
// grid of the angles
let provider = s.media.provider(s.project.clone(), s.services.clone());
let (grid, angles) = filmcraft_render::multicam::render_grid(&s.project, src, Tick::ZERO, 0.25, &provider).unwrap();
assert_eq!(angles, [0, 1, 2]);
assert_eq!((grid.w, grid.h), (2 * 40, 2 * 23));
}
#[test]
fn switch_audio_follows_video() {
let (mut s, _, src, _) = mc_session("switch");
// audio follows video off (Premiere's default): only the picture switches
s.execute("multicam.switchAngle", json!({"camera": 2, "frame": 5})).unwrap();
assert_eq!((v1(&s), a1(&s)), (vec![(0, 1)], vec![(0, 0)]));
s.execute("multicam.audioFollowsVideo", json!({"enabled": true})).unwrap();
s.execute("multicam.switchAngle", json!({"camera": 3})).unwrap();
assert_eq!((v1(&s), a1(&s)), (vec![(0, 2)], vec![(0, 2)]));
// video only (Ctrl-click) leaves the audio alone; audio only switches just the audio
s.execute("multicam.switchAngle", json!({"camera": 1, "videoOnly": true})).unwrap();
assert_eq!((v1(&s), a1(&s)), (vec![(0, 0)], vec![(0, 2)]));
s.execute("multicam.switchAngle", json!({"camera": 2, "audioOnly": true})).unwrap();
assert_eq!((v1(&s), a1(&s)), (vec![(0, 0)], vec![(0, 1)]));
// switching audio plays the selected angle's track of the source
let q = s.project.sequence(src).unwrap();
let on = q.multicam.as_ref().unwrap().audible_tracks(Some(1));
assert_eq!(on, q.multicam.as_ref().unwrap().cameras[1].audio_tracks);
let provider = s.media.provider(s.project.clone(), s.services.clone());
let edit_seq = s.active_sequence().unwrap().clone();
let mix = filmcraft_render::audio::mix_sequence(&s.project, &edit_seq, 24_000, 4_800, &provider);
let only_b = filmcraft_render::audio::mix_sequence(&s.project, &q.with_angle_audio(Some(1)), 24_000, 4_800, &provider);
let err = mix.channels[0].iter().zip(&only_b.channels[0]).map(|(a, b)| (a - b).abs()).fold(0.0f32, f32::max);
assert!(err < 1e-4, "multi-camera audio = camera 2's track ({err})");
assert!(mix.channels[0].iter().any(|v| v.abs() > 1e-3), "audible");
// undo steps back one switch at a time
s.execute("edit.undo", json!({})).unwrap();
assert_eq!(a1(&s), [(0, 2)]);
}
/// Live switching from a simulated playback clock: play 0 → 84 at 24 fps, press 2 at frame 24,
/// 3 at frame 48, 3 again at 60 (no new edit), 1 at 72; stop at 84.
#[test]
fn live_switch_recording_from_playback_clock() {
let (mut s, items, _, _) = mc_session("switch");
s.execute("multicam.audioFollowsVideo", json!({"enabled": true})).unwrap();
let undo_before = s.history.undo.len();
let presses: [(i64, u64); 4] = [(24, 2), (48, 3), (60, 3), (72, 1)];
let rate = s.sequence_rate();
s.execute("multicam.recordStart", json!({"frame": 0})).unwrap();
for f in 0..=84i64 {
// the clock: one tick per frame
s.set_playhead(rate.tick_of(f));
if let Some((_, cam)) = presses.iter().find(|(pf, _)| *pf == f) {
s.execute("multicam.cut", json!({"camera": cam})).unwrap();
// the cut is live: the program shows the new angle from this frame on
let want = items[*cam as usize - 1];
assert!(psnr(&frame(&mut s, f + 2), &camera_frame(&s, want, f + 2)) > 40.0, "live cut at {f}");
s.set_playhead(rate.tick_of(f));
}
}
let r = s.execute("multicam.recordStop", json!({"frame": 84})).unwrap();
assert_eq!(r["cuts"], 4);
// 0–24 cam 1, 24–48 cam 2, 48–72 cam 3 (the second press of 3 adds no edit), 72–96 cam 1
// (healed with the untouched rest of the clip after the stop point)
let expect = vec![(0, 0), (24, 1), (48, 2), (72, 0)];
assert_eq!(v1(&s), expect);
assert_eq!(a1(&s), expect, "audio followed the video");
assert_eq!(s.history.undo.len(), undo_before + 1, "one undo step for the whole pass");
assert_eq!(s.history.undo.last().unwrap().0, "Record Multi-Camera");
// pieces are continuous in source time
let q = s.active_sequence().unwrap();
for w in q.video_tracks[0].items.windows(2) {
assert_eq!(w[0].source_out(), w[1].source_in);
assert!(w[0].link.is_some());
}
// a second pass changes only its own range and is its own undo step
s.execute("multicam.recordStart", json!({"frame": 30})).unwrap();
s.execute("multicam.cut", json!({"camera": 3, "frame": 30})).unwrap();
s.execute("multicam.recordStop", json!({"frame": 40})).unwrap();
assert_eq!(v1(&s), vec![(0, 0), (24, 1), (30, 2), (40, 1), (48, 2), (72, 0)]);
s.execute("edit.undo", json!({})).unwrap();
assert_eq!(v1(&s), expect);
s.execute("edit.undo", json!({})).unwrap();
assert_eq!(v1(&s), vec![(0, 0)]);
// stopped: a key press switches the clip at the playhead instead of cutting
s.execute("multicam.cut", json!({"camera": 2, "frame": 10})).unwrap();
assert_eq!(v1(&s), vec![(0, 1)]);
}
#[test]
fn flatten_replaces_with_camera_clips() {
let (mut s, items, _, _) = mc_session("switch");
s.execute("multicam.audioFollowsVideo", json!({"enabled": true})).unwrap();
s.execute("multicam.recordStart", json!({"frame": 0})).unwrap();
s.execute("multicam.cut", json!({"camera": 2, "frame": 30})).unwrap();
s.execute("multicam.cut", json!({"camera": 3, "frame": 60})).unwrap();
s.execute("multicam.recordStop", json!({"frame": 96})).unwrap();
let before: Vec<Vec<u8>> = [5, 35, 70].iter().map(|f| frame(&mut s, *f)).collect();
let all: Vec<u64> = s.active_sequence().unwrap().all_tracks().flat_map(|t| t.items.iter().map(|i| i.id.0)).collect();
s.execute("timeline.select", json!({"clips": all})).unwrap();
let r = s.execute("clip.multicamFlatten", json!({})).unwrap();
assert_eq!(r["flattened"], 6);
let q = s.active_sequence().unwrap().clone();
let v: Vec<(ItemId, i64, i64)> = q.video_tracks[0].items.iter().map(|i| (i.item, i.start.0, i.source_in.0)).collect();
let rate = q.settings.frame_rate;
assert_eq!(v, vec![(items[0], 0, 0), (items[1], rate.tick_of(30).0, rate.tick_of(30).0), (items[2], rate.tick_of(60).0, rate.tick_of(60).0)]);
assert!(q.all_tracks().flat_map(|t| t.items.iter()).all(|i| i.multicam.is_none()));
let a: Vec<ItemId> = q.audio_tracks[0].items.iter().map(|i| i.item).collect();
assert_eq!(a, vec![items[0], items[1], items[2]], "audio flattened to the switched cameras");
for (i, f) in [5, 35, 70].iter().enumerate() {
assert!(psnr(&before[i], &frame(&mut s, *f)) > 45.0, "flattened frame {f} looks the same");
}
// linked camera video/audio stay linked
assert_eq!(q.video_tracks[0].items[1].link, q.audio_tracks[0].items[1].link);
s.execute("edit.undo", json!({})).unwrap();
assert_eq!(v1(&s).len(), 3);
}
#[test]
fn edit_cameras_and_save_load() {
let (mut s, _, src, _) = mc_session("all");
s.execute("multicam.editCameras", json!({"sequence": src.0, "cameras": [{"angle": 1, "name": "Wide", "enabled": false}], "audio": "switch"})).unwrap();
let mc = s.project.sequence(src).unwrap().multicam.clone().unwrap();
assert_eq!((mc.cameras[1].name.as_str(), mc.cameras[1].enabled, mc.audio), ("Wide", false, MulticamAudio::SwitchAudio));
assert_eq!(mc.shown_angles(), [0, 2]);
// keys pick from the shown cameras: camera 2 is angle 2 now
s.execute("multicam.switchAngle", json!({"camera": 2})).unwrap();
assert_eq!(v1(&s), [(0, 2)]);
assert!(s.execute("multicam.switchAngle", json!({"camera": 3})).is_err());
let dir = tmp_dir("mc-save");
let path = dir.join("mc.fcproj").to_string_lossy().into_owned();
s.execute("file.save", json!({"path": path})).unwrap();
let bytes = std::fs::read(&path).unwrap();
let doc: Value = serde_json::from_slice(&bytes).unwrap();
assert_eq!(doc["schema_version"], filmcraft_format::SCHEMA_VERSION);
let mut t = Session::default();
t.execute("file.open", json!({"path": path})).unwrap();
assert_eq!(*t.project, *s.project, "multi-camera data survives save/load");
assert_eq!(t.project.sequence(src).unwrap().multicam.as_ref().unwrap().cameras[1].name, "Wide");
}
#[test]
fn any_nest_can_be_multicam() {
// a plain nested sequence with two video tracks: angles are its tracks
let (mut s, items, _, _) = mc_session("camera1");
let mut p = (*s.project).clone();
let rate = FrameRate::FPS_24;
let nest = p.new_sequence("Nest", filmcraft_project::SequenceSettings { width: W, height: H, frame_rate: rate, ..Default::default() }, 2, 0, None);
for (k, it) in items[..2].iter().enumerate() {
let mut v = p
.make_track_item(*it, filmcraft_project::TrackKind::Video, Tick::ZERO, filmcraft_time::TimeRange::new(Tick::ZERO, rate.tick_of(48)), rate)
.unwrap();
for e in &mut v.effects {
filmcraft_project::resolve_auto_points(e, (W, H), (W, H));
}
p.sequence_mut(nest).unwrap().video_tracks[k].items.push(v);
}
let edit = s.state.active_sequence.unwrap();
let mut v =
p.make_track_item(nest, filmcraft_project::TrackKind::Video, Tick::ZERO, filmcraft_time::TimeRange::new(Tick::ZERO, rate.tick_of(48)), rate).unwrap();
assert!(v.multicam.is_none(), "plain nests start as plain nests");
for e in &mut v.effects {
filmcraft_project::resolve_auto_points(e, (W, H), (W, H));
}
let clip = v.id;
let q = p.sequence_mut(edit).unwrap();
q.video_tracks[1].items.push(v);
s.project = std::sync::Arc::new(p);
assert!(psnr(&frame(&mut s, 3), &camera_frame(&s, items[1], 3)) > 40.0, "nest: top track");
s.execute("timeline.select", json!({"clips": [clip.0]})).unwrap();
s.execute("clip.multicamEnable", json!({})).unwrap();
assert!(psnr(&frame(&mut s, 3), &camera_frame(&s, items[0], 3)) > 40.0, "angle 0 = track V1 of the nest");
s.execute("multicam.switchAngle", json!({"clips": [clip.0], "angle": 1})).unwrap();
assert!(psnr(&frame(&mut s, 3), &camera_frame(&s, items[1], 3)) > 40.0);
}
#[test]
fn camera_keys_cut_and_select() {
let (mut s, _, _, _) = mc_session("camera1");
// Ctrl+2 at frame 30: an edit there, camera 2 after it
s.set_playhead(FrameRate::FPS_24.tick_of(30));
s.execute("multicam.cutToCamera2", json!({})).unwrap();
assert_eq!(v1(&s), [(0, 0), (30, 1)]);
// Ctrl+3 at the existing edit: no new edit, the right piece switches
s.execute("multicam.cutToCamera3", json!({})).unwrap();
assert_eq!(v1(&s), [(0, 0), (30, 2)]);
// 2 while stopped switches the clip under the playhead
s.set_playhead(FrameRate::FPS_24.tick_of(10));
s.execute("multicam.selectCamera2", json!({})).unwrap();
assert_eq!(v1(&s), [(0, 1), (30, 2)]);
// the keys are bound by default
let keys: Vec<(String, String)> =
s.shortcuts.bindings.iter().filter(|b| b.command.starts_with("multicam.")).map(|b| (b.command.clone(), b.keys.clone())).collect();
assert!(keys.contains(&("multicam.selectCamera1".into(), "1".into())), "{keys:?}");
assert!(keys.contains(&("multicam.cutToCamera9".into(), "Ctrl+9".into())), "{keys:?}");
}
/// Process CPU time in seconds (user + system), for the perf test.
fn process_cpu() -> f64 {
let out = Command::new("ps").args(["-o", "cputime=", "-p", &std::process::id().to_string()]).output().ok();
let txt = out.map(|o| String::from_utf8_lossy(&o.stdout).trim().to_string()).unwrap_or_default();
// [[dd-]hh:]mm:ss(.ss)
txt.split(':').fold(0.0, |acc, p| acc * 60.0 + p.trim().parse::<f64>().unwrap_or(0.0))
}
/// Multi-Camera view cost: a 2×2 grid of four 1080p H.264 angles at ¼ scale per cell, played
/// sequentially, against the program (one angle at ½ scale). Run with
/// `cargo test --release -p filmcraft-engine --lib multicam_tests::perf -- --ignored --nocapture`.
#[test]
#[ignore]
fn perf_grid_of_four_1080p_angles() {
let Some(ffmpeg) = filmcraft_testkit::ffmpeg_or_skip("multicam perf") else { return };
let dir = filmcraft_testkit::fixtures_dir("multicam");
let srcs = ["testsrc2", "smptehdbars", "mandelbrot", "rgbtestsrc"];
let files: Vec<PathBuf> = srcs
.iter()
.map(|src| {
filmcraft_testkit::fixtures::generate(&dir.join(format!("perf-{src}.mp4")), |tmp| {
let v = format!("{src}=s=1920x1080:r=24");
ff(
&ffmpeg,
&["-f", "lavfi", "-i", &v, "-t", "4", "-c:v", "libx264", "-preset", "veryfast", "-g", "48", "-pix_fmt", "yuv420p", &tmp.to_string_lossy()],
)
})
.expect("fixture")
})
.collect();
// a fresh session (empty decoder caches) per measurement
let setup = || {
let mut s = Session::default();
let refs: Vec<&Path> = files.iter().map(|p| p.as_path()).collect();
let items = import(&mut s, &refs);
let r = on_items(&mut s, "clip.createMulticam", &items, json!({"method": "in"})).unwrap();
(s, ItemId(r["sequence"].as_u64().unwrap()))
};
let rate = FrameRate::FPS_24;
let frames = 72;
let (s, src) = setup();
let provider = s.media.provider(s.project.clone(), s.services.clone());
let q = s.project.sequence(src).unwrap().clone();
let t0 = std::time::Instant::now();
for f in 0..frames {
let opts = filmcraft_render::RenderOptions { scale: 0.5, ..Default::default() };
let _ = filmcraft_render::multicam::render_angle(&s.project, &q, 0, rate.tick_of(f), opts, &provider);
}
let one_ms = t0.elapsed().as_secs_f64() * 1000.0 / frames as f64;
let (s, src) = setup();
let provider = s.media.provider(s.project.clone(), s.services.clone());
let cpu0 = crate::multicam_tests::process_cpu();
let t0 = std::time::Instant::now();
for f in 0..frames {
let (g, _) = filmcraft_render::multicam::render_grid(&s.project, src, rate.tick_of(f), 0.25, &provider).unwrap();
assert_eq!((g.w, g.h), (960, 540));
}
let grid_ms = t0.elapsed().as_secs_f64() * 1000.0 / frames as f64;
let grid_cpu = (crate::multicam_tests::process_cpu() - cpu0) * 1000.0 / frames as f64;
eprintln!(
"Multi-Camera grid (4 × 1080p H.264, ¼-scale cells): {grid_ms:.1} ms/frame wall, {grid_cpu:.1} ms/frame CPU; one angle at ½ scale: {one_ms:.1} ms/frame wall (24 fps budget 41.7 ms)"
);
}

View File

@ -35,6 +35,27 @@ pub const PREMIERE: &[Entry] = &[
("tool.hand", "H", ""), ("tool.hand", "H", ""),
("tool.zoom", "Z", ""), ("tool.zoom", "Z", ""),
("tool.type", "T", ""), ("tool.type", "T", ""),
// Multi-camera
("multicam.selectCamera1", "1", ""),
("multicam.selectCamera2", "2", ""),
("multicam.selectCamera3", "3", ""),
("multicam.selectCamera4", "4", ""),
("multicam.selectCamera5", "5", ""),
("multicam.selectCamera6", "6", ""),
("multicam.selectCamera7", "7", ""),
("multicam.selectCamera8", "8", ""),
("multicam.selectCamera9", "9", ""),
("multicam.cutToCamera1", "Ctrl+1", ""),
("multicam.cutToCamera2", "Ctrl+2", ""),
("multicam.cutToCamera3", "Ctrl+3", ""),
("multicam.cutToCamera4", "Ctrl+4", ""),
("multicam.cutToCamera5", "Ctrl+5", ""),
("multicam.cutToCamera6", "Ctrl+6", ""),
("multicam.cutToCamera7", "Ctrl+7", ""),
("multicam.cutToCamera8", "Ctrl+8", ""),
("multicam.cutToCamera9", "Ctrl+9", ""),
("multicam.recordToggle", "0", ""),
("multicam.toggleView", "Shift+0", ""),
// Application // Application
("app.preferences.autoSave", "Cmd+,", ""), ("app.preferences.autoSave", "Cmd+,", ""),
("app.keyboardShortcuts", "Cmd+Alt+K", ""), ("app.keyboardShortcuts", "Cmd+Alt+K", ""),

234
crates/engine/src/sync.rs Normal file
View File

@ -0,0 +1,234 @@
//! Synchronising clips: Clip ▸ Synchronize (timeline clips), Merge Clips and Create
//! Multi-Camera Source Sequence (project items).
//!
//! Every method reduces a clip to an **anchor**: the media time of the clip that lines up with
//! one common instant (common time = media time − anchor).
//!
//! | method | anchor |
//! |---|---|
//! | `in` | the In point (project items: the source In mark, else the start; timeline clips: their source In) |
//! | `out` | the Out point (exclusive end) |
//! | `timecode` | −(start timecode), optionally ignoring hours |
//! | `marker` | the first clip marker (or the first one named `marker`) |
//! | `audio` | from the waveform cross-correlation against the reference clip ([`filmcraft_audio_dsp::sync`]) |
//!
//! `offset` (frames) shifts every clip but the reference by that much (a known delay to correct).
use filmcraft_audio_dsp::sync::{SyncOptions, SyncResult, find_offset, mixdown};
use filmcraft_project::{ItemId, ItemKind, Project, TrackItem};
use filmcraft_time::{FrameRate, TICKS_PER_SECOND, Tick, TimeRange};
use serde_json::{Value, json};
use crate::{EngineError, Result, Session};
/// Sample rate audio is analysed at.
pub const ANALYSIS_RATE: u32 = 48_000;
/// Longest stretch of a clip analysed for audio sync (from its start).
pub const MAX_ANALYSIS: Tick = Tick(30 * 60 * TICKS_PER_SECOND);
#[derive(Clone, Debug, PartialEq)]
pub enum Method {
In,
Out,
Timecode { ignore_hours: bool },
Marker { name: Option<String> },
Audio,
}
impl Method {
pub fn from_params(p: &Value) -> std::result::Result<Method, String> {
let m = p.get("method").and_then(Value::as_str).unwrap_or("in").to_ascii_lowercase();
Ok(match m.as_str() {
"in" | "inpoint" | "inpoints" | "start" | "clipstart" => Method::In,
"out" | "outpoint" | "outpoints" | "end" | "clipend" => Method::Out,
"timecode" | "tc" => Method::Timecode { ignore_hours: p.get("ignoreHours").and_then(Value::as_bool).unwrap_or(false) },
"marker" | "clipmarker" | "markers" => Method::Marker { name: p.get("marker").and_then(Value::as_str).map(str::to_string) },
"audio" | "sound" | "waveform" => Method::Audio,
o => return Err(format!("unknown sync method `{o}` (in, out, timecode, marker, audio)")),
})
}
pub fn name(&self) -> &'static str {
match self {
Method::In => "in",
Method::Out => "out",
Method::Timecode { .. } => "timecode",
Method::Marker { .. } => "marker",
Method::Audio => "audio",
}
}
}
/// What syncing needs to know about one clip.
#[derive(Clone, Debug)]
pub struct SyncClip {
pub item: ItemId,
pub name: String,
/// Media range the clip uses.
pub range: TimeRange,
/// In / Out points (media time, exclusive Out).
pub in_point: Tick,
pub out_point: Tick,
/// Start timecode in ticks (0 when the media has none).
pub timecode: Tick,
/// Clip markers: (media time, name).
pub markers: Vec<(Tick, String)>,
}
/// The media item behind an item (subclips resolve to their parent) and its subclip range.
fn media_of(p: &Project, item: ItemId) -> Option<(ItemId, &filmcraft_project::MediaClip, Option<TimeRange>)> {
let it = p.item(item)?;
match &it.kind {
ItemKind::Media(m) => Some((item, m, None)),
ItemKind::Subclip { parent, range } => media_of(p, *parent).map(|(i, m, _)| (i, m, Some(*range))),
_ => None,
}
}
fn start_timecode(m: &filmcraft_project::MediaClip) -> Tick {
let rate = m.frame_rate();
m.info.start_timecode.map(|f| rate.tick_of(f)).unwrap_or(Tick::ZERO)
}
/// A project item as a sync clip (the whole media, or a subclip's range; In/Out marks).
pub fn project_clip(p: &Project, item: ItemId) -> Option<SyncClip> {
let it = p.item(item)?;
let (_, m, sub) = media_of(p, item)?;
let range = sub.unwrap_or(TimeRange::new(Tick::ZERO, m.duration()));
let rate = m.frame_rate();
let in_point = m.mark_in.filter(|_| sub.is_none()).unwrap_or(range.start);
let out_point = m.mark_out.filter(|_| sub.is_none()).map(|o| o + rate.frame_duration()).unwrap_or(range.end());
Some(SyncClip {
item,
name: it.name.clone(),
range,
in_point,
out_point,
timecode: start_timecode(m),
markers: m.markers.iter().map(|k| (k.start, k.name.clone())).collect(),
})
}
/// A timeline clip as a sync clip (its source range; markers of the clip and of its media).
pub fn timeline_clip(p: &Project, ti: &TrackItem) -> SyncClip {
let media = media_of(p, ti.item);
let range = TimeRange::from_bounds(ti.source_in, ti.source_out());
let mut markers: Vec<(Tick, String)> = ti.markers.iter().map(|k| (k.start, k.name.clone())).collect();
if let Some((_, m, _)) = media {
markers.extend(m.markers.iter().map(|k| (k.start, k.name.clone())));
}
SyncClip {
item: ti.item,
name: ti.name.clone(),
range,
in_point: range.start,
out_point: range.end(),
timecode: media.map(|(_, m, _)| start_timecode(m)).unwrap_or(Tick::ZERO),
markers,
}
}
/// Mono audio of a clip's media range at [`ANALYSIS_RATE`] (None: no audio).
pub fn clip_audio(s: &Session, c: &SyncClip) -> Option<Vec<f32>> {
let src = s.source(c.item)?;
if !src.info().has_audio() {
return None;
}
let sr = ANALYSIS_RATE as i64;
let start = c.range.start.to_units_floor(sr);
let len = c.range.duration.min(MAX_ANALYSIS).to_units_floor(sr).max(0) as usize;
let buf = src.audio(start, len, ANALYSIS_RATE).ok()?;
let chans: Vec<&[f32]> = buf.channels.iter().map(|c| c.as_slice()).collect();
Some(mixdown(&chans))
}
/// The outcome for one clip: its anchor, and for audio the correlation figures.
#[derive(Clone, Debug)]
pub struct Anchor {
pub anchor: Tick,
pub audio: Option<SyncResult>,
}
/// Compute the anchors of `clips` (index `reference` is the reference for audio and `offset`).
/// `offset` (ticks) moves every other clip later by that much.
pub fn anchors(s: &Session, clips: &[SyncClip], reference: usize, method: &Method, offset: Tick) -> Result<Vec<Anchor>> {
if clips.is_empty() {
return Err(EngineError::Other("nothing to synchronize".into()));
}
let hour = Tick(3600 * TICKS_PER_SECOND);
let mut out: Vec<Anchor> = Vec::with_capacity(clips.len());
let ref_audio = if *method == Method::Audio {
Some(clip_audio(s, &clips[reference]).ok_or_else(|| EngineError::Other(format!("{} has no audio to synchronize by", clips[reference].name)))?)
} else {
None
};
for (i, c) in clips.iter().enumerate() {
let a = match method {
Method::In => Anchor { anchor: c.in_point, audio: None },
Method::Out => Anchor { anchor: c.out_point, audio: None },
Method::Timecode { ignore_hours } => {
let tc = if *ignore_hours { Tick(c.timecode.0.rem_euclid(hour.0)) } else { c.timecode };
Anchor { anchor: Tick::ZERO - tc, audio: None }
}
Method::Marker { name } => {
let m =
c.markers.iter().filter(|(_, n)| name.as_deref().is_none_or(|w| n.eq_ignore_ascii_case(w))).map(|(t, _)| *t).min().ok_or_else(|| {
EngineError::Other(format!("{} has no clip marker{}", c.name, name.as_ref().map(|n| format!(" named “{n}”")).unwrap_or_default()))
})?;
Anchor { anchor: m, audio: None }
}
Method::Audio => {
if i == reference {
Anchor { anchor: c.range.start, audio: None }
} else {
let a = ref_audio.as_deref().unwrap_or_default();
let b = clip_audio(s, c).ok_or_else(|| EngineError::Other(format!("{} has no audio to synchronize by", c.name)))?;
let r = find_offset(a, &b, ANALYSIS_RATE, &SyncOptions::default())
.ok_or_else(|| EngineError::Other(format!("{} and {} have no common audio", clips[reference].name, c.name)))?;
// b[n] (media r0_i + n) lines up with a[n + L] (media r0_ref + n + L); with the
// reference anchored at r0_ref, equal common times give anchor_i = r0_i − L
let lag = Tick::from_units(r.lag, ANALYSIS_RATE as i64);
Anchor { anchor: c.range.start - lag, audio: Some(r) }
}
}
};
out.push(a);
}
if offset != Tick::ZERO {
for (i, a) in out.iter_mut().enumerate() {
if i != reference {
// later on the common timeline = smaller anchor
a.anchor -= offset;
}
}
}
Ok(out)
}
/// Where each clip's range starts on a common timeline that begins at 0 (the earliest clip), at
/// the given frame rate (snapped to frames when `snap`).
pub fn placements(clips: &[SyncClip], anchors: &[Anchor], rate: FrameRate, snap: bool) -> Vec<Tick> {
let raw: Vec<Tick> = clips.iter().zip(anchors).map(|(c, a)| c.range.start - a.anchor).collect();
let min = raw.iter().copied().min().unwrap_or(Tick::ZERO);
raw.into_iter().map(|t| if snap { rate.snap_nearest(t - min) } else { t - min }).collect()
}
/// JSON report of one sync for agents.
pub fn report(clips: &[SyncClip], anchors: &[Anchor], starts: &[Tick], rate: FrameRate) -> Value {
Value::Array(
clips
.iter()
.zip(anchors)
.zip(starts)
.map(|((c, a), st)| {
json!({
"item": c.item.0,
"name": c.name,
"start": st.0,
"startFrame": rate.frame_at(*st),
"anchor": a.anchor.0,
"audio": a.audio.map(|r| json!({"lagSamples": r.lag, "lagFine": r.lag_fine, "confidence": r.confidence, "distinct": r.distinct, "reliable": r.reliable(), "inverted": r.inverted})),
})
})
.collect(),
)
}

View File

@ -27,6 +27,9 @@
//! - **v5** (M8.8): colour management (sequence working space, Interpret Footage colour space, //! - **v5** (M8.8): colour management (sequence working space, Interpret Footage colour space,
//! project LUT library). No-op step, for the same reason. //! project LUT library). No-op step, for the same reason.
//! - **v6** (M11.7/M11.9): media identity fingerprints and ingest settings. No-op step. //! - **v6** (M11.7/M11.9): media identity fingerprints and ingest settings. No-op step.
//! - **v7** (M5.5): effect masks (keyframable Bézier mask paths on effect instances). No-op step.
//! - **v8** (M12): multi-camera source sequences, multi-camera clips and merged clips. No-op step;
//! older builds would drop the camera data when saving.
pub mod atomic; pub mod atomic;
pub mod autosave; pub mod autosave;
@ -45,7 +48,7 @@ pub type Migration = fn(Value) -> Result<Value, String>;
/// `MIGRATIONS[i]` upgrades schema `i + 1` to `i + 2`. Append one function per schema bump; never /// `MIGRATIONS[i]` upgrades schema `i + 1` to `i + 2`. Append one function per schema bump; never
/// edit a shipped one. /// edit a shipped one.
pub const MIGRATIONS: &[Migration] = &[v1_to_v2, v2_to_v3, v3_to_v4, v4_to_v5, v5_to_v6, v6_to_v7]; pub const MIGRATIONS: &[Migration] = &[v1_to_v2, v2_to_v3, v3_to_v4, v4_to_v5, v5_to_v6, v6_to_v7, v7_to_v8];
/// The schema version this build writes (and the newest it reads). /// The schema version this build writes (and the newest it reads).
pub const SCHEMA_VERSION: u32 = MIGRATIONS.len() as u32 + 1; pub const SCHEMA_VERSION: u32 = MIGRATIONS.len() as u32 + 1;
@ -244,13 +247,19 @@ fn v6_to_v7(doc: Value) -> Result<Value, String> {
Ok(doc) Ok(doc)
} }
/// v7 → v8: multi-camera source sequences / clips and merged clips added; existing data needs no
/// change.
fn v7_to_v8(doc: Value) -> Result<Value, String> {
Ok(doc)
}
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::*; use super::*;
#[test] #[test]
fn schema_version_matches_table() { fn schema_version_matches_table() {
assert_eq!(SCHEMA_VERSION, 7); assert_eq!(SCHEMA_VERSION, 8);
} }
#[test] #[test]

View File

@ -425,6 +425,7 @@ impl Builder {
frame_hold: None, frame_hold: None,
scale_to_frame: false, scale_to_frame: false,
essential: None, essential: None,
multicam: None,
} }
} }
@ -483,6 +484,8 @@ pub(crate) fn empty_sequence(settings: SequenceSettings) -> Sequence {
master_mixer: Default::default(), master_mixer: Default::default(),
submix_tracks: Vec::new(), submix_tracks: Vec::new(),
caption_tracks: Vec::new(), caption_tracks: Vec::new(),
multicam: None,
merged: None,
} }
} }

View File

@ -778,6 +778,7 @@ mod tests {
frame_hold: None, frame_hold: None,
scale_to_frame: false, scale_to_frame: false,
essential: None, essential: None,
multicam: None,
} }
} }

View File

@ -14,6 +14,7 @@ pub mod graphic;
pub mod keyframe; pub mod keyframe;
pub mod mask; pub mod mask;
pub mod mixer; pub mod mixer;
pub mod multicam;
use std::collections::BTreeMap; use std::collections::BTreeMap;
@ -28,6 +29,7 @@ pub use essential::{AudioType, EssentialSound};
pub use keyframe::{Interpolation, Keyframe, Param, ParamValue}; pub use keyframe::{Interpolation, Keyframe, Param, ParamValue};
pub use mask::{Mask, MaskMode, MaskPath, MaskVertex, TrackMethod}; pub use mask::{Mask, MaskMode, MaskPath, MaskVertex, TrackMethod};
pub use mixer::{AutomationMode, InputMap, MixerStrip, TrackSend}; pub use mixer::{AutomationMode, InputMap, MixerStrip, TrackSend};
pub use multicam::{Camera, MergedClip, MulticamAudio, MulticamSel, MulticamSource};
macro_rules! id_type { macro_rules! id_type {
($name:ident) => { ($name:ident) => {
@ -320,6 +322,8 @@ impl ProjectItem {
filmcraft_media::MediaKind::ImageSequence => "Image Sequence", filmcraft_media::MediaKind::ImageSequence => "Image Sequence",
filmcraft_media::MediaKind::Synthetic => "Synthetic", filmcraft_media::MediaKind::Synthetic => "Synthetic",
}, },
ItemKind::Sequence(s) if s.multicam.is_some() => "Multi-Camera Source Sequence",
ItemKind::Sequence(s) if s.merged.is_some() => "Merged Clip",
ItemKind::Sequence(_) => "Sequence", ItemKind::Sequence(_) => "Sequence",
ItemKind::Subclip { .. } => "Subclip", ItemKind::Subclip { .. } => "Subclip",
ItemKind::AdjustmentLayer { .. } => "Adjustment Layer", ItemKind::AdjustmentLayer { .. } => "Adjustment Layer",
@ -470,6 +474,9 @@ pub struct TrackItem {
/// Essential Sound audio type and settings (audio clips). /// Essential Sound audio type and settings (audio clips).
#[serde(default, skip_serializing_if = "Option::is_none")] #[serde(default, skip_serializing_if = "Option::is_none")]
pub essential: Option<EssentialSound>, pub essential: Option<EssentialSound>,
/// Multi-camera clip (a nested multi-camera source sequence): enabled flag and angle.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub multicam: Option<MulticamSel>,
} }
impl TrackItem { impl TrackItem {
@ -727,6 +734,12 @@ pub struct Sequence {
/// Caption tracks (drawn above the video tracks; first = top). Older files have none (serde default). /// Caption tracks (drawn above the video tracks; first = top). Older files have none (serde default).
#[serde(default)] #[serde(default)]
pub caption_tracks: Vec<CaptionTrack>, pub caption_tracks: Vec<CaptionTrack>,
/// Set on multi-camera source sequences (cameras, audio mode).
#[serde(default, skip_serializing_if = "Option::is_none")]
pub multicam: Option<MulticamSource>,
/// Set on merged clips (the merged video and audio items).
#[serde(default, skip_serializing_if = "Option::is_none")]
pub merged: Option<MergedClip>,
} }
impl Sequence { impl Sequence {
@ -961,6 +974,8 @@ impl Project {
master_mixer: MixerStrip::default(), master_mixer: MixerStrip::default(),
submix_tracks: Vec::new(), submix_tracks: Vec::new(),
caption_tracks: Vec::new(), caption_tracks: Vec::new(),
multicam: None,
merged: None,
}; };
for i in 0..v { for i in 0..v {
let id = TrackId(self.alloc_id()); let id = TrackId(self.alloc_id());
@ -1025,6 +1040,9 @@ impl Project {
TrackKind::Video => effect::intrinsic_video(), TrackKind::Video => effect::intrinsic_video(),
TrackKind::Audio => effect::intrinsic_audio(), TrackKind::Audio => effect::intrinsic_audio(),
}; };
// a multi-camera source sequence edits in as a multi-camera clip showing its first angle
let multicam =
it.as_sequence().and_then(|q| q.multicam.as_ref()).map(|m| MulticamSel { enabled: true, angle: m.first_video_angle().unwrap_or(0) as u32 });
let id = ClipId(self.alloc_id()); let id = ClipId(self.alloc_id());
let dur = seq_rate.snap_nearest(source.duration).max(seq_rate.frame_duration()); let dur = seq_rate.snap_nearest(source.duration).max(seq_rate.frame_duration());
Some(TrackItem { Some(TrackItem {
@ -1046,6 +1064,7 @@ impl Project {
frame_hold: None, frame_hold: None,
scale_to_frame: false, scale_to_frame: false,
essential: None, essential: None,
multicam,
}) })
} }

View File

@ -0,0 +1,201 @@
//! Multi-camera source sequences and merged clips.
//!
//! A **multi-camera source sequence** is an ordinary [`Sequence`](crate::Sequence) whose
//! [`Sequence::multicam`](crate::Sequence::multicam) describes its cameras: each camera (angle) is
//! one video track plus its audio tracks, with the clips already synchronised. Edited into another
//! sequence it is a nested clip; with [`TrackItem::multicam`](crate::TrackItem::multicam) enabled
//! the clip shows only the selected angle's video track (video clips) or plays the selected
//! angle's audio (audio clips, when the source switches audio), instead of the composite.
//!
//! A **merged clip** is a sequence too ([`Sequence::merged`](crate::Sequence::merged)): one video
//! clip and up to 16 separately recorded audio clips, synchronised, used like a single clip.
use serde::{Deserialize, Serialize};
use crate::{ItemId, Sequence, TrackId};
/// How a multi-camera source sequence plays audio (Create Multi-Camera Source Sequence ▸ Audio).
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Serialize, Deserialize)]
pub enum MulticamAudio {
/// The audio of camera 1 (plus audio-only clips); the other cameras' audio tracks are muted.
#[default]
Camera1,
/// Every camera's audio, mixed.
AllCameras,
/// The audio of the selected angle (plus audio-only clips): multi-camera audio clips switch
/// with their angle.
SwitchAudio,
}
impl MulticamAudio {
pub fn from_name(s: &str) -> Option<Self> {
match s.to_ascii_lowercase().replace([' ', '_', '-'], "").as_str() {
"camera1" | "cam1" => Some(Self::Camera1),
"all" | "allcameras" => Some(Self::AllCameras),
"switch" | "switchaudio" => Some(Self::SwitchAudio),
_ => None,
}
}
pub fn name(self) -> &'static str {
match self {
Self::Camera1 => "camera1",
Self::AllCameras => "all",
Self::SwitchAudio => "switch",
}
}
}
/// One camera angle of a multi-camera source sequence.
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
pub struct Camera {
pub name: String,
/// The angle's video track (None for an audio-only source, which is never an angle of the
/// grid but is mixed in).
pub video_track: Option<TrackId>,
/// The angle's audio tracks.
#[serde(default)]
pub audio_tracks: Vec<TrackId>,
/// Shown in the Multi-Camera view (Edit Cameras).
#[serde(default = "yes")]
pub enabled: bool,
/// The project item the angle was made from.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub source: Option<ItemId>,
}
fn yes() -> bool {
true
}
/// [`Sequence::multicam`](crate::Sequence::multicam): what makes a sequence a multi-camera source.
#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)]
pub struct MulticamSource {
/// Angles in camera order (angle `n` = `cameras[n]`). Audio-only sources come last and have no
/// video track.
pub cameras: Vec<Camera>,
pub audio: MulticamAudio,
/// How the clips were synchronised (`in`, `out`, `timecode`, `marker`, `audio`), informational.
#[serde(default)]
pub sync: String,
}
impl MulticamSource {
/// Angles that have a video track (the grid), in camera order: `(angle, camera)`.
pub fn video_angles(&self) -> impl Iterator<Item = (usize, &Camera)> {
self.cameras.iter().enumerate().filter(|(_, c)| c.video_track.is_some())
}
/// Enabled video angles: the cameras shown in the Multi-Camera view (keys 1–9 pick from these).
pub fn shown_angles(&self) -> Vec<usize> {
self.video_angles().filter(|(_, c)| c.enabled).map(|(i, _)| i).collect()
}
/// Angle whose video is on `track`.
pub fn angle_of_video_track(&self, track: TrackId) -> Option<usize> {
self.cameras.iter().position(|c| c.video_track == Some(track))
}
/// Whether an audio track belongs to a camera with video (switchable) rather than to an
/// audio-only source.
pub fn camera_of_audio_track(&self, track: TrackId) -> Option<usize> {
self.cameras.iter().position(|c| c.audio_tracks.contains(&track))
}
/// The audio tracks that play for `angle` (None = all of them, as in a plain nest).
/// Audio-only sources always play; camera audio follows [`MulticamAudio`].
pub fn audible_tracks(&self, angle: Option<usize>) -> Vec<TrackId> {
let mut v = Vec::new();
for (i, c) in self.cameras.iter().enumerate() {
let audio_only = c.video_track.is_none();
let on = audio_only
|| match self.audio {
MulticamAudio::Camera1 => i == self.first_video_angle().unwrap_or(0),
MulticamAudio::AllCameras => true,
MulticamAudio::SwitchAudio => Some(i) == angle.or(self.first_video_angle()),
};
if on {
v.extend(c.audio_tracks.iter().copied());
}
}
v
}
pub fn first_video_angle(&self) -> Option<usize> {
self.video_angles().map(|(i, _)| i).next()
}
}
/// [`TrackItem::multicam`](crate::TrackItem::multicam): the angle a multi-camera clip shows.
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct MulticamSel {
/// Multi-Camera ▸ Enable. Off: the clip plays like any nested sequence.
pub enabled: bool,
/// Camera index into [`MulticamSource::cameras`] (video clips: the picture; audio clips: the
/// sound when the source switches audio).
pub angle: u32,
}
/// [`Sequence::merged`](crate::Sequence::merged): what makes a sequence a merged clip.
#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)]
pub struct MergedClip {
/// The video clip (None: audio-only merge).
pub video: Option<ItemId>,
pub audio: Vec<ItemId>,
/// How the clips were synchronised.
#[serde(default)]
pub sync: String,
}
impl crate::TrackItem {
/// The angle this clip shows when it is an enabled multi-camera clip, else None (it plays as an
/// ordinary nest). Any nested sequence can be a multi-camera clip: without camera data its
/// video tracks are the angles.
pub fn multicam_angle(&self, _nested: &Sequence) -> Option<usize> {
self.multicam.filter(|m| m.enabled).map(|m| m.angle as usize)
}
}
impl Sequence {
/// The cameras of this sequence used as a multi-camera source: its camera data, or for a plain
/// sequence one camera per video track (named after the track) with all audio mixed.
pub fn cameras(&self) -> MulticamSource {
if let Some(m) = &self.multicam {
return m.clone();
}
MulticamSource {
cameras: self
.video_tracks
.iter()
.enumerate()
.map(|(i, t)| Camera {
name: t.name.clone(),
video_track: Some(t.id),
audio_tracks: self.audio_tracks.get(i).map(|a| vec![a.id]).unwrap_or_default(),
enabled: true,
source: None,
})
.collect(),
audio: MulticamAudio::AllCameras,
sync: String::new(),
}
}
/// Video track index of `angle` when this sequence is used as a multi-camera source.
pub fn angle_video_track_index(&self, angle: usize) -> Option<usize> {
match &self.multicam {
Some(mc) => {
let id = mc.cameras.get(angle)?.video_track?;
self.video_tracks.iter().position(|t| t.id == id)
}
None => (angle < self.video_tracks.len()).then_some(angle),
}
}
/// A copy whose audio tracks are muted except those audible for `angle` (None = the source's
/// default: camera 1, all, or the first angle when switching). Plain sequences are unchanged.
pub fn with_angle_audio(&self, angle: Option<usize>) -> Sequence {
let mut q = self.clone();
if let Some(mc) = &self.multicam {
let on = mc.audible_tracks(angle);
for t in &mut q.audio_tracks {
if !on.contains(&t.id) {
t.muted = true;
}
}
}
q
}
}

View File

@ -76,16 +76,6 @@ pub fn pan_gains(pan: f32) -> (f32, f32) {
#[allow(clippy::too_many_arguments)] #[allow(clippy::too_many_arguments)]
fn mix_item(project: &Project, item: &TrackItem, start: i64, frames: usize, sr: u32, sources: &dyn SourceProvider, out: &mut AudioBuffer, extra: f32) { fn mix_item(project: &Project, item: &TrackItem, start: i64, frames: usize, sr: u32, sources: &dyn SourceProvider, out: &mut AudioBuffer, extra: f32) {
let Some(src) = sources.source(item.item) else { return };
if !src.info().has_audio() {
// nested sequences
if let Some(nested) = project.sequence(item.item) {
let rel0 = start - item.start.to_units_floor(sr as i64) + item.source_in.to_units_floor(sr as i64);
let b = mix_sequence(project, nested, rel0, frames, sources);
out.mix_from(&b, &[extra, extra]);
}
return;
}
let item_s0 = item.start.to_units_floor(sr as i64); let item_s0 = item.start.to_units_floor(sr as i64);
let item_s1 = item.end().to_units_floor(sr as i64); let item_s1 = item.end().to_units_floor(sr as i64);
let a0 = start.max(item_s0); let a0 = start.max(item_s0);
@ -97,7 +87,15 @@ fn mix_item(project: &Project, item: &TrackItem, start: i64, frames: usize, sr:
return; return;
} }
let n = (a1 - a0) as usize; let n = (a1 - a0) as usize;
let buf = effected(item, src.as_ref(), a0, n, sr); let buf = if let Some(nested) = project.sequence(item.item) {
nested_audio(project, item, nested, a0, n, sr, sources)
} else {
let Some(src) = sources.source(item.item) else { return };
if !src.info().has_audio() {
return;
}
effected(item, src.as_ref(), a0, n, sr)
};
// gains: clip gain × Volume (keyframed, per 64-sample block) × channel volume × panner // gains: clip gain × Volume (keyframed, per 64-sample block) × channel volume × panner
let clip_gain = db_to_gain(item.gain_db); let clip_gain = db_to_gain(item.gain_db);
let vol = item.effect("volume").filter(|e| e.enabled && !e.param("bypass").and_then(|p| p.value.as_bool()).unwrap_or(false)); let vol = item.effect("volume").filter(|e| e.enabled && !e.param("bypass").and_then(|p| p.value.as_bool()).unwrap_or(false));
@ -124,6 +122,23 @@ fn mix_item(project: &Project, item: &TrackItem, start: i64, frames: usize, sr:
} }
} }
/// A nested sequence's mix for timeline samples `[a0, a0 + n)` of the clip that shows it. A
/// multi-camera clip plays the audio of its source's audio setting: camera 1, all cameras, or (when
/// switching audio) the angle the clip selects.
fn nested_audio(project: &Project, item: &TrackItem, nested: &Sequence, a0: i64, n: usize, sr: u32, sources: &dyn SourceProvider) -> [Vec<f32>; 2] {
let rel0 = a0 - item.start.to_units_floor(sr as i64) + item.source_in.to_units_floor(sr as i64);
let b = if nested.multicam.is_some() {
let q = nested.with_angle_audio(item.multicam_angle(nested));
mix_sequence(project, &q, rel0, n, sources)
} else {
mix_sequence(project, nested, rel0, n, sources)
};
let mut it = b.channels.into_iter();
let l = it.next().unwrap_or_else(|| vec![0.0; n]);
let r = it.next().unwrap_or_else(|| l.clone());
[l, r]
}
/// The clip's audio after speed/reverse and clip effects (before clip gain, Volume and Panner) for /// The clip's audio after speed/reverse and clip effects (before clip gain, Volume and Panner) for
/// timeline samples `[a0, a0 + n)`. /// timeline samples `[a0, a0 + n)`.
fn effected(item: &TrackItem, src: &dyn filmcraft_media::MediaSource, a0: i64, n: usize, sr: u32) -> [Vec<f32>; 2] { fn effected(item: &TrackItem, src: &dyn filmcraft_media::MediaSource, a0: i64, n: usize, sr: u32) -> [Vec<f32>; 2] {

View File

@ -19,6 +19,7 @@ pub mod image;
pub mod luts; pub mod luts;
pub mod mask; pub mod mask;
pub mod mixer; pub mod mixer;
pub mod multicam;
pub mod offline; pub mod offline;
pub mod plan; pub mod plan;
pub mod preview; pub mod preview;
@ -80,18 +81,24 @@ pub fn render_sequence(project: &Project, seq_id: ItemId, t: Tick, opts: RenderO
} }
fn render_seq(project: &Project, seq: &Sequence, t: Tick, opts: RenderOptions, sources: &dyn SourceProvider) -> Image { fn render_seq(project: &Project, seq: &Sequence, t: Tick, opts: RenderOptions, sources: &dyn SourceProvider) -> Image {
render_seq_tracks(project, seq, t, opts, sources, None)
}
/// Render a sequence, or only its video track `only` (a multi-camera angle; drawn even when the
/// track's output is off).
pub(crate) fn render_seq_tracks(project: &Project, seq: &Sequence, t: Tick, opts: RenderOptions, sources: &dyn SourceProvider, only: Option<usize>) -> Image {
let (w, h) = output_size(seq, opts.scale); let (w, h) = output_size(seq, opts.scale);
let mut canvas = Image::new(w, h); let mut canvas = Image::new(w, h);
if opts.depth > 8 { if opts.depth > 8 {
return canvas; return canvas;
} }
let tc = format_time(t, seq.settings.frame_rate, seq.settings.drop_frame, TimeDisplay::Timecode, seq.settings.sample_rate as i64); let tc = format_time(t, seq.settings.frame_rate, seq.settings.drop_frame, TimeDisplay::Timecode, seq.settings.sample_rate as i64);
for track in &seq.video_tracks { for (ti, track) in seq.video_tracks.iter().enumerate() {
// A cancelled frame job (playback moved on) stops here; its result is discarded. // A cancelled frame job (playback moved on) stops here; its result is discarded.
if filmcraft_media::cancel::cancelled() { if filmcraft_media::cancel::cancelled() {
return canvas; return canvas;
} }
if !track.enabled { if only.is_some_and(|o| o != ti) || (!track.enabled && only.is_none()) {
continue; continue;
} }
// transition covering t? // transition covering t?
@ -285,7 +292,17 @@ pub(crate) fn item_layer(
} }
ItemKind::Sequence(nested) => { ItemKind::Sequence(nested) => {
let sub = RenderOptions { scale: want, effects: opts.effects, depth: opts.depth + 1, captions: false, working_output: true }; let sub = RenderOptions { scale: want, effects: opts.effects, depth: opts.depth + 1, captions: false, working_output: true };
render_seq(project, nested, mt, sub, sources) match item.multicam_angle(nested) {
// a multi-camera clip shows its angle's track only (nothing for an audio-only angle)
Some(angle) => match nested.angle_video_track_index(angle) {
Some(ti) => render_seq_tracks(project, nested, mt, sub, sources, Some(ti)),
None => {
let (nw, nh) = output_size(nested, want);
Image::new(nw, nh)
}
},
None => render_seq(project, nested, mt, sub, sources),
}
} }
ItemKind::AdjustmentLayer { .. } => return None, ItemKind::AdjustmentLayer { .. } => return None,
ItemKind::Graphic { .. } => { ItemKind::Graphic { .. } => {

View File

@ -0,0 +1,61 @@
//! Multi-camera rendering: one angle of a multi-camera source sequence, and the Multi-Camera
//! view's grid of angles.
//!
//! The grid renders every shown angle at a reduced scale (sources are asked for small frames, so
//! YUV→RGB conversion and compositing run at the cell size) in parallel, and tiles them into one
//! image. A 2×2 grid of 1080p angles at ¼ scale costs about as much as one ½-resolution frame of
//! compositing plus the four decodes.
use filmcraft_project::{ItemId, Project, Sequence};
use filmcraft_time::Tick;
use rayon::prelude::*;
use crate::{Image, RenderOptions, SourceProvider, output_size};
/// Grid layout for `n` angles: (columns, rows). 1 → 1×1, 2–4 → 2×2, 5–9 → 3×3, 10–16 → 4×4.
pub fn grid_dims(n: usize) -> (usize, usize) {
let c = (1..=4).find(|c| c * c >= n).unwrap_or(4);
let r = n.div_ceil(c).max(1);
(c, r.min(c))
}
/// Render angle `angle` of the multi-camera source sequence `seq` at its time `t` (transparent
/// for an audio-only angle). The image is in the sequence's working space unless `opts.depth == 0`
/// and `!opts.working_output` (then display-ready like [`crate::render_sequence`]).
pub fn render_angle(project: &Project, seq: &Sequence, angle: usize, t: Tick, opts: RenderOptions, sources: &dyn SourceProvider) -> Image {
match seq.angle_video_track_index(angle) {
Some(ti) => crate::render_seq_tracks(project, seq, t, opts, sources, Some(ti)),
None => {
let (w, h) = output_size(seq, opts.scale);
Image::new(w, h)
}
}
}
/// The Multi-Camera view's angle grid of the sequence `item` used as a multi-camera source at its time `t`: the shown
/// angles (Edit Cameras), each at `cell_scale` of the sequence frame size, tiled left to right,
/// top to bottom over black. Returns the image and the angles in grid order.
pub fn render_grid(project: &Project, item: ItemId, t: Tick, cell_scale: f32, sources: &dyn SourceProvider) -> Option<(Image, Vec<usize>)> {
let seq = project.sequence(item)?;
let angles = seq.cameras().shown_angles();
let (cols, rows) = grid_dims(angles.len().max(1));
let (cw, ch) = output_size(seq, cell_scale);
let opts = RenderOptions { scale: cell_scale, ..Default::default() };
let cells: Vec<Image> = angles.par_iter().map(|&a| render_angle(project, seq, a, t, opts, sources)).collect();
let mut out = Image::filled(cw * cols, ch * rows, [0.0, 0.0, 0.0, 1.0]);
for (k, cell) in cells.iter().enumerate().take(cols * rows) {
let (x0, y0) = ((k % cols) * cw, (k / cols) * ch);
for y in 0..ch.min(cell.h) {
for x in 0..cw.min(cell.w) {
let s = (y * cell.w + x) * 4;
let d = ((y0 + y) * out.w + x0 + x) * 4;
// cells over black: premultiplied over an opaque background
let a = cell.px[s + 3];
for c in 0..3 {
out.px[d + c] = cell.px[s + c] + out.px[d + c] * (1.0 - a);
}
}
}
}
Some((out, angles))
}

View File

@ -145,6 +145,22 @@ fn push_item(
) { ) {
let mt = item.source_time_at(t); let mt = item.source_time_at(t);
let (op, bl) = crate::opacity_blend(item, mt); let (op, bl) = crate::opacity_blend(item, mt);
// A multi-camera clip that only shows its angle (no effects, untransformed, same frame size)
// draws the angle's clip directly: no CPU pass over the nested sequence.
if let Some(ItemKind::Sequence(nested)) = project.item(item.item).map(|p| &p.kind)
&& let Some(angle) = item.multicam_angle(nested)
&& bl == Blend::Normal
&& !(opts.effects && item.has_standard_effects())
&& (nested.settings.width, nested.settings.height) == (seq.settings.width, seq.settings.height)
&& near_identity(&motion_matrix(seq, item, (nested.settings.width, nested.settings.height), mt))
&& let Some(tr) = nested.angle_video_track_index(angle).and_then(|i| nested.video_tracks.get(i))
&& !tr.transitions.iter().any(|x| x.range().contains(mt))
{
if let Some(inner) = tr.item_at(mt).filter(|i| i.enabled) {
push_item(project, nested, inner, mt, opts, sources, extra_opacity * op, out);
}
return;
}
// Graphic clips without standard effects: the layers are rasterised (cached) into one tight // Graphic clips without standard effects: the layers are rasterised (cached) into one tight
// image the GPU places as a layer. // image the GPU places as a layer.
if bl == Blend::Normal if bl == Blend::Normal
@ -183,6 +199,10 @@ fn push_item(
} }
} }
fn near_identity(m: &Affine) -> bool {
(m.a - 1.0).abs() < 1e-9 && (m.d - 1.0).abs() < 1e-9 && m.b.abs() < 1e-9 && m.c.abs() < 1e-9 && m.e.abs() < 1e-6 && m.f.abs() < 1e-6
}
/// Execute a plan on the CPU (reference for the GPU compositor). /// Execute a plan on the CPU (reference for the GPU compositor).
pub fn execute_cpu(plan: &FramePlan) -> crate::Image { pub fn execute_cpu(plan: &FramePlan) -> crate::Image {
match plan { match plan {

View File

@ -26,6 +26,9 @@ pub enum Target {
Item(ItemId), Item(ItemId),
/// A GPU frame plan of a sequence (decoded layers + transforms), composited on the GPU. /// A GPU frame plan of a sequence (decoded layers + transforms), composited on the GPU.
SequencePlan(ItemId), SequencePlan(ItemId),
/// The Multi-Camera view's angle grid of a multi-camera source at its time (all shown angles,
/// each at the job's scale).
MulticamGrid(ItemId),
} }
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] #[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
@ -707,6 +710,7 @@ fn render_job(job: &Job, pool: &Arc<MediaPool>, services: &Arc<dyn Services>, pr
let img = match job.key.target { let img = match job.key.target {
Target::Sequence(seq) | Target::SequencePlan(seq) => Some(filmcraft_render::render_sequence(&job.project, seq, job.time, opts, &provider)), Target::Sequence(seq) | Target::SequencePlan(seq) => Some(filmcraft_render::render_sequence(&job.project, seq, job.time, opts, &provider)),
Target::Item(item) => filmcraft_render::render_item(&job.project, item, job.time, job.scale, &provider), Target::Item(item) => filmcraft_render::render_item(&job.project, item, job.time, job.scale, &provider),
Target::MulticamGrid(item) => filmcraft_render::multicam::render_grid(&job.project, item, job.time, job.scale, &provider).map(|(img, _)| img),
}; };
let rgba = match img { let rgba = match img {
Some(img) => Rgba { w: img.w, h: img.h, px: img.over_black_rgba8() }, Some(img) => Rgba { w: img.w, h: img.h, px: img.over_black_rgba8() },

View File

@ -338,6 +338,8 @@ impl FilmcraftApp {
let t = self.session.playhead(); let t = self.session.playhead();
let _ = self.session.execute("mixer.recordStart", json!({"time": t.0})); let _ = self.session.execute("mixer.recordStart", json!({"time": t.0}));
} }
// Multi-Camera view: playing records live cuts (keys 1–9 / clicking angles)
panels::multicam::on_play(self);
} }
pub fn stop(&mut self) { pub fn stop(&mut self) {
@ -355,6 +357,7 @@ impl FilmcraftApp {
self.ui.status = e.to_string(); self.ui.status = e.to_string();
} }
} }
panels::multicam::on_stop(self);
} }
fn start_audio(&mut self) { fn start_audio(&mut self) {

View File

@ -39,6 +39,8 @@ pub const UI_COMMANDS: &[UiCommand] = &[
uic!("view.playbackRes.quarter", "1/4", ["View", "Playback Resolution"], None), uic!("view.playbackRes.quarter", "1/4", ["View", "Playback Resolution"], None),
uic!("view.playbackRes.eighth", "1/8", ["View", "Playback Resolution"], None), uic!("view.playbackRes.eighth", "1/8", ["View", "Playback Resolution"], None),
uic!("view.safeMargins", "Safe Margins", ["View"], None), uic!("view.safeMargins", "Safe Margins", ["View"], None),
uic!("multicam.toggleView", "Multi-Camera View", ["View"], Some("Shift+0")),
uic!("multicam.recordToggle", "Multi-Camera Record On/Off Toggle", [], Some("0")),
uic!("view.theme.dark", "Darkest", ["View", "Appearance"], None), uic!("view.theme.dark", "Darkest", ["View", "Appearance"], None),
uic!("view.theme.medium", "Medium", ["View", "Appearance"], None), uic!("view.theme.medium", "Medium", ["View", "Appearance"], None),
uic!("view.theme.light", "Light", ["View", "Appearance"], None), uic!("view.theme.light", "Light", ["View", "Appearance"], None),
@ -100,6 +102,12 @@ pub fn invoke(app: &mut FilmcraftApp, ctx: &egui::Context, id: &str, params: Val
if let Some(r) = crate::panels::trim_monitor::route_transport(app, ctx, id) { if let Some(r) = crate::panels::trim_monitor::route_transport(app, ctx, id) {
return r; return r;
} }
if let Some(r) = crate::panels::multicam::route(app, id, &params) {
if let Err(e) = &r {
app.ui.status = e.clone();
}
return r;
}
if let Some(r) = crate::panels::media_dialogs::route(app, id, &params) { if let Some(r) = crate::panels::media_dialogs::route(app, id, &params) {
if let Err(e) = &r { if let Err(e) = &r {
app.ui.status = e.clone(); app.ui.status = e.clone();

View File

@ -7,6 +7,7 @@ pub fn show(app: &mut FilmcraftApp, ctx: &egui::Context) {
crate::panels::media_dialogs::show(app, ctx); crate::panels::media_dialogs::show(app, ctx);
crate::panels::color_dialogs::show(app, ctx); crate::panels::color_dialogs::show(app, ctx);
crate::panels::presets::save_dialog(app, ctx); crate::panels::presets::save_dialog(app, ctx);
crate::panels::multicam::show_dialog(app, ctx);
let Some(d) = app.dialog else { return }; let Some(d) = app.dialog else { return };
if let Some(still_open) = crate::panels::file_dialogs::show(app, ctx, d) { if let Some(still_open) = crate::panels::file_dialogs::show(app, ctx, d) {
if !still_open && app.dialog == Some(d) { if !still_open && app.dialog == Some(d) {

View File

@ -18,6 +18,7 @@ pub mod misc;
pub mod mixer; pub mod mixer;
pub mod monitor; pub mod monitor;
pub mod presets; pub mod presets;
pub mod multicam;
pub mod project; pub mod project;
pub mod shortcuts_dialog; pub mod shortcuts_dialog;
pub mod text; pub mod text;

View File

@ -66,6 +66,17 @@ pub fn show(app: &mut FilmcraftApp, ui: &mut egui::Ui, rect: Rect, which: Which)
} }
}; };
let prefix = if which == Which::Program { "program" } else { "source" }; let prefix = if which == Which::Program { "program" } else { "source" };
// Multi-Camera view: the angle grid on the left, the program on the right
let multicam = which == Which::Program && app.ui.program.multicam;
let (grid_area, video_area) = if multicam {
let mid = video_area.center().x;
(Some(Rect::from_min_max(video_area.min, pos2(mid - 2.0, video_area.max.y))), Rect::from_min_max(pos2(mid + 2.0, video_area.min.y), video_area.max))
} else {
(None, video_area)
};
if let Some(g) = grid_area {
crate::panels::multicam::grid(app, ui, g);
}
ui.painter().rect_filled(video_area, 0.0, t.panel_bg); ui.painter().rect_filled(video_area, 0.0, t.panel_bg);
// ---- picture // ---- picture
let has_video = frame_size.0 > 0; let has_video = frame_size.0 > 0;
@ -203,7 +214,14 @@ pub fn show(app: &mut FilmcraftApp, ui: &mut egui::Ui, rect: Rect, which: Which)
if which == Which::Program { if which == Which::Program {
ui.separator(); ui.separator();
ui.checkbox(&mut app.ui.show_scopes, "Lumetri Scopes"); ui.checkbox(&mut app.ui.show_scopes, "Lumetri Scopes");
ui.checkbox(&mut app.ui.program.multicam, "Multi-Camera");
ui.checkbox(&mut app.playback.looping, "Loop"); ui.checkbox(&mut app.playback.looping, "Loop");
ui.separator();
let mut follows = app.session.state.multicam_audio_follows_video;
if ui.checkbox(&mut follows, "Multi-Camera Audio Follows Video").changed() {
let _ = app.session.execute("multicam.audioFollowsVideo", json!({"enabled": follows}));
}
ui.checkbox(&mut app.ui.multicam_record, "Multi-Camera Record");
} }
}); });
@ -216,7 +234,7 @@ pub fn show(app: &mut FilmcraftApp, ui: &mut egui::Ui, rect: Rect, which: Which)
transport(app, ui, row3, which); transport(app, ui, row3, which);
} }
fn quantize_scale(s: f32) -> f32 { pub fn quantize_scale(s: f32) -> f32 {
// Buckets keep the cache effective while resizing the panel. // Buckets keep the cache effective while resizing the panel.
let buckets = [1.0 / 32.0, 1.0 / 16.0, 1.0 / 8.0, 0.1875, 0.25, 0.375, 0.5, 0.75, 1.0]; let buckets = [1.0 / 32.0, 1.0 / 16.0, 1.0 / 8.0, 0.1875, 0.25, 0.375, 0.5, 0.75, 1.0];
*buckets.iter().find(|b| **b >= s - 1e-4).unwrap_or(&1.0) *buckets.iter().find(|b| **b >= s - 1e-4).unwrap_or(&1.0)

View File

@ -0,0 +1,381 @@
//! Multi-camera UI: the Program monitor's Multi-Camera view (angle grid next to the program) and
//! the Synchronize, Merge Clips and Create Multi-Camera Source Sequence dialogs.
//!
//! The grid is one frame job ([`Target::MulticamGrid`]): every shown angle rendered at the grid
//! cell's size (reduced resolution) and tiled, prefetched like the program while playing.
//! Clicking an angle (or keys 1–9) cuts to it while recording (playing with Multi-Camera Record
//! on) and switches the clip at the playhead when stopped; Ctrl/⌘-click switches the video only.
//!
//! Automation ids:
//! - Multi-Camera view: `program.multicam.grid`, `program.multicam.angle.<n>` (n = 1-based camera
//! in grid order), `program.multicam.record`.
//! - Synchronize: `sync.method.<in|out|timecode|marker|audio>`, `sync.ignoreHours`, `sync.marker`,
//! `sync.offset`, `sync.track`, `sync.ok`, `sync.cancel`.
//! - Merge Clips: `merge.name`, `merge.method.<…>`, `merge.ignoreHours`, `merge.marker`,
//! `merge.offset`, `merge.removeVideoAudio`, `merge.ok`, `merge.cancel`.
//! - Create Multi-Camera Source Sequence: `mcam.name`, `mcam.method.<…>`, `mcam.ignoreHours`,
//! `mcam.marker`, `mcam.offset`, `mcam.audio.<camera1|all|switch>`,
//! `mcam.cameraNames.<clip|track|metadata>`, `mcam.processedBin`, `mcam.ok`, `mcam.cancel`.
use egui::{Align2, Color32, Rect, RichText, Sense, Stroke, StrokeKind, pos2, vec2};
use filmcraft_project::ItemId;
use filmcraft_time::Tick;
use serde_json::{Value, json};
use crate::FilmcraftApp;
use crate::frames::{FrameKey, Target};
use crate::state::SyncDraft;
use crate::theme::Tokens;
type Elems = Vec<(String, Rect, String)>;
/// Menu / shortcut entry points: the three dialogs (without params) and the view toggles.
pub fn route(app: &mut FilmcraftApp, id: &str, params: &Value) -> Option<Result<Value, String>> {
match id {
"multicam.toggleView" => {
app.ui.program.multicam = params.get("enabled").and_then(Value::as_bool).unwrap_or(!app.ui.program.multicam);
return Some(Ok(json!({"multicam": app.ui.program.multicam})));
}
"multicam.recordToggle" => {
app.ui.multicam_record = params.get("enabled").and_then(Value::as_bool).unwrap_or(!app.ui.multicam_record);
app.ui.status = format!("Multi-Camera Record {}", if app.ui.multicam_record { "on" } else { "off" });
return Some(Ok(json!({"record": app.ui.multicam_record})));
}
_ => {}
}
if !params.as_object().is_none_or(|m| m.is_empty()) {
return None;
}
let kind = match id {
"clip.synchronize" => "synchronize",
"clip.mergeClips" => "merge",
"clip.createMulticam" => "multicam",
_ => return None,
};
if let Err(e) = filmcraft_engine::find_command(id).map_or(Ok(()), |c| (c.enabled)(&app.session)) {
return Some(Err(e));
}
let items: Vec<u64> = app.session.state.project_selection.iter().map(|i| i.0).collect();
let first = app.session.state.project_selection.first().and_then(|i| app.session.project.item(*i)).map(|i| i.name.clone()).unwrap_or_default();
let name = match kind {
"merge" => format!("{first} - Merged"),
"multicam" => format!("{first} Multicam"),
_ => String::new(),
};
app.ui.sync_dialog = Some(SyncDraft { kind: kind.into(), items, name, ..Default::default() });
Some(Ok(json!({"dialog": kind})))
}
fn push(elems: &mut Elems, id: impl Into<String>, r: &egui::Response, label: impl Into<String>) {
elems.push((id.into(), r.rect, label.into()));
}
const METHODS: [(&str, &str); 5] = [("in", "In Points"), ("out", "Out Points"), ("timecode", "Timecode"), ("marker", "Clip Marker"), ("audio", "Audio")];
/// Draw the open Synchronize / Merge Clips / Create Multi-Camera dialog.
pub fn show_dialog(app: &mut FilmcraftApp, ctx: &egui::Context) {
let Some(mut d) = app.ui.sync_dialog.clone() else { return };
let (title, pre) = match d.kind.as_str() {
"merge" => ("Merge Clips", "merge"),
"multicam" => ("Create Multi-Camera Source Sequence", "mcam"),
_ => ("Synchronize Clips", "sync"),
};
let mut elems: Elems = Vec::new();
let mut action: Option<&str> = None;
let accent = app.tokens.accent;
let names: Vec<String> = d.items.iter().filter_map(|i| app.session.project.item(ItemId(*i)).map(|it| it.name.clone())).collect();
egui::Window::new(title).collapsible(false).resizable(false).default_width(440.0).anchor(Align2::CENTER_CENTER, [0.0, 0.0]).show(ctx, |ui| {
if pre != "sync" {
ui.horizontal(|ui| {
ui.label(if pre == "merge" { "Clip Name:" } else { "Sequence Name:" });
let r = ui.add(egui::TextEdit::singleline(&mut d.name).desired_width(300.0));
push(&mut elems, format!("{pre}.name"), &r, "name");
});
ui.label(RichText::new(format!("{} clip(s): {}", names.len(), names.join(", "))).weak().small());
ui.add_space(4.0);
} else {
let n = app.session.state.selection.len();
ui.label(RichText::new(format!("{n} clip(s) selected; the clip on the reference track stays put.")).weak().small());
}
ui.group(|ui| {
ui.set_min_width(400.0);
ui.label(RichText::new("Synchronize Point").strong());
for (m, label) in METHODS {
ui.horizontal(|ui| {
let r = ui.radio(d.method == m, label);
push(&mut elems, format!("{pre}.method.{m}"), &r, label);
if r.clicked() {
d.method = m.into();
}
match m {
"timecode" => {
let r = ui.add_enabled(d.method == "timecode", egui::Checkbox::new(&mut d.ignore_hours, "Ignore Hours"));
push(&mut elems, format!("{pre}.ignoreHours"), &r, "Ignore Hours");
}
"marker" => {
let r =
ui.add_enabled(d.method == "marker", egui::TextEdit::singleline(&mut d.marker).hint_text("first marker").desired_width(140.0));
push(&mut elems, format!("{pre}.marker"), &r, "marker name");
}
_ => {}
}
});
}
ui.horizontal(|ui| {
ui.label(if pre == "merge" { "Offset Audio by:" } else { "Offset:" });
let r = ui.add(egui::DragValue::new(&mut d.offset).range(-100_000..=100_000).suffix(" frames"));
push(&mut elems, format!("{pre}.offset"), &r, "offset");
});
});
match pre {
"sync" => {
ui.horizontal(|ui| {
ui.label("Reference track:");
let r = ui.add(egui::TextEdit::singleline(&mut d.track).hint_text("lowest (V1…)").desired_width(90.0));
push(&mut elems, "sync.track", &r, "reference track");
});
}
"merge" => {
let r = ui.checkbox(&mut d.remove_video_audio, "Remove Audio from AV Clip");
push(&mut elems, "merge.removeVideoAudio", &r, "Remove Audio from AV Clip");
}
_ => {
ui.group(|ui| {
ui.set_min_width(400.0);
ui.label(RichText::new("Audio").strong());
ui.horizontal(|ui| {
for (a, label) in [("camera1", "Camera 1"), ("all", "All Cameras"), ("switch", "Switch Audio")] {
let r = ui.radio(d.audio == a, label);
push(&mut elems, format!("mcam.audio.{a}"), &r, label);
if r.clicked() {
d.audio = a.into();
}
}
});
ui.label(RichText::new("Camera Names").strong());
ui.horizontal(|ui| {
for (a, label) in [("clip", "Clip Names"), ("track", "Enumerate Cameras"), ("metadata", "Camera Angle Metadata")] {
let r = ui.radio(d.camera_names == a, label);
push(&mut elems, format!("mcam.cameraNames.{a}"), &r, label);
if r.clicked() {
d.camera_names = a.into();
}
}
});
});
let r = ui.checkbox(&mut d.processed_bin, "Move source clips to Processed Clips bin");
push(&mut elems, "mcam.processedBin", &r, "Move source clips to Processed Clips bin");
}
}
if !d.message.is_empty() {
ui.colored_label(Color32::from_rgb(0xe0, 0x8a, 0x6a), &d.message);
}
ui.add_space(6.0);
ui.horizontal(|ui| {
let r = ui.button("Cancel");
push(&mut elems, format!("{pre}.cancel"), &r, "Cancel");
if r.clicked() {
action = Some("cancel");
}
let r = ui.add(egui::Button::new(RichText::new("OK").color(Color32::WHITE)).fill(accent));
push(&mut elems, format!("{pre}.ok"), &r, "OK");
if r.clicked() {
action = Some("ok");
}
});
});
for (id, r, l) in elems {
app.auto.add(&id, r, &l);
}
if ctx.input(|i| i.key_pressed(egui::Key::Escape)) {
action = Some("cancel");
}
match action {
Some("cancel") => {
app.ui.sync_dialog = None;
return;
}
Some("ok") => {
let mut p = json!({"method": d.method, "offset": d.offset, "ignoreHours": d.ignore_hours});
if !d.marker.is_empty() {
p["marker"] = json!(d.marker);
}
let cmd = match pre {
"merge" => {
p["items"] = json!(d.items);
p["removeVideoAudio"] = json!(d.remove_video_audio);
if !d.name.is_empty() {
p["name"] = json!(d.name);
}
"clip.mergeClips"
}
"mcam" => {
p["items"] = json!(d.items);
p["audio"] = json!(d.audio);
p["cameraNames"] = json!(d.camera_names);
p["processedBin"] = json!(d.processed_bin);
if !d.name.is_empty() {
p["name"] = json!(d.name);
}
"clip.createMulticam"
}
_ => {
if !d.track.is_empty() {
p["track"] = json!(d.track);
}
"clip.synchronize"
}
};
match app.session.execute(cmd, p) {
Ok(v) => {
app.ui.status = match pre {
"sync" => format!("Synchronized {} clip group(s)", v["moved"]),
"merge" => "Merged clip created".into(),
_ => format!("Multi-camera source sequence with {} camera(s)", v["cameras"].as_array().map_or(0, Vec::len)),
};
app.ui.sync_dialog = None;
return;
}
Err(e) => d.message = e.to_string(),
}
}
_ => {}
}
app.ui.sync_dialog = Some(d);
}
/// A revision for grid frames: changes when the multi-camera source (or its cameras' media)
/// changes, not with every edit of the cut (cuts recorded live must not throw away the grid).
fn grid_revision(app: &FilmcraftApp, src: ItemId) -> u64 {
use std::hash::{Hash, Hasher};
let mut h = std::collections::hash_map::DefaultHasher::new();
if let Some(q) = app.session.project.sequence(src) {
serde_json::to_vec(q).unwrap_or_default().hash(&mut h);
for t in &q.video_tracks {
for it in &t.items {
app.item_revision(it.item).hash(&mut h);
}
}
}
h.finish()
}
/// The angle grid of the Multi-Camera view in `area`.
pub fn grid(app: &mut FilmcraftApp, ui: &mut egui::Ui, area: Rect) {
let t = app.tokens;
let ctx = ui.ctx().clone();
ui.painter().rect_filled(area, 0.0, t.panel_bg);
app.auto.add("program.multicam.grid", area, "Multi-Camera");
let playhead = app.session.playhead();
let info = filmcraft_engine::multicam::inspect_at(&app.session, playhead);
let Some(src) = info["source"].as_u64().map(ItemId) else {
ui.painter().text(area.center(), Align2::CENTER_CENTER, "No multi-camera clip at the playhead", Tokens::ui(12.0), t.text_dim);
return;
};
let Some(q) = app.session.project.sequence(src) else { return };
let cams = q.cameras();
let shown = cams.shown_angles();
let (cols, rows) = filmcraft_render::multicam::grid_dims(shown.len().max(1));
let (sw, sh) = (q.settings.width as f32, q.settings.height as f32);
let rate = q.settings.frame_rate;
let pic = crate::panels::monitor::fit(area.shrink(2.0), sw * cols as f32, sh * rows as f32);
let (cw, ch) = (pic.width() / cols as f32, pic.height() / rows as f32);
// decode and composite at the cell's size (never more than the playback resolution)
let cell_px = cw * ctx.pixels_per_point();
let scale = crate::panels::monitor::quantize_scale(app.ui.program.res.scale().min((cell_px / sw).max(1.0 / 32.0)));
let mt = Tick(info["sourceTime"].as_i64().unwrap_or(0));
let frame = rate.frame_at(mt);
let key = FrameKey { target: Target::MulticamGrid(src), frame, size: (scale * 1000.0) as u32, revision: grid_revision(app, src) };
let project = app.session.project.clone();
if app.playback.playing {
app.frames.schedule_playback(key, rate, scale, &project, app.playback.speed, app.playback.preroll.is_some());
} else {
app.frames.request(key, rate.tick_of(frame), scale, &project, 0);
}
let tex = match app.frames.get(&key) {
Some(img) => Some(app.texture_for(&ctx, "monitor-multicam", key, &img)),
None => match app.frames.nearest(key.target, frame, key.size, key.revision, 6) {
Some(img) => Some(app.texture_for(&ctx, "monitor-multicam", FrameKey { frame: frame - 1, ..key }, &img)),
None => app.texture_existing("monitor-multicam").map(|(id, _)| id),
},
};
ui.painter().rect_filled(pic, 0.0, t.monitor_bg);
if let Some(tex) = tex {
ui.painter().image(tex, pic, Rect::from_min_max(pos2(0.0, 0.0), pos2(1.0, 1.0)), Color32::WHITE);
}
let current = info["angle"].as_u64().map(|a| a as usize);
let recording = app.session.mcrec.active();
let mut click: Option<(usize, bool)> = None;
for (k, &angle) in shown.iter().enumerate() {
let cell = Rect::from_min_size(pic.min + vec2((k % cols) as f32 * cw, (k / cols) as f32 * ch), vec2(cw, ch));
let name = cams.cameras.get(angle).map(|c| c.name.clone()).unwrap_or_default();
let resp = ui.interact(cell, egui::Id::new(("multicam-angle", k)), Sense::click()).on_hover_text(format!("Camera {} — {name}", k + 1));
app.auto.add(&format!("program.multicam.angle.{}", k + 1), cell, &name);
if resp.clicked() {
// modifiers held now, or carried by the click event (synthetic input)
let video_only = ui.input(|i| {
let ev = i.events.iter().any(|e| matches!(e, egui::Event::PointerButton { modifiers: m, .. } if m.ctrl || m.command));
i.modifiers.ctrl || i.modifiers.command || ev
});
click = Some((k + 1, video_only));
}
if resp.hovered() {
ui.painter().rect_stroke(cell.shrink(1.0), 0.0, Stroke::new(1.0, Color32::from_white_alpha(90)), StrokeKind::Inside);
}
if current == Some(angle) {
// the active angle: yellow, red while recording
let col = if recording { Color32::from_rgb(0xe0, 0x3a, 0x3a) } else { Color32::from_rgb(0xf2, 0xc9, 0x4c) };
ui.painter().rect_stroke(cell.shrink(1.5), 0.0, Stroke::new(3.0, col), StrokeKind::Inside);
}
let label = format!("{} {name}", k + 1);
let galley = ui.painter().layout_no_wrap(label, Tokens::ui(11.0), Color32::WHITE);
let lr = Rect::from_min_size(cell.left_bottom() + vec2(6.0, -galley.size().y - 8.0), galley.size() + vec2(8.0, 4.0));
ui.painter().rect_filled(lr, 3.0, Color32::from_black_alpha(160));
ui.painter().galley(lr.min + vec2(4.0, 2.0), galley, Color32::WHITE);
}
// record indicator
let rr = Rect::from_min_size(area.min + vec2(8.0, 8.0), vec2(64.0, 18.0));
let rec_on = app.ui.multicam_record;
let resp = ui.interact(rr, egui::Id::new("multicam-record"), Sense::click()).on_hover_text("Multi-Camera Record On/Off (0)");
app.auto.add("program.multicam.record", rr, "Multi-Camera Record");
let dot = if recording {
Color32::from_rgb(0xe0, 0x3a, 0x3a)
} else if rec_on {
Color32::from_rgb(0x9a, 0x3a, 0x3a)
} else {
t.text_faint
};
ui.painter().rect_filled(rr, 3.0, Color32::from_black_alpha(140));
ui.painter().circle_filled(rr.left_center() + vec2(9.0, 0.0), 4.5, dot);
ui.painter().text(rr.left_center() + vec2(18.0, 0.0), Align2::LEFT_CENTER, if recording { "REC" } else { "Record" }, Tokens::ui(10.5), Color32::WHITE);
if resp.clicked() {
app.ui.multicam_record = !app.ui.multicam_record;
}
if let Some((n, video_only)) = click
&& let Err(e) = app.session.execute("multicam.cut", json!({"camera": n, "videoOnly": video_only}))
{
app.ui.status = e.to_string();
}
}
/// Start a recording pass when playback starts in the Multi-Camera view.
pub fn on_play(app: &mut FilmcraftApp) {
if app.ui.program.multicam && app.ui.multicam_record && !app.session.mcrec.active() && (app.playback.speed - 1.0).abs() < 1e-9 {
let t = app.session.playhead();
if filmcraft_engine::multicam::multicam_clip_at(&app.session, t).is_some() {
let _ = app.session.execute("multicam.recordStart", json!({"time": t.0}));
}
}
}
/// End the recording pass (one undo step) when playback stops.
pub fn on_stop(app: &mut FilmcraftApp) {
if app.session.mcrec.active() {
let t = app.session.playhead();
match app.session.execute("multicam.recordStop", json!({"time": t.0})) {
Ok(v) if v["cuts"].as_u64().unwrap_or(0) > 0 => app.ui.status = format!("Recorded {} multi-camera cut(s)", v["cuts"]),
Ok(_) => {}
Err(e) => app.ui.status = e.to_string(),
}
}
}

View File

@ -428,7 +428,18 @@ fn draw_clip(
if let Some(aspect) = aspect { if let Some(aspect) = aspect {
let tw = (th.height() * aspect).min(th.width()); let tw = (th.height() * aspect).min(th.width());
let mt = rate.snap(it.source_in); let mt = rate.snap(it.source_in);
if let Some((tex, _)) = app.thumbnail(ctx, it.item, mt, 160) { // a multi-camera clip shows its angle's clip
let (thumb_item, mt) = app
.session
.project
.sequence(it.item)
.and_then(|q| {
let ti = q.angle_video_track_index(it.multicam_angle(q)?)?;
let inner = q.video_tracks[ti].item_at(mt)?;
Some((inner.item, inner.source_time_at(mt)))
})
.unwrap_or((it.item, mt));
if let Some((tex, _)) = app.thumbnail(ctx, thumb_item, mt, 160) {
let r = Rect::from_min_size(th.min, vec2(tw, th.height())); let r = Rect::from_min_size(th.min, vec2(tw, th.height()));
let cp = p.with_clip_rect(th.intersect(p.clip_rect())); let cp = p.with_clip_rect(th.intersect(p.clip_rect()));
cp.image(tex, r, Rect::from_min_max(pos2(0.0, 0.0), pos2((tw / (th.height() * aspect)).min(1.0), 1.0)), Color32::WHITE); cp.image(tex, r, Rect::from_min_max(pos2(0.0, 0.0), pos2((tw / (th.height() * aspect)).min(1.0), 1.0)), Color32::WHITE);
@ -443,6 +454,10 @@ fn draw_clip(
// name band // name band
if w > 24.0 { if w > 24.0 {
let mut label = it.name.clone(); let mut label = it.name.clone();
if let Some(m) = it.multicam.filter(|m| m.enabled) {
let cam = app.session.project.sequence(it.item).and_then(|q| q.cameras().cameras.get(m.angle as usize).map(|c| c.name.clone()));
label = format!("[MC{}] {}", m.angle + 1, cam.unwrap_or(label));
}
if (it.speed - 1.0).abs() > 1e-6 || it.reverse { if (it.speed - 1.0).abs() > 1e-6 || it.reverse {
label = format!("{label} [{}%]", ((if it.reverse { -1.0 } else { 1.0 }) * it.speed * 100.0).round()); label = format!("{label} [{}%]", ((if it.reverse { -1.0 } else { 1.0 }) * it.speed * 100.0).round());
} }

View File

@ -220,11 +220,14 @@ pub struct MonitorView {
pub zoom: Option<f32>, pub zoom: Option<f32>,
pub safe_margins: bool, pub safe_margins: bool,
pub show_transport: bool, pub show_transport: bool,
/// Program monitor display mode Multi-Camera (angle grid + program).
#[serde(default)]
pub multicam: bool,
} }
impl Default for MonitorView { impl Default for MonitorView {
fn default() -> Self { fn default() -> Self {
Self { res: PlaybackRes::Half, zoom: None, safe_margins: false, show_transport: true } Self { res: PlaybackRes::Half, zoom: None, safe_margins: false, show_transport: true, multicam: false }
} }
} }
@ -316,6 +319,12 @@ pub struct UiState {
/// Effect presets: the preset being renamed / the Save Preset dialog (open when Some). /// Effect presets: the preset being renamed / the Save Preset dialog (open when Some).
#[serde(default)] #[serde(default)]
pub save_preset: Option<SavePresetDraft>, pub save_preset: Option<SavePresetDraft>,
/// Synchronize / Merge Clips / Create Multi-Camera Source Sequence dialog (open when Some).
#[serde(default)]
pub sync_dialog: Option<SyncDraft>,
/// Multi-Camera Record On/Off (key 0): playing in the Multi-Camera view records cuts.
#[serde(default = "yes")]
pub multicam_record: bool,
} }
/// A pen mask in progress: vertices placed so far, in clip pixels (`[x, y, tangent x, tangent y]`). /// A pen mask in progress: vertices placed so far, in clip pixels (`[x, y, tangent x, tangent y]`).
@ -337,6 +346,54 @@ pub struct SavePresetDraft {
pub keyframes: String, pub keyframes: String,
} }
/// Clip ▸ Synchronize…, Merge Clips… and Create Multi-Camera Source Sequence… share one draft;
/// `kind` says which dialog it is (`synchronize`, `merge`, `multicam`).
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
#[serde(default)]
pub struct SyncDraft {
pub kind: String,
/// Project items (Merge Clips, Create Multi-Camera).
pub items: Vec<u64>,
pub name: String,
/// `in` | `out` | `timecode` | `marker` | `audio`
pub method: String,
pub ignore_hours: bool,
/// Clip marker name to sync on (empty = the first marker).
pub marker: String,
/// Frames added to every clip but the reference.
pub offset: i64,
/// Synchronize: reference track (`V1`, `A2`…; empty = the lowest track).
pub track: String,
/// Create Multi-Camera: `camera1` | `all` | `switch`.
pub audio: String,
/// Create Multi-Camera: `clip` | `track` | `metadata`.
pub camera_names: String,
pub processed_bin: bool,
/// Merge Clips: drop the video clip's own audio.
pub remove_video_audio: bool,
pub message: String,
}
impl Default for SyncDraft {
fn default() -> Self {
Self {
kind: "synchronize".into(),
items: vec![],
name: String::new(),
method: "audio".into(),
ignore_hours: false,
marker: String::new(),
offset: 0,
track: String::new(),
audio: "camera1".into(),
camera_names: "clip".into(),
processed_bin: true,
remove_video_audio: false,
message: String::new(),
}
}
}
/// File ▸ Link Media… (shown automatically when a project opens with missing media). /// File ▸ Link Media… (shown automatically when a project opens with missing media).
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] #[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
#[serde(default)] #[serde(default)]
@ -508,6 +565,8 @@ impl Default for UiState {
color_dialog: None, color_dialog: None,
mask_pen: None, mask_pen: None,
save_preset: None, save_preset: None,
sync_dialog: None,
multicam_record: true,
} }
} }
} }

View File

@ -0,0 +1,301 @@
//! Headless UI tests of multi-camera editing: the Create Multi-Camera Source Sequence, Synchronize
//! and Merge Clips dialogs, the Program monitor's Multi-Camera view (angle grid, clicking an
//! angle, Ctrl-click for video only) and live switching with the 1–9 keys while playing.
//!
//! Set `FILMCRAFT_UI_SNAPSHOT_DIR=<dir>` to also render the window offscreen with wgpu and write PNGs
//! there (`multicam-*.png`).
use std::path::{Path, PathBuf};
use std::sync::Arc;
use std::sync::mpsc::{Sender, channel};
use egui_kittest::Harness;
use filmcraft_engine::Session;
use filmcraft_engine::project::{ItemKind, Label, MediaClip, MediaRef, Project, SequenceSettings, TrackKind};
use filmcraft_media::generators::GeneratorSource;
use filmcraft_media::{DemoScene, Generator, MediaSource};
use filmcraft_time::{FrameRate, Tick, TimeRange};
use filmcraft_ui_egui::FilmcraftApp;
use filmcraft_ui_egui::control::ControlRequest;
use serde_json::{Value, json};
const W: u32 = 320;
const H: u32 = 180;
fn tmp_dir(name: &str) -> PathBuf {
let nanos = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).unwrap().as_nanos();
let d = std::env::temp_dir().join(format!("filmcraft-ui-{name}-{}-{nanos}", std::process::id()));
std::fs::create_dir_all(&d).unwrap();
d
}
/// A 24 fps ProRes movie of a demo scene (our own encoder).
fn make_movie(path: &Path, scene: DemoScene, frames: i64) {
let rate = FrameRate::FPS_24;
let dur = rate.tick_of(frames);
let src = GeneratorSource::new(Generator::Demo(scene), W, H, rate, dur);
let clip = MediaClip {
media: MediaRef::Generator(Generator::Demo(scene)),
info: src.info().clone(),
interpret: Default::default(),
mark_in: None,
mark_out: None,
markers: vec![],
offline: false,
proxy: None,
identity: None,
};
let mut p = Project::new("fixture");
let item = p.add_item("scene", Label::Iris, ItemKind::Media(clip), None);
let seq = p.new_sequence("s", SequenceSettings { width: W, height: H, frame_rate: rate, ..Default::default() }, 1, 0, None);
let mut v = p.make_track_item(item, TrackKind::Video, Tick::ZERO, TimeRange::new(Tick::ZERO, dur), rate).unwrap();
for e in &mut v.effects {
filmcraft_engine::project::resolve_auto_points(e, (W, H), (W, H));
}
p.sequence_mut(seq).unwrap().video_tracks[0].items.push(v);
let settings = filmcraft_engine::export::ExportSettings {
format: filmcraft_engine::export::Format::ProRes,
path: path.to_string_lossy().into_owned(),
include_audio: false,
..Default::default()
};
let shared: filmcraft_media::SharedSource = Arc::new(src);
let provider = move |id| (id == item).then(|| shared.clone());
filmcraft_engine::export::export(&Arc::new(p), seq, &settings, &provider, &Default::default()).unwrap();
}
/// Four cameras imported into a session with an empty 320×180 sequence "Cut".
fn session() -> (Session, Vec<u64>) {
let dir = tmp_dir("multicam");
let scenes = [DemoScene::OceanSunset, DemoScene::CityNight, DemoScene::Aurora, DemoScene::Forest];
let files: Vec<PathBuf> = (0..4).map(|i| dir.join(format!("Cam {}.mov", (b'A' + i as u8) as char))).collect();
for (f, sc) in files.iter().zip(scenes) {
make_movie(f, sc, 96);
}
let mut s = Session::default();
let r = s.execute("file.import", json!({"paths": files.iter().map(|f| f.to_string_lossy()).collect::<Vec<_>>()})).unwrap();
let items: Vec<u64> = r["items"].as_array().unwrap().iter().map(|v| v.as_u64().unwrap()).collect();
let mut p = (*s.project).clone();
let seq = p.new_sequence("Cut", SequenceSettings { width: W, height: H, frame_rate: FrameRate::FPS_24, ..Default::default() }, 2, 2, None);
s.project = Arc::new(p);
s.state.active_sequence = Some(seq);
s.state.open_sequences = vec![seq];
(s, items)
}
struct Driver {
harness: Harness<'static, FilmcraftApp>,
tx: Sender<ControlRequest>,
snapshots: Option<PathBuf>,
}
impl Driver {
fn new(session: Session) -> Self {
let (tx, rx) = channel();
let app = FilmcraftApp::new(session).with_control(rx);
let snapshots = std::env::var_os("FILMCRAFT_UI_SNAPSHOT_DIR").map(PathBuf::from);
let mut b = Harness::builder().with_size(egui::vec2(1600.0, 980.0)).with_max_steps(10_000);
if snapshots.is_some() {
b = b.wgpu();
}
let harness = b.build_eframe(move |_cc| app);
let mut d = Driver { harness, tx, snapshots };
d.frames(4);
d
}
fn frames(&mut self, n: usize) {
for _ in 0..n {
let ctx = self.harness.ctx.clone();
let mut raw = std::mem::take(self.harness.input_mut());
eframe::App::raw_input_hook(self.harness.state_mut(), &ctx, &mut raw);
*self.harness.input_mut() = raw;
self.harness.step();
}
}
/// Run frames (with short sleeps so frame workers finish) until `done`.
fn until(&mut self, what: &str, mut done: impl FnMut(&mut Self) -> bool) {
for _ in 0..600 {
if done(self) {
return;
}
self.frames(1);
std::thread::sleep(std::time::Duration::from_millis(10));
}
panic!("timed out waiting for {what}");
}
fn call(&mut self, method: &str, params: Value) -> Value {
let (req, reply) = ControlRequest::new(method, params.clone());
self.tx.send(req).unwrap();
for _ in 0..600 {
self.frames(1);
if let Ok(v) = reply.try_recv() {
return v;
}
}
panic!("no reply to {method} {params}");
}
fn ok(&mut self, method: &str, params: Value) -> Value {
let v = self.call(method, params.clone());
assert_eq!(v["ok"], json!(true), "{method} {params} failed: {v}");
v["result"].clone()
}
fn exec(&mut self, command: &str, params: Value) -> Value {
self.ok("engine.execute", json!({"command": command, "params": params}))
}
fn click(&mut self, id: &str) {
self.ok("ui.click", json!({"id": id}));
self.frames(3);
}
fn ids(&mut self, prefix: &str) -> Vec<String> {
let v = self.ok("ui.elements", json!({"prefix": prefix}));
v.as_array().unwrap().iter().filter_map(|e| e["id"].as_str().map(str::to_string)).collect()
}
fn app(&mut self) -> &mut FilmcraftApp {
self.harness.state_mut()
}
/// Angles of the clips on V1 / A1 of the active sequence.
fn angles(&mut self, audio: bool) -> Vec<u32> {
let q = self.app().session.active_sequence().unwrap().clone();
let t = if audio { &q.audio_tracks[0] } else { &q.video_tracks[0] };
t.items.iter().map(|i| i.multicam.map_or(99, |m| m.angle)).collect()
}
fn snapshot(&mut self, name: &str) {
let Some(dir) = self.snapshots.clone() else { return };
self.frames(3);
let img = match self.harness.render() {
Ok(i) => i,
Err(e) => {
eprintln!("snapshot {name} skipped: {e}");
return;
}
};
std::fs::create_dir_all(&dir).unwrap();
let path = dir.join(format!("{name}.png"));
img.save(&path).unwrap();
eprintln!("snapshot: {}", path.display());
}
}
#[test]
fn create_multicam_dialog_view_and_live_switching() {
let (s, items) = session();
let mut d = Driver::new(s);
// Create Multi-Camera Source Sequence from the Project panel selection
d.exec("project.select", json!({"items": items}));
d.ok("ui.menu.invoke", json!({"id": "clip.createMulticam"}));
d.frames(3);
assert_eq!(d.app().ui.sync_dialog.as_ref().map(|x| x.kind.clone()).as_deref(), Some("multicam"));
for id in ["mcam.name", "mcam.method.audio", "mcam.method.timecode", "mcam.audio.switch", "mcam.cameraNames.track", "mcam.processedBin", "mcam.ok"] {
assert!(!d.ids(id).is_empty(), "{id}");
}
d.click("mcam.method.in");
d.click("mcam.audio.switch");
d.snapshot("multicam-create-dialog");
d.click("mcam.ok");
assert!(d.app().ui.sync_dialog.is_none(), "dialog closed: {}", d.app().ui.status);
let src = d.app().session.state.project_selection[0];
let mc = d.app().session.project.sequence(src).unwrap().multicam.clone().unwrap();
assert_eq!(mc.cameras.len(), 4);
assert_eq!(mc.cameras[0].name, "Cam A.mov");
// edit it into the sequence, open the Multi-Camera view
d.exec("source.open", json!({"item": src.0}));
d.exec("playhead.set", json!({"frame": 0}));
d.exec("source.overwrite", json!({}));
d.exec("playhead.set", json!({"frame": 12}));
d.ok("ui.menu.invoke", json!({"id": "multicam.toggleView"}));
d.frames(2);
assert!(d.app().ui.program.multicam);
let angles = d.ids("program.multicam.angle.");
assert_eq!(angles.len(), 4, "{angles:?}");
d.until("grid frame", |d| {
let k = d.app().frames.queue_len();
k == 0
});
d.frames(6);
d.snapshot("multicam-view");
// clicking an angle switches the clip at the playhead; audio follows video when asked
d.exec("multicam.audioFollowsVideo", json!({"enabled": true}));
d.click("program.multicam.angle.2");
assert_eq!(d.angles(false), [1]);
assert_eq!(d.angles(true), [1], "audio followed");
// Ctrl-click: video only
d.ok("ui.click", json!({"id": "program.multicam.angle.3", "modifiers": {"ctrl": true}}));
d.frames(3);
assert_eq!((d.angles(false), d.angles(true)), (vec![2], vec![1]));
d.until("program frame", |d| d.app().frames.queue_len() == 0);
d.frames(6);
d.snapshot("multicam-view-camera3");
d.exec("playhead.set", json!({"frame": 0}));
d.click("program.multicam.angle.1");
// live switching: play, press 2 then 4, stop → one undo step with the recorded cuts
let undo_before = d.app().session.history.undo.len();
d.ok("ui.playback", json!({"action": "play"}));
d.until("recording", |d| d.app().session.mcrec.active());
d.until("playhead past 0.3 s", |d| d.app().session.playhead() > FrameRate::FPS_24.tick_of(7));
d.ok("ui.key", json!({"key": "2"}));
d.frames(2);
d.until("playhead past 0.8 s", |d| d.app().session.playhead() > FrameRate::FPS_24.tick_of(19));
d.ok("ui.key", json!({"key": "4"}));
d.frames(4);
d.snapshot("multicam-recording");
d.until("playhead past 1.2 s", |d| d.app().session.playhead() > FrameRate::FPS_24.tick_of(29));
d.ok("ui.playback", json!({"action": "stop"}));
d.frames(2);
assert!(!d.app().session.mcrec.active());
let v = d.angles(false);
assert_eq!(&v[..3], [0, 1, 3], "{v:?}");
assert_eq!(d.angles(true)[..3], [0, 1, 3], "audio followed the recording");
assert_eq!(d.app().session.history.undo.len(), undo_before + 1, "one undo step per pass");
d.frames(4);
d.snapshot("multicam-after-recording");
d.exec("edit.undo", json!({}));
assert_eq!(d.angles(false), [0]);
}
#[test]
fn synchronize_and_merge_dialogs() {
let (s, items) = session();
let mut d = Driver::new(s);
// two cameras on V1 and V2 at different times
d.exec("source.open", json!({"item": items[0]}));
d.exec("playhead.set", json!({"frame": 0}));
d.exec("source.overwrite", json!({}));
d.exec("timeline.setTargeting", json!({"track": "V2", "sourcePatch": true}));
d.exec("timeline.setTargeting", json!({"track": "A2", "sourcePatch": true}));
d.exec("source.open", json!({"item": items[1]}));
d.exec("playhead.set", json!({"frame": 30}));
d.exec("source.overwrite", json!({}));
assert_eq!(d.app().session.active_sequence().unwrap().video_tracks[1].items.len(), 1);
let all: Vec<u64> = d.app().session.active_sequence().unwrap().all_tracks().flat_map(|t| t.items.iter().map(|i| i.id.0)).collect();
d.exec("timeline.select", json!({"clips": all}));
d.ok("ui.menu.invoke", json!({"id": "clip.synchronize"}));
d.frames(3);
assert_eq!(d.app().ui.sync_dialog.as_ref().map(|x| x.kind.clone()).as_deref(), Some("synchronize"));
d.click("sync.method.in");
d.snapshot("multicam-sync-dialog");
d.click("sync.ok");
assert!(d.app().ui.sync_dialog.is_none(), "{}", d.app().ui.status);
let q = d.app().session.active_sequence().unwrap().clone();
assert_eq!(q.video_tracks[0].items[0].start, q.video_tracks[1].items[0].start, "In points aligned");
// Merge Clips needs audio clips: the dialog explains what is wrong
d.exec("project.select", json!({"items": [items[0], items[1]]}));
d.ok("ui.menu.invoke", json!({"id": "clip.mergeClips"}));
d.frames(3);
assert!(!d.ids("merge.removeVideoAudio").is_empty());
d.snapshot("multicam-merge-dialog");
d.click("merge.ok");
let msg = d.app().ui.sync_dialog.as_ref().map(|x| x.message.clone()).unwrap_or_default();
assert!(msg.contains("one video clip"), "{msg}");
d.click("merge.cancel");
assert!(d.app().ui.sync_dialog.is_none());
}

View File

@ -384,6 +384,47 @@ frame (Y'CbCr/RGB + metadata) ─► source colour space: Interpret Footage over
conversions and looks). `filmcraft-gpu::GpuLut` is the WGSL tetrahedral counterpart, tested for conversions and looks). `filmcraft-gpu::GpuLut` is the WGSL tetrahedral counterpart, tested for
parity. parity.
### 5.4 Multi-camera and synchronisation
```text
clips ──sync (in | out | timecode[±hours] | marker | audio)──► anchors (media time ↔ common instant)
├─ clip.synchronize moves selected timeline clips (link groups together) onto the reference
├─ clip.mergeClips video + ≤16 audio clips → a merged-clip sequence
└─ clip.createMulticam cameras → a multi-camera source sequence (one video track per angle)
multi-camera clip = nested source + TrackItem::multicam {enabled, angle}
render: only the angle's video track · audio: camera 1 | all | the angle (Switch Audio)
```
- **Model** (`project::multicam`). `Sequence::multicam` (`MulticamSource`: cameras with their video
track, audio tracks, name, shown flag and source item; audio mode) marks a multi-camera source;
`Sequence::merged` a merged clip. `TrackItem::multicam` (`MulticamSel`) makes a nested clip a
multi-camera clip; any nest can be one (its video tracks are then the angles,
`Sequence::cameras()`). Editing a multi-camera source into a sequence gives an enabled clip on the
first angle (`Project::make_track_item`). Project schema v7.
- **Sync** (`engine::sync`). Each method reduces a clip to an *anchor* (the media time that lines up
with the common instant). Audio uses `audio_dsp::sync::find_offset`: DC removal, windowed-sinc
decimation to ≤ 8 kHz, GCC-PHAT-β (β = 0.75) via one packed complex FFT for the coarse lag, then
the same at the full rate on the loudest common window (≤ 2.7 s) and parabolic interpolation.
Recordings with different gains, microphones (filtered), 0 dB SNR noise or a strong echo are
aligned to the sample; two 10-minute recordings take ~2.4 s (release). Clips are placed on frame
boundaries with the sub-frame remainder taken from their source In, so video stays on frames
while audio keeps sample accuracy.
- **Render.** `render::item_layer` renders a multi-camera clip's angle track only
(`render_seq_tracks`); `render::audio` mixes the nested source with only the audible tracks
(`Sequence::with_angle_audio`). Nested audio now plays at all (it was skipped when the nest had
no media source) and is limited to the clip's range. `render::multicam::render_grid` renders the
shown angles at the cell scale in parallel (rayon) and tiles them: the Multi-Camera view is one
frame job (`frames::Target::MulticamGrid`), prefetched while playing like the program.
- **Editing** (`edit::multicam`, `engine::multicam`). `multicam.switchAngle` (click an angle,
Ctrl/⌘-click for video only), `multicam.selectCamera1…9` (keys 1–9) and `cutToCamera1…9`
(Ctrl+1–9), Enable/Flatten, Edit Cameras, Audio Follows Video (`EditorState`). Live switching:
playback in the Multi-Camera view runs `multicam.recordStart`; each key/click is a
`multicam.cut` that is applied at once (the program shows it) by re-applying the whole pass to
the project from before the pass (`edit_merged`), so a pass is one undo step; Stop runs
`multicam.recordStop`, which ends the last angle at the stop point. Through edits inside the
recorded range are healed, so pressing the angle already showing adds no edit. Flatten replaces
a clip by the clip(s) its angle shows (outer effects carried over; linked pairs stay linked).
## 6. Export jobs (`filmcraft-export`) ## 6. Export jobs (`filmcraft-export`)
```text ```text

View File

@ -8,7 +8,7 @@ writing them is `crates/format` (`filmcraft-format`); the engine's `file.*` comm
```json ```json
{ {
"format": "filmcraft.project", "format": "filmcraft.project",
"schema_version": 7, "schema_version": 8,
"generator": "FilmCraft 0.1.0", "generator": "FilmCraft 0.1.0",
"project": { "name": "…", "settings": { … }, "root": { … }, "items": { … }, "next_id": 48 } "project": { "name": "…", "settings": { … }, "root": { … }, "items": { … }, "next_id": 48 }
} }
@ -31,6 +31,7 @@ writing them is `crates/format` (`filmcraft-format`); the engine's `file.*` comm
| 5 | M8.8 | colour management: working space, Interpret Footage colour space, LUT library; no-op step | | 5 | M8.8 | colour management: working space, Interpret Footage colour space, LUT library; no-op step |
| 6 | M11.7/M11.9 | media identity fingerprints, ingest settings; no-op step | | 6 | M11.7/M11.9 | media identity fingerprints, ingest settings; no-op step |
| 7 | M5.5 | effect masks: `masks` on effect instances with keyframable Bézier `Path` values; no-op step | | 7 | M5.5 | effect masks: `masks` on effect instances with keyframable Bézier `Path` values; no-op step |
| 8 | M12 | multi-camera source sequences and clips, merged clips; no-op step |
### Migrations ### Migrations
@ -58,8 +59,8 @@ on disk is unchanged until you save. The **first save over it** keeps the origin
A file with a `schema_version` newer than the build supports is **refused**, not half-read: A file with a `schema_version` newer than the build supports is **refused**, not half-read:
> this project was saved by a newer version of FilmCraft (project schema v7); this build reads up to > this project was saved by a newer version of FilmCraft (project schema v9); this build reads up to
> v6. Update FilmCraft to open it. > v8. Update FilmCraft to open it.
Reading it partially and saving it back would silently drop whatever the newer version added. Reading it partially and saving it back would silently drop whatever the newer version added.

View File

@ -20,6 +20,7 @@ FILMCRAFT_REQUIRE_ORACLES=1 cargo test --workspace # CI: missing ffmpeg fails
| Golden images | `crates/golden/tests/golden.rs` | CPU renders vs committed PNGs; GPU vs CPU on the same scenes (§3) | | Golden images | `crates/golden/tests/golden.rs` | CPU renders vs committed PNGs; GPU vs CPU on the same scenes (§3) |
| Engine / command | `crates/engine/src/tests.rs` | run commands on the demo project, assert the sequence, undo/redo, disabled cases | | Engine / command | `crates/engine/src/tests.rs` | run commands on the demo project, assert the sequence, undo/redo, disabled cases |
| Render | `crates/render/src/tests.rs` | compositing, opacity, Motion, cross dissolve midpoint, ½-res vs full, GPU plan vs reference, audio mix, audio-effect continuity | | Render | `crates/render/src/tests.rs` | compositing, opacity, Motion, cross dissolve midpoint, ½-res vs full, GPU plan vs reference, audio mix, audio-effect continuity |
| Multi-camera / sync | `crates/audio-dsp/src/sync.rs`, `crates/engine/src/multicam_tests.rs`, `crates/ui-egui/tests/multicam_ui.rs` | audio offsets within one sample: Rust-generated multi-mic speech (gains −20…+10 dB, SNR 40/25/10/3 dB, zero-phase colouring, fractional delays, a 0 dB-SNR room with a −4 dB echo) and ffmpeg-made camera/recorder files (pink-noise bursts, independent noise; exact lags recovered); timecode (± hours) and marker sync within a frame; Merge Clips; timeline Synchronize keeps frame-aligned starts; multicam structure, angle render (PSNR vs the camera), Switch Audio mix = the angle's track, live switching from a simulated playback clock (cuts, healing, one undo step), Ctrl+N cuts, flatten (frames identical), save/load; `perf_grid_of_four_1080p_angles` (ignored) prints the grid cost |
| Mixer | `crates/render/src/mixer_tests.rs`, `crates/engine/src/mixer_tests.rs` | sample-exact fader gain and pan laws, automation at block boundaries (bit-identical however requests are cut), solo/mute/solo-safe, sends and submix routing, latency-compensation alignment, render-vs-playback identity, Touch ramp-back, recorder modes (Latch/Touch/Write), thinning, Audio Gain, transition curves; `perf_24_tracks_3_effects_realtime_factor` prints the realtime factor | | Mixer | `crates/render/src/mixer_tests.rs`, `crates/engine/src/mixer_tests.rs` | sample-exact fader gain and pan laws, automation at block boundaries (bit-identical however requests are cut), solo/mute/solo-safe, sends and submix routing, latency-compensation alignment, render-vs-playback identity, Touch ramp-back, recorder modes (Latch/Touch/Write), thinning, Audio Gain, transition curves; `perf_24_tracks_3_effects_realtime_factor` prints the realtime factor |
| Oracle | `crates/*/tests/*oracle*.rs`, `conformance.rs` | compare with ffmpeg/ffprobe (§2) | | Oracle | `crates/*/tests/*oracle*.rs`, `conformance.rs` | compare with ffmpeg/ffprobe (§2) |
| Robustness / fuzz | `*/tests/robustness.rs`, `*/tests/fuzz.rs` | seeded mutation and truncation of real and synthetic files; nothing may panic | | Robustness / fuzz | `*/tests/robustness.rs`, `*/tests/fuzz.rs` | seeded mutation and truncation of real and synthetic files; nothing may panic |