diff --git a/ROADMAP.md b/ROADMAP.md index c979d906..e5b2f255 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -31,7 +31,7 @@ Status: ✅ done · 🟡 in progress · ⬜ not started. Estimates are remaining | M0 | Skeleton + visual shell | ✅ | Workspace, 20 crates, dock/workspaces, Premiere 26 look, native menus, control channel, MCP, xtask gates (layers, wasm) | — | — | | M1 | Media I/O | ✅ | MP4/MOV demux+mux, WAV, stills, MJPEG, symphonia audio (MP3/FLAC/ALAC/Vorbis), GOP seek + frame cache, import | Media Browser polish | 2 | | M2 | H.264 decoder | ✅ | Own decoder, bit-exact on 37+ streams, 500–600 fps 1080p | — | — | -| M3 | Editing core | 🟡 | Edit algebra (insert/overwrite/razor/lift/extract/ripple/roll/slip/slide/rate-stretch/nest/paste), tools, markers, trim mode + Trim Monitor + dynamic J/K/L trimming, Keyboard Shortcuts editor with FilmCraft/Premiere/FCP/Avid presets | Multicam, subclip polish, ~220 Premiere default shortcuts whose commands don't exist yet | 6–10 | +| M3 | Editing core | 🟡 | Edit algebra (insert/overwrite/razor/lift/extract/ripple/roll/slip/slide/rate-stretch/nest/paste), tools, markers, trim mode + Trim Monitor + dynamic J/K/L trimming, Keyboard Shortcuts editor with FilmCraft/Premiere/FCP/Avid presets; multicam (Create Multi-Camera Source Sequence, Multi-Camera view with live switching on 1–9, angle switching, Enable/Flatten, Edit Cameras) and sync (Synchronize, Merge Clips; In/Out/timecode/marker/audio — GCC-PHAT, sample-accurate) | Multicam paging >16 angles and grid thumbnails, subclip polish, ~220 Premiere default shortcuts whose commands don't exist yet | 6–10 | | M4 | Playback | 🟡 | Audio-clock master, prefetch, J/K/L, dropped-frame stats, playback resolution | Render previews, render bar, 4K/8K tuning | 8–12 | | M5 | Effects, keyframes, GPU | 🟡 | ~60 CPU effects, 30 transitions, keyframes + value/velocity graphs, wgpu compositor | Full ~150-effect catalogue, WGSL parity for all, masks + tracking, adjustment layers, presets, Warp Stabilizer, Morph Cut | 30–45 | | M6 | Export | ✅ | Own H.264 encoder (High/Main/Baseline, B-frames, VBR/CBR/2-pass) → MP4 + own AAC; ProRes, MJPEG, PNG, GIF, WAV; background jobs | Preset library, queue UI, smart render | 6–8 | @@ -48,6 +48,7 @@ Status: ✅ done · 🟡 in progress · ⬜ not started. Estimates are remaining ## Log +- **2026-10-01 (M12 multicam):** Synchronize / Merge Clips / Create Multi-Camera Source Sequence (sync by In, Out, timecode ± hours, clip markers, or audio: GCC-PHAT cross-correlation, offsets recovered to the sample on noisy multi-mic recordings), multi-camera clips (angle render, Switch Audio), Program monitor Multi-Camera view with live switching (keys 1–9, one undo step per pass), Ctrl+1–9 cuts, Enable / Flatten / Edit Cameras, project schema v7. Fixed: nested sequence audio was silent. - **2026-10-01 (evening):** M11 done: offline media with our own slate and the Link Media dialog (fingerprint-checked relink, folder remap, search), proxies (create in background / attach / toggle; export stays full-res) with ingest settings, Project Manager (collect, consolidate + transcode). Fixed: GOP cache deadlock when a rayon decoder inside a parallel export re-entered the same source. - **2026-10-01 (M7.4):** Essential Sound panel: Dialogue/Music/SFX/Ambience types, Loudness auto-match (BS.1770, exact to the target), Repair (noise, rumble, hum, new DeEsser and spectral DeReverb), Clarity (dynamics, EQ presets, Enhance Speech DSP chain), Creative reverb / stereo width, ducking that writes Volume keyframes, presets; all `essentialSound.*` commands, effects visible and keyframable in Effect Controls. - **2026-10-01 (night):** M8 colour: camera log curves and gamuts from published specs, colour-managed pipeline (PQ/HLG working spaces, Interpret Footage, BT.2390 tone mapping), HDR export signalling verified with ffprobe, LUTs (.cube/.3dl, tetrahedral CPU + WGSL, project library, Lumetri Input LUT / Look), Colour Match, Lumetri section bypass, HDR scopes. diff --git a/crates/audio-dsp/README.md b/crates/audio-dsp/README.md index 83928438..c86d548d 100644 --- a/crates/audio-dsp/README.md +++ b/crates/audio-dsp/README.md @@ -89,6 +89,22 @@ Pure functions for Essential Sound auto-ducking: merge; clamped to the clip (a ramp cut by an edge gets the ramp's value at the edge); strictly increasing times. +## Audio synchronisation (`sync`) + +`find_offset(a, b, sample_rate, &SyncOptions)` returns the lag `L` with `b[n] ≈ g·a[n + L]`, a +sub-sample estimate, and two trust figures (`confidence` = peak / RMS of the correlation, +`distinct` = peak / next peak > 50 ms away; `reliable()` = ≥ 8 and ≥ 1.5). + +- Coarse: DC removal + normalisation, Blackman-windowed-sinc decimation to ≤ 8 kHz (further for + long recordings, so the transform stays ≤ 4 M points), GCC-PHAT-β (`R = A·B*/|A·B*|^0.75`, + 40 Hz – 0.4·rate band) with both signals packed into one complex FFT. +- Refine: the same correlation at the full rate on the loudest common window (≤ 2¹⁷ samples), ±2 + coarse steps around the coarse peak, parabolic interpolation. +- Tested within one sample on speech-like multi-microphone signals (gains −20…+10 dB, SNR down to + 3 dB, coloured microphones, fractional delays: worst 0.4 samples integer / 0.11 samples + sub-sample), a 0 dB SNR room with a strong echo, inverted polarity, unrelated signals + (unreliable) and `max_lag`. Two 10-minute recordings: 2.4 s (release, one core). + ## Known limits - Effects assume ≤ the channel count they were built with; extra channels pass through diff --git a/crates/audio-dsp/src/lib.rs b/crates/audio-dsp/src/lib.rs index 9908cf6b..7af30a29 100644 --- a/crates/audio-dsp/src/lib.rs +++ b/crates/audio-dsp/src/lib.rs @@ -6,6 +6,7 @@ //! registry ([`effects()`]) describing every effect's parameters (range, unit, default) so the //! engine and UI can build controls generically. //! * [`biquad`] — RBJ-cookbook biquads (TDF-II) with analytic magnitude response. +//! * [`sync`] — offset between two recordings of one event (GCC-PHAT, sample-accurate). //! //! Conventions: audio is **planar f32** (`&mut [&mut [f32]]`, one slice per channel, all the same //! length). `process` never allocates; all buffers are sized at construction. Parameters may be @@ -21,6 +22,7 @@ pub mod fft; pub mod loudness; pub mod oversample; mod smooth; +pub mod sync; pub use effects::{Category, EffectInfo, create_effect, effect_info, effects}; pub use loudness::{LoudnessMeter, normalize_gain_db}; diff --git a/crates/audio-dsp/src/sync.rs b/crates/audio-dsp/src/sync.rs new file mode 100644 index 00000000..73b6e0e8 --- /dev/null +++ b/crates/audio-dsp/src/sync.rs @@ -0,0 +1,505 @@ +//! Synchronising recordings of the same event by their audio (Synchronize, Merge Clips and +//! Create Multi-Camera Source Sequence "Audio"). +//! +//! [`find_offset`] returns the lag `L` (in samples) for which `b[n] ≈ g · a[n + L]`: when `b` +//! started recording two seconds after `a`, `L = 2 · sample_rate`. It works in two passes: +//! +//! 1. **Coarse.** Both signals are DC-removed, normalised, low-pass filtered (windowed sinc) and +//! decimated to ≤ 8 kHz (further for very long recordings, so the transform stays ≤ 4 M points). +//! The generalised cross-correlation with partial phase transform weighting (GCC-PHAT-β: +//! `R = A·B* / |A·B*|^β`) is computed with one complex FFT of both signals packed as `a + i·b`. +//! PHAT weighting whitens the spectra, so the peak is sharp and independent of the microphones' +//! gains and frequency responses; β < 1 keeps noise-only bins from dominating. +//! 2. **Refine.** Around the coarse peak, the same correlation is computed at the full sample rate +//! on the loudest common window (≤ 2.7 s at 48 kHz), and the peak is refined by parabolic +//! interpolation to a fraction of a sample. +//! +//! Two figures say how much to trust the result: `confidence`, the coarse peak's height over the +//! RMS of the correlation, and `distinct`, its height over the highest peak more than 50 ms away +//! (periodic material such as music or a steady hum can match at several lags). +//! +//! References: Knapp & Carter, "The generalized correlation method for estimation of time delay", +//! IEEE Trans. ASSP 24(4), 1976; the β-weighted PHAT variant is standard textbook practice. + +use crate::fft::Fft; + +/// Tuning of [`find_offset`]. +#[derive(Clone, Copy, Debug, PartialEq)] +pub struct SyncOptions { + /// Largest |lag| considered, in samples (None = anything with enough overlap). + pub max_lag: Option, + /// Upper bound of the coarse search's sample rate (Hz). + pub coarse_rate: u32, + /// Most samples (both signals together) the coarse transform may hold; longer recordings are + /// decimated further. + pub max_coarse_samples: usize, + /// PHAT weighting exponent (1 = full whitening, 0 = plain cross-correlation). + pub phat_beta: f32, + /// Length of the full-rate refinement window (samples). + pub refine_window: usize, +} + +impl Default for SyncOptions { + fn default() -> Self { + Self { max_lag: None, coarse_rate: 8000, max_coarse_samples: 1 << 22, phat_beta: 0.75, refine_window: 1 << 17 } + } +} + +/// The offset found between two recordings. +#[derive(Clone, Copy, Debug, PartialEq)] +pub struct SyncResult { + /// `b[n]` lines up with `a[n + lag]` (samples at the input rate). + pub lag: i64, + /// The same with sub-sample precision (parabolic interpolation of the refined peak). + pub lag_fine: f64, + /// Coarse peak height over the RMS of the correlation. + pub confidence: f32, + /// Coarse peak height over the next-highest peak more than 50 ms away. + pub distinct: f32, + /// True when the signals correlate negatively (one recording has inverted polarity). + pub inverted: bool, +} + +impl SyncResult { + /// Whether the match is clear enough to trust. + pub fn reliable(&self) -> bool { + self.confidence >= 8.0 && self.distinct >= 1.5 + } +} + +/// Average interleaved or planar channels into one mono signal. +pub fn mixdown(channels: &[&[f32]]) -> Vec { + let n = channels.iter().map(|c| c.len()).min().unwrap_or(0); + let k = channels.len().max(1) as f32; + (0..n).map(|i| channels.iter().map(|c| c[i]).sum::() / k).collect() +} + +/// Low-pass filter (Blackman-windowed sinc, cutoff at 0.42 × the new sample rate) and keep every +/// `d`-th sample. +pub fn decimate(x: &[f32], d: usize) -> Vec { + if d <= 1 { + return x.to_vec(); + } + let half = 8 * d; + let fc = 0.42 / d as f64; // cycles per input sample + let taps: Vec = { + let n = 2 * half + 1; + let mut h: Vec = (0..n) + .map(|k| { + let t = k as f64 - half as f64; + let sinc = if t == 0.0 { 2.0 * fc } else { (2.0 * std::f64::consts::PI * fc * t).sin() / (std::f64::consts::PI * t) }; + let w = 0.42 - 0.5 * (2.0 * std::f64::consts::PI * k as f64 / (n - 1) as f64).cos() + + 0.08 * (4.0 * std::f64::consts::PI * k as f64 / (n - 1) as f64).cos(); + sinc * w + }) + .collect(); + let s: f64 = h.iter().sum(); + h.iter_mut().for_each(|v| *v /= s); + h.into_iter().map(|v| v as f32).collect() + }; + let out_len = x.len().div_ceil(d); + let mut out = Vec::with_capacity(out_len); + for i in 0..out_len { + let c = (i * d) as isize; + let lo = (c - half as isize).max(0) as usize; + let hi = ((c + half as isize + 1) as usize).min(x.len()); + let k0 = (lo as isize - (c - half as isize)) as usize; + let mut acc = 0.0f32; + for (j, &v) in x[lo..hi].iter().enumerate() { + acc += v * taps[k0 + j]; + } + out.push(acc); + } + out +} + +/// Remove the mean and scale to unit RMS (silence stays silence). +fn normalise(x: &mut [f32]) { + if x.is_empty() { + return; + } + let mean = (x.iter().map(|&v| v as f64).sum::() / x.len() as f64) as f32; + x.iter_mut().for_each(|v| *v -= mean); + let rms = (x.iter().map(|&v| (v as f64) * (v as f64)).sum::() / x.len() as f64).sqrt(); + if rms > 1e-12 { + let k = (1.0 / rms) as f32; + x.iter_mut().for_each(|v| *v *= k); + } +} + +/// GCC-PHAT-β of `a` and `b` (zero-padded to `n`, a power of two ≥ `a.len() + b.len()`): +/// `r[L mod n] = Σ a[i + L]·b[i]` with whitened spectra. Only bins whose frequency (cycles per +/// sample) lies in `band` contribute. +fn gcc(a: &[f32], b: &[f32], n: usize, beta: f32, band: (f32, f32)) -> Vec { + let fft = Fft::new(n); + let mut re = vec![0.0f32; n]; + let mut im = vec![0.0f32; n]; + re[..a.len()].copy_from_slice(a); + im[..b.len()].copy_from_slice(b); + fft.forward(&mut re, &mut im); + // X = A + iB for real a, b: A(k) = (X(k) + X*(n-k)) / 2, B(k) = (X(k) - X*(n-k)) / 2i. + let weigh = |k: usize, xr: f32, xi: f32, yr: f32, yi: f32| -> (f32, f32) { + // y = X*(n-k) given as (yr, yi) already conjugated + let (ar, ai) = (0.5 * (xr + yr), 0.5 * (xi + yi)); + let (dr, di) = (xr - yr, xi - yi); + let (br, bi) = (0.5 * di, -0.5 * dr); + // R = A·conj(B) + let rr = ar * br + ai * bi; + let ri = ai * br - ar * bi; + let f = k.min(n - k) as f32 / n as f32; + if f < band.0 || f > band.1 { + return (0.0, 0.0); + } + let mag = (rr * rr + ri * ri).sqrt(); + if mag <= 1e-30 { + return (0.0, 0.0); + } + let w = mag.powf(-beta); + (rr * w, ri * w) + }; + for k in 0..=n / 2 { + let j = (n - k) % n; + let (xkr, xki, xjr, xji) = (re[k], im[k], re[j], im[j]); + let rk = weigh(k, xkr, xki, xjr, -xji); + let rj = weigh(j, xjr, xji, xkr, -xki); + re[k] = rk.0; + im[k] = rk.1; + re[j] = rj.0; + im[j] = rj.1; + } + fft.inverse(&mut re, &mut im); + re +} + +fn corr_at(r: &[f32], lag: i64) -> f32 { + let n = r.len() as i64; + r[lag.rem_euclid(n) as usize] +} + +/// Find the lag `L` with `b[n] ≈ g·a[n + L]` (see the module docs). `None` when either signal is +/// too short or silent, or no lag leaves enough overlap. +pub fn find_offset(a: &[f32], b: &[f32], sample_rate: u32, opts: &SyncOptions) -> Option { + let sr = sample_rate.max(1) as usize; + let min_len = (sr / 10).max(64); + if a.len() < min_len || b.len() < min_len { + return None; + } + let energy = |x: &[f32]| x.iter().map(|&v| (v as f64) * (v as f64)).sum::(); + if energy(a) < 1e-12 || energy(b) < 1e-12 { + return None; + } + // ---- coarse + let d = (sr.div_ceil(opts.coarse_rate.max(500) as usize)).max((a.len() + b.len()).div_ceil(opts.max_coarse_samples.max(1 << 12))).max(1); + let mut ca = a.to_vec(); + let mut cb = b.to_vec(); + normalise(&mut ca); + normalise(&mut cb); + let (ca, cb) = (decimate(&ca, d), decimate(&cb, d)); + let (la, lb) = (ca.len() as i64, cb.len() as i64); + let n = ((la + lb) as usize).next_power_of_two(); + let rate_c = sr as f32 / d as f32; + let band = ((40.0 / rate_c).min(0.05), if d > 1 { 0.40 } else { 0.5 }); + let r = gcc(&ca, &cb, n, opts.phat_beta, band); + // valid lags: b[0..lb) against a[L..L+lb) must overlap by a minimum amount + let min_overlap = ((rate_c as i64) / 2).min(la.min(lb) / 4).max(8); + let max_lag_c = opts.max_lag.map(|m| m.abs() / d as i64 + 1); + let lo = -(lb - min_overlap); + let hi = la - min_overlap; + if hi < lo { + return None; + } + let mut best = (0i64, 0.0f32); + let mut sum2 = 0.0f64; + let mut count = 0usize; + for l in lo..=hi { + if max_lag_c.is_some_and(|m| l.abs() > m) { + continue; + } + let v = corr_at(&r, l); + sum2 += (v as f64) * (v as f64); + count += 1; + if v.abs() > best.1.abs() { + best = (l, v); + } + } + if count == 0 || best.1 == 0.0 { + return None; + } + let rms = (sum2 / count as f64).sqrt().max(1e-30) as f32; + let confidence = best.1.abs() / rms; + let guard = ((rate_c * 0.05) as i64).max(2); + let mut second = 0.0f32; + for l in lo..=hi { + if (l - best.0).abs() <= guard || max_lag_c.is_some_and(|m| l.abs() > m) { + continue; + } + second = second.max(corr_at(&r, l).abs()); + } + let distinct = if second > 0.0 { best.1.abs() / second } else { f32::INFINITY }; + let inverted = best.1 < 0.0; + let coarse = best.0 * d as i64; + // ---- refine at the full rate + let margin = (2 * d + 4) as i64; + let (fa, fb) = (a.len() as i64, b.len() as i64); + let ov0 = (-coarse).max(0); + let ov1 = fb.min(fa - coarse); + if ov1 - ov0 < 16 { + return Some(SyncResult { lag: coarse, lag_fine: coarse as f64, confidence, distinct, inverted }); + } + let w = ((ov1 - ov0) as usize).min(opts.refine_window.max(256)) as i64; + // the window of b with the most energy (both sides: a lag-shifted) + let start = { + let hop = (w / 4).max(1); + let pre = |x: &[f32]| { + let mut p = Vec::with_capacity(x.len() + 1); + p.push(0.0f64); + let mut s = 0.0; + for &v in x { + s += (v as f64) * (v as f64); + p.push(s); + } + p + }; + let (pa, pb) = (pre(a), pre(b)); + let mut s = ov0; + let mut best_s = ov0; + let mut best_e = -1.0f64; + while s + w <= ov1 { + let eb = pb[(s + w) as usize] - pb[s as usize]; + let ea = pa[(s + coarse + w) as usize] - pa[(s + coarse) as usize]; + let e = eb.min(ea); + if e > best_e { + best_e = e; + best_s = s; + } + s += hop; + } + best_s + }; + let bw: Vec = b[start as usize..(start + w) as usize].to_vec(); + let aw: Vec = (start + coarse - margin..start + coarse + w + margin).map(|i| if i >= 0 && i < fa { a[i as usize] } else { 0.0 }).collect(); + let mut aw = aw; + let mut bw = bw; + normalise(&mut aw); + normalise(&mut bw); + let n2 = (aw.len() + bw.len()).next_power_of_two(); + let r2 = gcc(&aw, &bw, n2, opts.phat_beta, (20.0 / sr as f32, 0.5)); + // r2[L'] = Σ aw[i + L']·bw[i]; L' = margin + δ + let sign = if inverted { -1.0 } else { 1.0 }; + let mut bl = 0i64; + let mut bv = f32::MIN; + for l in 0..=2 * margin { + let v = corr_at(&r2, l) * sign; + if v > bv { + bv = v; + bl = l; + } + } + let (y0, y1, y2) = (corr_at(&r2, bl - 1) * sign, bv, corr_at(&r2, bl + 1) * sign); + let den = y0 - 2.0 * y1 + y2; + let frac = if den.abs() > 1e-20 { (0.5 * (y0 - y2) / den).clamp(-0.5, 0.5) } else { 0.0 }; + let delta = bl - margin; + let lag = coarse + delta; + Some(SyncResult { lag, lag_fine: lag as f64 + frac as f64, confidence, distinct, inverted }) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::testutil::Rng; + + /// Speech-like test signal: voiced syllables (harmonics of a gliding pitch under an envelope), + /// noise bursts (consonants) and pauses. + fn speechy(sr: f64, secs: f64, seed: u64) -> Vec { + let n = (sr * secs) as usize; + let mut rng = Rng::new(seed); + let mut out = vec![0.0f32; n]; + let mut i = 0usize; + while i < n { + let len = ((0.08 + 0.25 * (rng.uniform() * 0.5 + 0.5) as f64) * sr) as usize; + let kind = rng.uniform(); + let f0 = 110.0 + 120.0 * (rng.uniform() as f64 * 0.5 + 0.5); + let glide = 1.0 + 0.3 * rng.uniform() as f64; + let amp = 0.2 + 0.3 * (rng.uniform() * 0.5 + 0.5); + for k in 0..len.min(n - i) { + let t = k as f64 / sr; + let env = (std::f64::consts::PI * k as f64 / len as f64).sin() as f32; + let v = if kind > 0.3 { + let f = f0 * (1.0 + (glide - 1.0) * k as f64 / len as f64); + (1..8).map(|h| ((2.0 * std::f64::consts::PI * f * h as f64 * t).sin() / h as f64) as f32).sum::() + } else if kind > -0.4 { + rng.uniform() + } else { + 0.0 + }; + out[i + k] = v * env * amp; + } + i += len + ((rng.uniform() * 0.5 + 0.5) * 0.15 * sr as f32) as usize; + } + out + } + + /// `x` delayed by `delay` samples (fractional: windowed-sinc interpolation), scaled by `gain`, + /// optionally coloured by a zero-phase low-pass (`[k, 1-2k, k]`, so the delay stays exact), with + /// white noise `snr_db` below the signal. + fn mic(x: &[f32], delay: f64, gain: f32, lowpass: Option, snr_db: f32, len: usize, seed: u64) -> Vec { + let mut out = vec![0.0f32; len]; + let di = delay.floor() as i64; + let fr = delay - di as f64; + for (n, o) in out.iter_mut().enumerate() { + let src = n as i64 - di; + if fr == 0.0 { + if src >= 0 && (src as usize) < x.len() { + *o = x[src as usize]; + } + } else { + let mut acc = 0.0f64; + for k in -16i64..=16 { + let j = src - k; + if j < 0 || j as usize >= x.len() { + continue; + } + let t = k as f64 - fr; + let s = (std::f64::consts::PI * t).sin() / (std::f64::consts::PI * t); + let w = 0.5 + 0.5 * (std::f64::consts::PI * t / 17.0).cos(); + acc += x[j as usize] as f64 * s * w; + } + *o = acc as f32; + } + } + if let Some(k) = lowpass { + let src = out.clone(); + for i in 1..len.saturating_sub(1) { + out[i] = k * src[i - 1] + (1.0 - 2.0 * k) * src[i] + k * src[i + 1]; + } + } + let rms = crate::testutil::rms(&out) as f32; + let nrms = rms * 10f32.powf(-snr_db / 20.0) * 3f32.sqrt(); + let mut rng = Rng::new(seed); + for v in &mut out { + *v = *v * gain + rng.uniform() * nrms * gain; + } + out + } + + #[test] + fn decimate_keeps_low_frequencies_and_removes_high_ones() { + let sr = 48_000.0; + let low = crate::testutil::sine(300.0, 1.0, sr, 48_000, 0.0); + let high = crate::testutil::sine(7_000.0, 1.0, sr, 48_000, 0.0); + let dl = decimate(&low, 6); + let dh = decimate(&high, 6); + assert_eq!(dl.len(), 8000); + let rl = crate::testutil::rms(&dl[100..7900]); + let rh = crate::testutil::rms(&dh[100..7900]); + assert!((rl - std::f64::consts::FRAC_1_SQRT_2).abs() < 0.01, "{rl}"); + assert!(rh < 0.01, "{rh}"); + } + + #[test] + fn pure_delay_exact_both_signs() { + let sr = 48_000; + let s = speechy(sr as f64, 12.0, 1); + for delay in [0i64, 1, 7, 12_345, 96_001, -4_321, -50_000] { + // b = s shifted: b[n] = s[n + delay] + let b: Vec = (0..400_000i64).map(|n| if n + delay >= 0 && ((n + delay) as usize) < s.len() { s[(n + delay) as usize] } else { 0.0 }).collect(); + let r = find_offset(&s, &b, sr, &SyncOptions::default()).unwrap(); + assert_eq!(r.lag, delay, "{r:?}"); + assert!(r.reliable(), "{r:?}"); + assert!(!r.inverted); + } + } + + /// Three microphones recording one talker: different gains (−20…+10 dB), noise (SNR 25, 10 and + /// 3 dB), one coloured by a low-pass, fractional delays. Every offset within one sample. + #[test] + fn multi_mic_offsets_within_one_sample() { + let sr = 48_000u32; + let src = speechy(sr as f64, 30.0, 7); + let len = 30 * sr as usize; + // mic k records src delayed by d_k: mic_k[n] = src[n - d_k] + let cases: [(f64, f32, Option, f32); 4] = + [(0.0, 1.0, None, 40.0), (2_345.0, 0.1, None, 25.0), (-31_337.4, 3.0, Some(0.3), 10.0), (150_000.6, 0.5, None, 3.0)]; + let mics: Vec> = cases.iter().enumerate().map(|(i, c)| mic(&src, c.0, c.1, c.2, c.3, len, 100 + i as u64)).collect(); + let mut worst = 0.0f64; + let mut worst_fine = 0.0f64; + for (k, c) in cases.iter().enumerate().skip(1) { + // b = mic_k: b[n] = src[n - d_k] = a[n - d_k + d_0] → lag = d_0 - d_k + let expect = cases[0].0 - c.0; + let r = find_offset(&mics[0], &mics[k], sr, &SyncOptions::default()).unwrap(); + let err = (r.lag as f64 - expect).abs(); + let err_fine = (r.lag_fine - expect).abs(); + worst = worst.max(err); + worst_fine = worst_fine.max(err_fine); + assert!(err <= 1.0, "mic {k}: lag {} vs {expect} ({r:?})", r.lag); + assert!(err_fine < 0.75, "mic {k}: fine {} vs {expect}", r.lag_fine); + assert!(r.reliable(), "{r:?}"); + } + eprintln!("multi-mic worst error: {worst} samples (integer lag), {worst_fine:.3} samples (sub-sample estimate)"); + } + + /// A room: the far microphone hears the direct sound plus a strong reflection 23 ms later, at + /// 0 dB SNR. The direct path wins. + #[test] + fn echo_and_heavy_noise() { + let sr = 48_000u32; + let src = speechy(sr as f64, 20.0, 21); + let len = src.len(); + let near = mic(&src, 0.0, 1.0, None, 35.0, len, 1); + let direct = mic(&src, 777.0, 1.0, None, 60.0, len, 2); + let echo = mic(&src, 777.0 + 0.023 * sr as f64, 0.6, None, 60.0, len, 3); + let mut rng = Rng::new(9); + let room: Vec = direct.iter().zip(&echo).map(|(a, b)| a + b).collect(); + let nrms = crate::testutil::rms(&room) as f32 * 3f32.sqrt(); + let far: Vec = room.iter().map(|v| (v + rng.uniform() * nrms) * 0.3).collect(); + let r = find_offset(&near, &far, sr, &SyncOptions::default()).unwrap(); + assert!((r.lag + 777).abs() <= 1, "{r:?}"); + assert!(r.reliable(), "{r:?}"); + } + + /// Two 10-minute recordings (run with `--ignored --nocapture` in release for the timing). + #[test] + #[ignore] + fn long_recordings_timing() { + let sr = 48_000u32; + let src = speechy(sr as f64, 600.0, 31); + let a = mic(&src, 0.0, 1.0, None, 20.0, src.len(), 1); + let b = mic(&src, -1_234_567.0, 0.5, None, 10.0, src.len() - 2_000_000, 2); + let t = std::time::Instant::now(); + let r = find_offset(&a, &b, sr, &SyncOptions::default()).unwrap(); + eprintln!("10 min + 9.3 min at 48 kHz: {:?} in {:.2} s", r, t.elapsed().as_secs_f64()); + assert!((r.lag - 1_234_567).abs() <= 1); + } + + #[test] + fn inverted_polarity_is_found() { + let sr = 48_000; + let s = speechy(sr as f64, 8.0, 3); + let b: Vec = (0..300_000usize).map(|n| if n + 1000 < s.len() { -0.5 * s[n + 1000] } else { 0.0 }).collect(); + let r = find_offset(&s, &b, sr, &SyncOptions::default()).unwrap(); + assert_eq!(r.lag, 1000); + assert!(r.inverted); + } + + #[test] + fn unrelated_signals_are_not_reliable() { + let sr = 48_000; + let a = speechy(sr as f64, 10.0, 11); + let b = speechy(sr as f64, 10.0, 12); + let r = find_offset(&a, &b, sr, &SyncOptions::default()).unwrap(); + assert!(!r.reliable(), "{r:?}"); + assert!(find_offset(&a, &[0.0; 48_000], sr, &SyncOptions::default()).is_none()); + assert!(find_offset(&a[..100], &a, sr, &SyncOptions::default()).is_none()); + } + + #[test] + fn max_lag_limits_the_search() { + let sr = 48_000; + let s = speechy(sr as f64, 10.0, 5); + let b: Vec = s[48_000..].to_vec(); + let r = find_offset(&s, &b, sr, &SyncOptions { max_lag: Some(60_000), ..Default::default() }).unwrap(); + assert_eq!(r.lag, 48_000); + let r = find_offset(&s, &b, sr, &SyncOptions { max_lag: Some(10_000), ..Default::default() }).unwrap(); + assert!(!r.reliable() || r.lag != 48_000); + } +} diff --git a/crates/edit/src/lib.rs b/crates/edit/src/lib.rs index 272d6f53..bf6de640 100644 --- a/crates/edit/src/lib.rs +++ b/crates/edit/src/lib.rs @@ -16,6 +16,7 @@ //! Keyframes are stored in media time, so trims and splits never need to move them. pub mod captions; +pub mod multicam; use std::collections::HashMap; diff --git a/crates/edit/src/multicam.rs b/crates/edit/src/multicam.rs new file mode 100644 index 00000000..77838f9e --- /dev/null +++ b/crates/edit/src/multicam.rs @@ -0,0 +1,213 @@ +//! Multi-camera editing: switching angles, recording cuts live, healing through edits and +//! flattening multi-camera clips into the clips they show. +//! +//! A multi-camera clip is a track item whose [`TrackItem::multicam`] is set; it refers to a +//! multi-camera source sequence. Its angle is a camera index of that source. + +use std::collections::HashMap; + +use filmcraft_project::{ClipId, ItemId, MulticamSel, Sequence, Track, TrackId, TrackItem}; +use filmcraft_time::{Tick, TimeRange}; + +use crate::{EditCtx, EditError, Result, remove_orphan_transitions, split_track_at}; + +/// Whether `it` is a multi-camera clip (enabled or not). +pub fn is_multicam(it: &TrackItem) -> bool { + it.multicam.is_some() +} + +/// Set the angle of multi-camera clips. Returns how many changed. +pub fn set_angle(seq: &mut Sequence, clips: &[ClipId], angle: u32) -> usize { + let mut n = 0; + for t in seq.all_tracks_mut() { + if t.locked { + continue; + } + for it in t.items.iter_mut().filter(|i| clips.contains(&i.id)) { + if let Some(m) = &mut it.multicam + && m.angle != angle + { + m.angle = angle; + n += 1; + } + } + } + n +} + +/// Enable or disable multi-camera behaviour of clips (Multi-Camera ▸ Enable). Clips that are not +/// multi-camera clips are left alone. Returns how many changed. +pub fn set_enabled(seq: &mut Sequence, clips: &[ClipId], enabled: bool) -> usize { + let mut n = 0; + for t in seq.all_tracks_mut() { + for it in t.items.iter_mut().filter(|i| clips.contains(&i.id)) { + if let Some(m) = &mut it.multicam + && m.enabled != enabled + { + m.enabled = enabled; + n += 1; + } + } + } + n +} + +/// Two adjacent pieces of one multi-camera clip that a cut between them would not change. +fn continuous(a: &TrackItem, b: &TrackItem) -> bool { + a.end() == b.start + && a.item == b.item + && a.multicam.is_some() + && a.multicam == b.multicam + && a.speed == b.speed + && !a.reverse + && !b.reverse + && a.frame_hold.is_none() + && b.frame_hold.is_none() + && a.source_out() == b.source_in + && a.enabled == b.enabled + && a.gain_db == b.gain_db + && a.effects == b.effects +} + +/// Remove through edits between pieces of the same multi-camera clip showing the same angle, +/// for edit points in `range` (inclusive of its ends). Returns the ids of the removed pieces. +pub fn heal_track(track: &mut Track, range: TimeRange) -> Vec { + let mut removed = Vec::new(); + let mut i = 0; + while i + 1 < track.items.len() { + let (a, b) = (&track.items[i], &track.items[i + 1]); + let at = b.start; + if at >= range.start && at <= range.end() && continuous(a, b) { + let (aid, bid) = (a.id, b.id); + let dur = b.duration; + track.items[i].duration += dur; + track.items.remove(i + 1); + track.transitions.retain(|tr| !(tr.from == Some(aid) && tr.to == Some(bid))); + for tr in &mut track.transitions { + if tr.from == Some(bid) { + tr.from = Some(aid); + } + if tr.to == Some(bid) { + tr.to = Some(aid); + } + } + removed.push(bid); + } else { + i += 1; + } + } + remove_orphan_transitions(track); + removed +} + +/// One live switch: from `time` on, show `angle`. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct Cut { + pub time: Tick, + pub angle: u32, +} + +/// Apply a recording pass: on `tracks`, the multi-camera clips of `source` get angle `cuts[i].angle` from +/// `cuts[i].time` until the next cut, the last one until `end`. Clips are split at each cut and at +/// `end`; afterwards through edits inside the recorded range are healed, so pressing the angle that +/// is already showing adds no edit. Locked tracks are skipped. Returns the number of edit points +/// in the recorded range after the pass. +pub fn record(seq: &mut Sequence, tracks: &[TrackId], source: ItemId, cuts: &[Cut], end: Tick, ctx: &mut EditCtx) -> usize { + let mut cuts: Vec = cuts.iter().copied().filter(|c| c.time < end).collect(); + cuts.sort_by_key(|c| c.time); + // two presses at one time: the later wins + let mut dedup: Vec = Vec::new(); + for c in cuts { + match dedup.last_mut() { + Some(l) if l.time == c.time => *l = c, + _ => dedup.push(c), + } + } + let Some(first) = dedup.first().map(|c| c.time) else { return 0 }; + let mut links = HashMap::new(); + let mut points = 0; + for tr in seq.all_tracks_mut() { + if tr.locked || !tracks.contains(&tr.id) { + continue; + } + let mc_at = |tr: &Track, t: Tick| tr.items.iter().any(|i| i.start < t && t < i.end() && is_multicam(i) && i.item == source); + for t in dedup.iter().map(|c| c.time).chain([end]) { + if mc_at(tr, t) { + split_track_at(tr, t, ctx, &mut links); + } + } + for (k, c) in dedup.iter().enumerate() { + let r_end = dedup.get(k + 1).map_or(end, |n| n.time); + for it in tr.items.iter_mut().filter(|i| i.start >= c.time && i.end() <= r_end && i.item == source) { + if let Some(m) = &mut it.multicam { + m.angle = c.angle; + } + } + } + heal_track(tr, TimeRange::from_bounds(first, end)); + points += tr.items.iter().filter(|i| i.start > first && i.start < end).count(); + } + points +} + +/// The clips a multi-camera (or any nested) clip shows, as track items of the outer sequence: +/// the parts of `source_track`'s items inside the clip's source range, moved to timeline time. +/// Inner links are remapped through `links` (so a flattened video/audio pair stays linked). +/// Fails for clips with a speed change, reverse or frame hold (no exact flattening). +pub fn flatten_items(outer: &TrackItem, source_track: &Track, ctx: &mut EditCtx, links: &mut HashMap) -> Result> { + if (outer.speed - 1.0).abs() > 1e-9 || outer.reverse || outer.frame_hold.is_some() { + return Err(EditError::Other("can't flatten a clip with a speed change, reverse or frame hold".into())); + } + let (s0, s1) = (outer.source_in, outer.source_out()); + let mut out = Vec::new(); + for ni in source_track.items.iter().filter(|i| i.start < s1 && i.end() > s0) { + let mut it = ni.clone(); + if it.start < s0 { + let cut = s0 - it.start; + it.source_in += Tick((cut.0 as f64 * it.speed.abs()).round() as i64); + it.duration -= cut; + it.start = s0; + } + if it.end() > s1 { + it.duration = s1 - it.start; + } + if it.duration <= Tick::ZERO { + continue; + } + it.start = outer.start + (it.start - s0); + it.id = ClipId(ctx.alloc()); + it.link = it.link.map(|l| *links.entry(l).or_insert_with(|| ctx.alloc())); + it.group = None; + // the outer clip's look carries over: its standard effects follow the inner ones, and its + // Motion/Opacity/Volume replace the inner ones when only the outer clip changed them + let take_intrinsics = outer.has_modified_intrinsics() && !it.has_modified_intrinsics(); + for e in &outer.effects { + let intrinsic = e.def().is_some_and(|d| d.intrinsic); + if !intrinsic { + it.effects.push(e.clone()); + } else if take_intrinsics && let Some(slot) = it.effects.iter_mut().find(|x| x.effect == e.effect) { + *slot = e.clone(); + } + } + out.push(it); + } + Ok(out) +} + +/// Replace a multi-camera clip on `track` by `pieces` (from [`flatten_items`]); pieces whose +/// track is not `track` go to the tracks given with them. Returns the new ids. +pub fn replace_with(seq: &mut Sequence, clip: ClipId, pieces: Vec<(TrackId, TrackItem)>, ctx: &mut EditCtx) -> Result> { + let (tid, _) = seq.find_item(clip).ok_or(EditError::NoItem(clip))?; + let t = seq.track_mut(tid).ok_or(EditError::NoTrack(tid))?; + if t.locked { + return Err(EditError::Locked); + } + t.items.retain(|i| i.id != clip); + remove_orphan_transitions(t); + crate::overwrite(seq, pieces, ctx) +} + +/// The multi-camera selection a freshly edited multi-camera clip gets. +pub fn default_sel(angle: u32) -> MulticamSel { + MulticamSel { enabled: true, angle } +} diff --git a/crates/edit/src/tests.rs b/crates/edit/src/tests.rs index 9ca16a77..de336128 100644 --- a/crates/edit/src/tests.rs +++ b/crates/edit/src/tests.rs @@ -55,6 +55,7 @@ impl Fx { frame_hold: None, scale_to_frame: false, essential: None, + multicam: None, } } fn put(&mut self, track: TrackId, start: i64, dur: i64, src_in: i64) -> ClipId { diff --git a/crates/engine/src/commands.rs b/crates/engine/src/commands.rs index 758cb08d..fadfaadb 100644 --- a/crates/engine/src/commands.rs +++ b/crates/engine/src/commands.rs @@ -75,7 +75,7 @@ fn has_recovery(s: &Session) -> std::result::Result<(), String> { pub(crate) fn has_seq(s: &Session) -> std::result::Result<(), String> { s.active_sequence().map(|_| ()).ok_or_else(|| "no sequence is open".into()) } -fn has_selection(s: &Session) -> std::result::Result<(), String> { +pub(crate) fn has_selection(s: &Session) -> std::result::Result<(), String> { has_seq(s)?; if s.state.selection.is_empty() { Err("no clips selected".into()) } else { Ok(()) } } @@ -102,7 +102,7 @@ fn has_previews(s: &Session) -> std::result::Result<(), String> { has_seq(s)?; if s.previews.count() == 0 { Err("there are no render files".into()) } else { Ok(()) } } -fn has_project_selection(s: &Session) -> std::result::Result<(), String> { +pub(crate) fn has_project_selection(s: &Session) -> std::result::Result<(), String> { if s.state.project_selection.is_empty() { Err("select an item in the Project panel".into()) } else { Ok(()) } } fn has_clipboard(s: &Session) -> std::result::Result<(), String> { @@ -152,13 +152,13 @@ pub(crate) fn time_p(s: &Session, p: &Value, prefix: &str) -> Option { pub(crate) fn clip_p(p: &Value, k: &str) -> Option { u64_p(p, k).map(ClipId) } -fn clips_p(s: &Session, p: &Value) -> Vec { +pub(crate) fn clips_p(s: &Session, p: &Value) -> Vec { match p.get("clips").and_then(Value::as_array) { Some(a) => a.iter().filter_map(|v| v.as_u64().map(ClipId)).collect(), None => clip_p(p, "clip").map(|c| vec![c]).unwrap_or_else(|| s.state.selection.clone()), } } -fn track_p(s: &Session, p: &Value, k: &str) -> Option { +pub(crate) fn track_p(s: &Session, p: &Value, k: &str) -> Option { let v = p.get(k)?; if let Some(id) = v.as_u64() { return Some(TrackId(id)); @@ -192,7 +192,7 @@ pub fn with_links(s: &Session, clips: &[ClipId]) -> Vec { out } -fn default_seq_settings_for(info: &filmcraft_media::MediaInfo) -> SequenceSettings { +pub(crate) fn default_seq_settings_for(info: &filmcraft_media::MediaInfo) -> SequenceSettings { let mut st = SequenceSettings::default(); if let Some(v) = &info.video { st.width = v.width; @@ -1876,6 +1876,7 @@ fn build() -> Vec { ]; v.extend(crate::captions::commands()); v.extend(crate::mixer::commands()); + v.extend(crate::multicam::commands()); v.extend(crate::essential_sound::commands()); v.extend(crate::color::commands()); v.extend(crate::graphics::commands()); diff --git a/crates/engine/src/lib.rs b/crates/engine/src/lib.rs index b64991ef..cad14791 100644 --- a/crates/engine/src/lib.rs +++ b/crates/engine/src/lib.rs @@ -20,12 +20,14 @@ pub mod masks; pub mod media_pool; pub mod mixer; pub mod presets; +pub mod multicam; pub mod previews; pub mod project_manager; pub mod proxies; pub mod relink; pub mod shortcut_presets; pub mod shortcuts; +pub mod sync; pub mod trim; use std::sync::Arc; @@ -173,6 +175,10 @@ pub struct EditorState { /// The mask being edited on the Program monitor (Effect Controls selection). #[serde(default)] pub selected_mask: Option, + /// Multi-Camera Audio Follows Video: switching a multi-camera clip's angle switches its linked + /// audio clips too. + #[serde(default)] + pub multicam_audio_follows_video: bool, } /// Events for frontends (drained each frame). @@ -210,6 +216,8 @@ pub struct Session { pub previews: Arc, /// Audio Track Mixer automation pass in progress. pub mixrec: mixer::Recorder, + /// Multi-camera live switching pass in progress. + pub mcrec: multicam::Recorder, /// Dynamic (J/K/L) trimming and trim-mode loop playback in progress. pub trim_play: trim::TrimPlayback, /// Keyboard shortcuts (active bindings, presets; `shortcuts.*` commands). @@ -284,6 +292,7 @@ impl Session { loaded_schema: filmcraft_format::SCHEMA_VERSION, previews: Arc::new(previews::PreviewStore::temp()), mixrec: Default::default(), + mcrec: Default::default(), trim_play: Default::default(), shortcuts: shortcuts::Shortcuts::new(), offline: Default::default(), @@ -649,6 +658,8 @@ mod mixer_tests; #[cfg(test)] mod presets_tests; #[cfg(test)] +mod multicam_tests; +#[cfg(test)] mod previews_tests; #[cfg(test)] mod project_manager_tests; diff --git a/crates/engine/src/multicam.rs b/crates/engine/src/multicam.rs new file mode 100644 index 00000000..6071eab6 --- /dev/null +++ b/crates/engine/src/multicam.rs @@ -0,0 +1,968 @@ +//! Multi-camera editing and clip synchronisation commands. +//! +//! | Id | Menu | | +//! |---|---|---| +//! | `clip.synchronize` | Clip ▸ Synchronize… | align selected timeline clips to a reference by In, Out, timecode, clip marker or audio | +//! | `clip.mergeClips` | Clip ▸ Merge Clips… | one video + audio clips → a merged clip | +//! | `clip.createMulticam` | Clip ▸ Create Multi-Camera Source Sequence… | synchronised cameras → a multi-camera source sequence | +//! | `clip.multicamEnable` | Clip ▸ Multi-Camera ▸ Enable | toggle multi-camera behaviour of nested clips | +//! | `clip.multicamFlatten` | Clip ▸ Multi-Camera ▸ Flatten | replace multi-camera clips by the clips their angle shows | +//! | `multicam.switchAngle` | | set the angle of clips (keys 1–9 when stopped, clicking an angle) | +//! | `multicam.recordStart` / `multicam.cut` / `multicam.recordStop` | | live switching while playing: one undo step per pass | +//! | `multicam.selectCamera1`…`9` | keys 1–9 | while recording: cut to camera N; stopped: switch the clip at the playhead | +//! | `multicam.cutToCamera1`…`9` | Ctrl+1–9 | add an edit at the playhead and show camera N from there | +//! | `multicam.audioFollowsVideo` | | switching video switches the linked audio clips too | +//! | `multicam.editCameras` | | rename / show / hide cameras, change the audio mode | +//! | `multicam.inspect` | | the multi-camera clip at the playhead, its cameras and the recorder state | +//! +//! Sync methods are described in [`crate::sync`]. + +use std::collections::HashMap; +use std::sync::Arc; + +use filmcraft_edit as edit; +use filmcraft_edit::EditCtx; +use filmcraft_edit::multicam::Cut; +use filmcraft_project::{ + BinEntry, Camera, ClipId, ItemId, ItemKind, Label, MergedClip, MulticamAudio, MulticamSel, MulticamSource, Project, Sequence, SequenceSettings, Track, + TrackId, TrackItem, TrackKind, resolve_auto_points, +}; +use filmcraft_time::{FrameRate, Tick, TimeRange}; +use serde_json::{Value, json}; + +use crate::commands::{CommandSpec, always, bad, bool_p, clips_p, has_selection, has_seq, item_p, str_p, time_p, track_p, u64_p, with_links}; +use crate::sync::{self, Method, SyncClip}; +use crate::{EngineError, MediaPool, Result, Session}; + +type Run = fn(&mut Session, &Value) -> Result; +type Enabled = fn(&Session) -> std::result::Result<(), String>; + +#[allow(clippy::too_many_arguments)] +fn spec(id: &'static str, label: &'static str, menu: &'static [&'static str], params: &'static str, enabled: Enabled, run: Run, journal: bool) -> CommandSpec { + CommandSpec { id, label, menu, shortcut: None, params, enabled, run, journal } +} + +macro_rules! camera_keys { + ($($n:literal),*) => { + [$( + CommandSpec { + id: concat!("multicam.selectCamera", $n), + label: concat!("Select Camera ", $n), + menu: &[], + shortcut: Some(concat!($n)), + params: r#"{"videoOnly":bool?}"#, + enabled: has_seq, + run: |s, p| cut(s, &with_camera(p, $n)), + journal: true, + }, + CommandSpec { + id: concat!("multicam.cutToCamera", $n), + label: concat!("Cut to Camera ", $n), + menu: &[], + shortcut: Some(concat!("Ctrl+", $n)), + params: r#"{"videoOnly":bool?}"#, + enabled: has_seq, + run: |s, p| cut_to_camera(s, &with_camera(p, $n)), + journal: true, + }, + )*] + }; +} + +fn with_camera(p: &Value, n: u64) -> Value { + let mut v = if p.is_object() { p.clone() } else { json!({}) }; + v["camera"] = json!(n); + v +} + +pub fn commands() -> Vec { + let mut v = base_commands(); + v.extend(camera_keys!(1, 2, 3, 4, 5, 6, 7, 8, 9)); + v +} + +fn base_commands() -> Vec { + vec![ + spec( + "clip.synchronize", + "Synchronize…", + &["Clip"], + r#"{"method":"in|out|timecode|marker|audio","clips":[id]?,"reference":clip?,"track":"V1"|"A1"?,"ignoreHours":bool?,"marker":str?,"offset":frames?}"#, + can_synchronize, + synchronize, + true, + ), + spec( + "clip.mergeClips", + "Merge Clips…", + &["Clip"], + r#"{"items":[id]?,"method":"in|out|timecode|marker|audio","name":str?,"removeVideoAudio":bool?,"ignoreHours":bool?,"marker":str?,"offset":frames?}"#, + has_two_items, + merge_clips, + true, + ), + spec( + "clip.createMulticam", + "Create Multi-Camera Source Sequence…", + &["Clip"], + r#"{"items":[id]?,"name":str?,"method":"in|out|timecode|marker|audio","ignoreHours":bool?,"marker":str?,"offset":frames?,"audio":"camera1|all|switch","cameraNames":"clip|track|metadata","processedBin":bool?,"reference":item?}"#, + has_items, + create_multicam, + true, + ), + spec("clip.multicamEnable", "Enable", &["Clip", "Multi-Camera"], r#"{"clips":[id]?,"enabled":bool?}"#, has_selection, enable, true), + spec("clip.multicamFlatten", "Flatten", &["Clip", "Multi-Camera"], r#"{"clips":[id]?}"#, has_selection, flatten, true), + spec( + "multicam.switchAngle", + "Switch Multi-Camera Angle", + &[], + r#"{"camera":1..16 (shown order)|"angle":0-based,"clips":[id]?,"time":ticks?,"videoOnly":bool?,"audioOnly":bool?}"#, + has_seq, + switch_angle, + true, + ), + spec("multicam.recordStart", "Start Multi-Camera Recording", &[], r#"{"time":ticks?}"#, has_seq, record_start, false), + spec("multicam.cut", "Cut to Camera", &[], r#"{"camera":1..16|"angle":0-based,"time":ticks?,"videoOnly":bool?}"#, has_seq, cut, true), + spec("multicam.recordStop", "Stop Multi-Camera Recording", &[], r#"{"time":ticks?}"#, always, record_stop, true), + spec("multicam.audioFollowsVideo", "Multi-Camera Audio Follows Video", &[], r#"{"enabled":bool?}"#, always, audio_follows, true), + spec( + "multicam.editCameras", + "Edit Cameras…", + &[], + r#"{"sequence":id?,"cameras":[{"angle":0-based,"name":str?,"enabled":bool?}]?,"audio":"camera1|all|switch"?}"#, + always, + edit_cameras, + true, + ), + spec("multicam.cutToCamera", "Cut to Camera", &[], r#"{"camera":1..16|"angle":0-based,"time":ticks?,"videoOnly":bool?}"#, has_seq, cut_to_camera, true), + CommandSpec { + id: "multicam.inspect", + label: "Inspect Multi-Camera", + menu: &[], + shortcut: None, + params: r#"{"time":ticks?}"#, + enabled: always, + run: inspect, + journal: false, + }, + ] +} + +// ---------------------------------------------------------------- enablement + +fn can_synchronize(s: &Session) -> std::result::Result<(), String> { + has_seq(s)?; + let sel = with_links(s, &s.state.selection); + if sel.len() < 2 { Err("select at least two clips to synchronize".into()) } else { Ok(()) } +} +fn has_items(s: &Session) -> std::result::Result<(), String> { + if s.state.project_selection.iter().any(|i| sync::project_clip(&s.project, *i).is_some()) { + Ok(()) + } else { + Err("select clips in the Project panel".into()) + } +} +fn has_two_items(s: &Session) -> std::result::Result<(), String> { + if s.state.project_selection.iter().filter(|i| sync::project_clip(&s.project, **i).is_some()).count() >= 2 { + Ok(()) + } else { + Err("select a video clip and audio clips in the Project panel".into()) + } +} + +// ---------------------------------------------------------------- helpers + +fn items_p(s: &Session, p: &Value) -> Vec { + match p.get("items").and_then(Value::as_array) { + Some(a) => a.iter().filter_map(|v| v.as_u64().map(ItemId)).collect(), + None => s.state.project_selection.clone(), + } +} + +fn method_p(p: &Value, cmd: &str) -> Result { + Method::from_params(p).map_err(|e| bad(cmd, e)) +} + +fn offset_p(p: &Value, rate: FrameRate) -> Tick { + p.get("offset").and_then(Value::as_i64).map(|f| rate.tick_of(f)).unwrap_or(Tick::ZERO) +} + +/// Run `f` on sequence `seq_id` of `p` with an edit context (ids, media durations), then check it. +fn seq_edit(p: &mut Project, seq_id: ItemId, media: &Arc, f: impl FnOnce(&mut Sequence, &mut EditCtx) -> Result) -> Result { + let snapshot = p.clone(); + let media = media.clone(); + let durations = move |id: ItemId| crate::media_duration(&snapshot, &media, id); + let min = p.sequence(seq_id).map(|s| s.settings.frame_rate.frame_duration()).unwrap_or(Tick(1)); + let mut next = p.next_id; + let r = { + let seq = p.sequence_mut(seq_id).ok_or(EngineError::NoSequence)?; + let mut ctx = EditCtx { next_id: &mut next, media_duration: &durations, min_duration: min }; + f(seq, &mut ctx)? + }; + p.next_id = next; + if let Some(q) = p.sequence(seq_id) { + q.check().map_err(EngineError::Other)?; + } + Ok(r) +} + +/// Frame-aligned placement that keeps sub-frame (sample) accuracy: a clip whose media range should +/// start at the exact timeline time `exact` starts at the next frame boundary instead, with its +/// source In moved by the difference. Returns (start, source In shift). +fn frame_align(exact: Tick, rate: FrameRate) -> (Tick, Tick) { + let snapped = rate.snap(exact); + let start = if snapped < exact { snapped + rate.frame_duration() } else { snapped }; + (start, start - exact) +} + +/// A track item showing `range` of `item` whose media time `range.start` sits at timeline `exact`. +fn placed_item(p: &mut Project, item: ItemId, kind: TrackKind, exact: Tick, range: TimeRange, settings: &SequenceSettings) -> Option { + let rate = settings.frame_rate; + let (start, shift) = frame_align(exact, rate); + let src = TimeRange::from_bounds(range.start + shift, range.end()); + let mut ti = p.make_track_item(item, kind, start, src, rate)?; + // whole frames that fit in the media + let frames = rate.frame_at(src.duration).max(1); + ti.duration = rate.tick_of(frames); + if kind == TrackKind::Video { + let size = sync_source_size(p, item).unwrap_or((settings.width, settings.height)); + for e in &mut ti.effects { + resolve_auto_points(e, (settings.width, settings.height), size); + } + } + Some(ti) +} + +fn sync_source_size(p: &Project, item: ItemId) -> Option<(u32, u32)> { + match &p.item(item)?.kind { + ItemKind::Media(m) => m.info.video.as_ref().map(|v| (v.width, v.height)), + ItemKind::Subclip { parent, .. } => sync_source_size(p, *parent), + _ => None, + } +} + +// ---------------------------------------------------------------- Synchronize + +/// A set of linked timeline clips that move together, with the clip used to sync it. +struct Unit { + members: Vec, + rep: TrackItem, + rep_track: (TrackKind, usize), +} + +fn synchronize(s: &mut Session, p: &Value) -> Result { + let cmd = "clip.synchronize"; + let method = method_p(p, cmd)?; + let seq_id = s.state.active_sequence.ok_or(EngineError::NoSequence)?; + let q = s.active_sequence().ok_or(EngineError::NoSequence)?.clone(); + let rate = q.settings.frame_rate; + let sel = with_links(s, &clips_p(s, p)); + // group into link units + let mut units: Vec = Vec::new(); + let mut seen: Vec = Vec::new(); + let pos = |c: ClipId| -> Option<(TrackKind, usize, TrackItem)> { + for (k, ts) in [(TrackKind::Video, &q.video_tracks), (TrackKind::Audio, &q.audio_tracks)] { + for (ti, t) in ts.iter().enumerate() { + if let Some(it) = t.item(c) { + return Some((k, ti, it.clone())); + } + } + } + None + }; + for &c in &sel { + if seen.contains(&c) { + continue; + } + let Some((_, _, it)) = pos(c) else { continue }; + let members: Vec = match it.link { + Some(l) => sel.iter().copied().filter(|o| pos(*o).is_some_and(|x| x.2.link == Some(l))).collect(), + None => vec![c], + }; + seen.extend(&members); + let mut cands: Vec<(TrackKind, usize, TrackItem)> = members.iter().filter_map(|m| pos(*m)).collect(); + // the representative: audio for audio sync, else video; lowest track first + let audio_first = method == Method::Audio; + cands.sort_by_key(|(k, ti, _)| (if (*k == TrackKind::Audio) == audio_first { 0 } else { 1 }, *ti)); + let (k, ti, rep) = cands.remove(0); + units.push(Unit { members, rep, rep_track: (k, ti) }); + } + if units.len() < 2 { + return Err(bad(cmd, "select clips from at least two different sources")); + } + // reference unit + let reference = if let Some(c) = u64_p(p, "reference").map(ClipId) { + units.iter().position(|u| u.members.contains(&c)).ok_or_else(|| bad(cmd, "`reference` is not among the clips"))? + } else if let Some(t) = track_p(s, p, "track") { + units + .iter() + .position(|u| u.members.iter().any(|m| q.track(t).is_some_and(|tr| tr.item(*m).is_some()))) + .ok_or_else(|| bad(cmd, "no selected clip on the reference track"))? + } else { + (0..units.len()).min_by_key(|&i| (if units[i].rep_track.0 == TrackKind::Video { 0 } else { 1 }, units[i].rep_track.1)).unwrap_or(0) + }; + let clips: Vec = units.iter().map(|u| sync::timeline_clip(&s.project, &u.rep)).collect(); + let anchors = sync::anchors(s, &clips, reference, &method, offset_p(p, rate))?; + // timeline time of each unit's anchor must equal the reference's + let at = |u: &Unit, a: Tick| u.rep.start + Tick(((a - u.rep.source_in).0 as f64 / u.rep.speed.abs().max(1e-9)).round() as i64); + let t_ref = at(&units[reference], anchors[reference].anchor); + let mut deltas: Vec = units.iter().zip(&anchors).map(|(u, a)| t_ref - at(u, a.anchor)).collect(); + // nothing may move before the sequence start: shift everything (reference too) right + let min_start = units + .iter() + .zip(&deltas) + .flat_map(|(u, d)| u.members.iter().filter_map(|m| pos(*m)).map(move |x| x.2.start + *d).collect::>()) + .min() + .unwrap_or(Tick::ZERO); + if min_start < Tick::ZERO { + deltas.iter_mut().for_each(|d| *d -= min_start); + } + let mut moved = 0usize; + let mut placements: Vec<(TrackId, TrackItem)> = Vec::new(); + let mut remove: Vec = Vec::new(); + for (u, d) in units.iter().zip(&deltas) { + if *d == Tick::ZERO { + continue; + } + moved += 1; + // frame-aligned start, sub-frame remainder taken from the source In + let (start, shift) = frame_align(u.rep.start + *d, rate); + let dd = start - u.rep.start; + for m in &u.members { + let Some((tid, it)) = q.find_item(*m) else { continue }; + let mut it = it.clone(); + it.start += dd; + if shift > Tick::ZERO { + it.source_in += Tick((shift.0 as f64 * it.speed.abs()).round() as i64); + it.duration = (it.duration - rate.frame_duration()).max(rate.frame_duration()); + } + remove.push(*m); + placements.push((tid, it)); + } + } + let starts: Vec = units.iter().zip(&deltas).map(|(u, d)| u.rep.start + *d).collect(); + let rep = sync::report(&clips, &anchors, &starts, rate); + if moved > 0 { + let media = s.media.clone(); + s.edit(&format!("Synchronize ({})", method.name()), |pr, st| { + seq_edit(pr, seq_id, &media, |seq, ctx| { + edit::delete_items(seq, &remove); + let ids = edit::overwrite(seq, placements, ctx)?; + Ok(ids) + }) + .map(|ids| { + st.selection = ids; + }) + })?; + } + Ok(json!({"moved": moved, "reference": reference, "clips": rep})) +} + +// ---------------------------------------------------------------- Merge Clips / Create Multi-Camera + +fn camera_name(p: &Project, item: ItemId, mode: &str, n: usize) -> String { + let it = p.item(item); + let clip = || it.map(|i| i.name.clone()).unwrap_or_else(|| format!("Camera {n}")); + match mode { + "track" | "tracks" | "trackname" | "trackNames" | "enumerate" => format!("Camera {n}"), + "metadata" | "cameraAngle" | "label" => it + .and_then(|i| { + ["Camera Angle", "Camera Label", "Camera", "camera", "cameraAngle"].iter().find_map(|k| i.metadata.get(*k).filter(|v| !v.is_empty()).cloned()) + }) + .unwrap_or_else(clip), + _ => clip(), + } +} + +/// Resolve and order the clips for Merge Clips / Create Multi-Camera. +fn sync_items(s: &Session, items: &[ItemId], cmd: &str) -> Result> { + let mut out = Vec::new(); + for i in items { + let c = sync::project_clip(&s.project, *i).ok_or_else(|| bad(cmd, format!("item {} is not a clip", i.0)))?; + if !out.iter().any(|o: &SyncClip| o.item == c.item) { + out.push(c); + } + } + if out.is_empty() { + return Err(bad(cmd, "no clips")); + } + Ok(out) +} + +fn settings_for(s: &Session, item: ItemId) -> SequenceSettings { + let info = match s.project.item(item).map(|i| &i.kind) { + Some(ItemKind::Media(m)) => Some(m.info.clone()), + Some(ItemKind::Subclip { parent, .. }) => s.project.item(*parent).and_then(|i| i.as_media()).map(|m| m.info.clone()), + _ => None, + }; + let mut st = info.as_ref().map(crate::commands::default_seq_settings_for).unwrap_or_default(); + if let Some(m) = s.project.item(item).and_then(|i| i.as_media()) + && let Some(r) = m.interpret.frame_rate + { + st.frame_rate = r; + } + st +} + +fn processed_bin(p: &mut Project) -> filmcraft_project::BinId { + if let Some(b) = p.root.children.iter().find_map(|c| match c { + BinEntry::Bin(b) if b.name == "Processed Clips" => Some(b.id), + _ => None, + }) { + return b; + } + p.add_bin("Processed Clips", None) +} + +fn merge_clips(s: &mut Session, p: &Value) -> Result { + let cmd = "clip.mergeClips"; + let method = method_p(p, cmd)?; + let clips = sync_items(s, &items_p(s, p), cmd)?; + let has_v = |c: &SyncClip| s.project.item(c.item).is_some_and(|i| i.has_video()); + let has_a = |c: &SyncClip| s.project.item(c.item).is_some_and(|i| i.has_audio()); + let videos: Vec = (0..clips.len()).filter(|&i| has_v(&clips[i])).collect(); + if videos.len() > 1 { + return Err(bad(cmd, "Merge Clips takes one video clip plus audio clips")); + } + let audio_n = clips.iter().filter(|c| !has_v(c)).count(); + if audio_n == 0 || audio_n > 16 { + return Err(bad(cmd, "Merge Clips takes 1–16 audio clips")); + } + // the video clip (or the first audio clip) is the reference + let reference = videos.first().copied().unwrap_or(0); + let mut order: Vec = vec![reference]; + order.extend((0..clips.len()).filter(|&i| i != reference)); + let clips: Vec = order.iter().map(|&i| clips[i].clone()).collect(); + let settings = settings_for(s, clips[0].item); + let rate = settings.frame_rate; + let anchors = sync::anchors(s, &clips, 0, &method, offset_p(p, rate))?; + let starts = sync::placements(&clips, &anchors, rate, false); + let remove_video_audio = bool_p(p, "removeVideoAudio").unwrap_or(false); + let name = str_p(p, "name").map(str::to_string).unwrap_or_else(|| format!("{} - Merged", clips[0].name)); + let video = videos.first().map(|_| clips[0].item); + let vid_audio = video.is_some() && has_a(&clips[0]) && !remove_video_audio; + let audio_items: Vec = (1..clips.len()).collect(); + let n_audio = audio_items.len() + vid_audio as usize; + let rep = sync::report(&clips, &anchors, &starts, rate); + let bin = s.project.root.parent_of(clips[0].item).filter(|b| *b != s.project.root.id); + let id = s.edit("Merge Clips", |pr, st| { + let sid = pr.new_sequence(&name, settings.clone(), video.is_some() as usize, n_audio, bin); + let mut v_items = Vec::new(); + let mut a_items = Vec::new(); + let link = pr.alloc_id(); + if video.is_some() { + let mut v = placed_item(pr, clips[0].item, TrackKind::Video, starts[0], clips[0].range, &settings).ok_or_else(|| bad(cmd, "bad video clip"))?; + v.link = Some(link); + v_items.push(v); + if vid_audio { + let mut a = placed_item(pr, clips[0].item, TrackKind::Audio, starts[0], clips[0].range, &settings).ok_or_else(|| bad(cmd, "bad clip"))?; + a.link = Some(link); + a_items.push(a); + } + } + for &i in &audio_items { + let mut a = placed_item(pr, clips[i].item, TrackKind::Audio, starts[i], clips[i].range, &settings).ok_or_else(|| bad(cmd, "bad audio clip"))?; + a.link = Some(link); + a_items.push(a); + } + let q = pr.sequence_mut(sid).expect("new sequence"); + for (t, it) in q.video_tracks.iter_mut().zip(v_items) { + t.items.push(it); + } + for (t, it) in q.audio_tracks.iter_mut().zip(a_items) { + t.items.push(it); + } + q.merged = Some(MergedClip { video, audio: audio_items.iter().map(|&i| clips[i].item).collect(), sync: method.name().into() }); + q.check().map_err(EngineError::Other)?; + if let Some(it) = pr.item_mut(sid) { + it.label = Label::Iris; + } + st.project_selection = vec![sid]; + Ok(sid) + })?; + Ok(json!({"item": id.0, "clips": rep})) +} + +fn create_multicam(s: &mut Session, p: &Value) -> Result { + let cmd = "clip.createMulticam"; + let method = method_p(p, cmd)?; + let mut clips = sync_items(s, &items_p(s, p), cmd)?; + let has_v = |s: &Session, c: &SyncClip| s.project.item(c.item).is_some_and(|i| i.has_video()); + let has_a = |s: &Session, c: &SyncClip| s.project.item(c.item).is_some_and(|i| i.has_audio()); + // cameras with video first (selection order), then audio-only sources + clips.sort_by_key(|c| !has_v(s, c)); + if !clips.iter().any(|c| has_v(s, c)) { + return Err(bad(cmd, "select at least one clip with video")); + } + let names_mode = str_p(p, "cameraNames").unwrap_or("clip").to_string(); + if names_mode == "metadata" { + // cameras ordered by their Camera Angle metadata when present + let key = |c: &SyncClip| s.project.item(c.item).and_then(|i| i.metadata.get("Camera Angle").cloned()); + let mut v: Vec<(bool, Option, SyncClip)> = clips.iter().map(|c| (!has_v(s, c), key(c), c.clone())).collect(); + v.sort_by_key(|a| (a.0, a.1.is_none(), a.1.clone())); + clips = v.into_iter().map(|x| x.2).collect(); + } + let reference = match item_p(p, "reference") { + Some(r) => clips.iter().position(|c| c.item == r).ok_or_else(|| bad(cmd, "`reference` is not among the items"))?, + None => 0, + }; + let audio = match str_p(p, "audio") { + Some(a) => MulticamAudio::from_name(a).ok_or_else(|| bad(cmd, format!("unknown audio mode `{a}` (camera1, all, switch)")))?, + None => MulticamAudio::Camera1, + }; + let settings = settings_for(s, clips[reference].item); + let rate = settings.frame_rate; + let anchors = sync::anchors(s, &clips, reference, &method, offset_p(p, rate))?; + let starts = sync::placements(&clips, &anchors, rate, false); + let rep = sync::report(&clips, &anchors, &starts, rate); + let name = str_p(p, "name").map(str::to_string).unwrap_or_else(|| format!("{} Multicam", clips[reference].name)); + let move_processed = bool_p(p, "processedBin").unwrap_or(false); + let flags: Vec<(bool, bool)> = clips.iter().map(|c| (has_v(s, c), has_a(s, c))).collect(); + let names: Vec = clips.iter().enumerate().map(|(i, c)| camera_name(&s.project, c.item, &names_mode, i + 1)).collect(); + let nv = flags.iter().filter(|f| f.0).count(); + let na = flags.iter().filter(|f| f.1).count(); + let id = s.edit("Create Multi-Camera Source Sequence", |pr, st| { + let sid = pr.new_sequence(&name, settings.clone(), nv, na.max(1), None); + let mut cameras = Vec::new(); + let (mut vi, mut ai) = (0usize, 0usize); + let mut placed: Vec<(TrackKind, usize, TrackItem)> = Vec::new(); + let (vtracks, atracks): (Vec, Vec) = { + let q = pr.sequence(sid).expect("new sequence"); + (q.video_tracks.iter().map(|t| t.id).collect(), q.audio_tracks.iter().map(|t| t.id).collect()) + }; + for (k, c) in clips.iter().enumerate() { + let (hv, ha) = flags[k]; + let link = (hv && ha).then(|| pr.alloc_id()); + let mut cam = Camera { name: names[k].clone(), video_track: None, audio_tracks: Vec::new(), enabled: true, source: Some(c.item) }; + if hv { + let mut v = placed_item(pr, c.item, TrackKind::Video, starts[k], c.range, &settings).ok_or_else(|| bad(cmd, "bad clip"))?; + v.link = link; + placed.push((TrackKind::Video, vi, v)); + cam.video_track = Some(vtracks[vi]); + vi += 1; + } + if ha { + let mut a = placed_item(pr, c.item, TrackKind::Audio, starts[k], c.range, &settings).ok_or_else(|| bad(cmd, "bad clip"))?; + a.link = link; + placed.push((TrackKind::Audio, ai, a)); + cam.audio_tracks.push(atracks[ai]); + ai += 1; + } + cameras.push(cam); + } + { + let q = pr.sequence_mut(sid).expect("new sequence"); + for (kind, idx, it) in placed { + q.tracks_mut(kind)[idx].items.push(it); + } + for cam in &cameras { + if let Some(vt) = cam.video_track.and_then(|id| q.video_tracks.iter_mut().find(|x| x.id == id)) { + vt.name = cam.name.clone(); + } + for at in &cam.audio_tracks { + if let Some(x) = q.audio_tracks.iter_mut().find(|x| x.id == *at) { + x.name = cam.name.clone(); + } + } + } + q.multicam = Some(MulticamSource { cameras, audio, sync: method.name().into() }); + q.check().map_err(EngineError::Other)?; + } + if let Some(it) = pr.item_mut(sid) { + it.label = Label::Mango; + } + if move_processed { + let bin = processed_bin(pr); + for c in &clips { + pr.root.remove_item(c.item); + if let Some(b) = pr.root.find_bin_mut(bin) { + b.children.push(BinEntry::Item(c.item)); + } + } + } + st.project_selection = vec![sid]; + Ok(sid) + })?; + Ok(json!({"sequence": id.0, "cameras": names, "clips": rep})) +} + +// ---------------------------------------------------------------- multi-camera clips + +/// The topmost enabled multi-camera video clip at `t` (targeted tracks first) in the active +/// sequence: (track, clip). +pub fn multicam_clip_at(s: &Session, t: Tick) -> Option<(TrackId, TrackItem)> { + let q = s.active_sequence()?; + let tg = s.targeting().targeted; + let at = |tr: &Track| tr.item_at(t).filter(|i| i.multicam.is_some_and(|m| m.enabled) && s.project.sequence(i.item).is_some()).cloned(); + let targeted = q.video_tracks.iter().rev().filter(|tr| tg.contains(&tr.id)).find_map(|tr| at(tr).map(|i| (tr.id, i))); + targeted.or_else(|| q.video_tracks.iter().rev().find_map(|tr| at(tr).map(|i| (tr.id, i)))) +} + +/// Resolve `camera` (1-based, shown order) or `angle` (0-based) for the source of `clip`. +fn angle_p(s: &Session, p: &Value, source: ItemId, cmd: &str) -> Result { + let q = s.project.sequence(source).ok_or_else(|| bad(cmd, "not a multi-camera clip"))?; + let cams = q.cameras(); + if let Some(a) = u64_p(p, "angle") { + if (a as usize) < cams.cameras.len() { + return Ok(a as u32); + } + return Err(bad(cmd, format!("angle {a} out of range (0–{})", cams.cameras.len().saturating_sub(1)))); + } + let n = u64_p(p, "camera").ok_or_else(|| bad(cmd, "need `camera` (1-based) or `angle`"))? as usize; + let shown = cams.shown_angles(); + shown.get(n.wrapping_sub(1)).map(|a| *a as u32).ok_or_else(|| bad(cmd, format!("there is no camera {n} ({} shown)", shown.len()))) +} + +/// Linked audio clips of `video` that are multi-camera clips. +fn audio_partners(q: &Sequence, video: &TrackItem) -> Vec { + let Some(l) = video.link else { return Vec::new() }; + q.audio_tracks.iter().flat_map(|t| t.items.iter()).filter(|i| i.link == Some(l) && i.multicam.is_some()).map(|i| i.id).collect() +} + +fn switch_angle(s: &mut Session, p: &Value) -> Result { + let cmd = "multicam.switchAngle"; + let q = s.active_sequence().ok_or(EngineError::NoSequence)?.clone(); + let t = time_p(s, p, "").unwrap_or(s.playhead()); + let video_only = bool_p(p, "videoOnly").unwrap_or(false); + let audio_only = bool_p(p, "audioOnly").unwrap_or(false); + let follows = s.state.multicam_audio_follows_video; + let videos: Vec = if p.get("clips").is_some() || p.get("clip").is_some() { + clips_p(s, p).iter().filter_map(|c| q.find_item(*c).map(|x| x.1.clone())).filter(|i| i.multicam.is_some()).collect() + } else { + multicam_clip_at(s, t).map(|x| vec![x.1]).unwrap_or_default() + }; + if videos.is_empty() { + return Err(EngineError::Other("no multi-camera clip at the playhead".into())); + } + let angle = angle_p(s, p, videos[0].item, cmd)?; + let mut targets: Vec = Vec::new(); + for v in &videos { + let is_audio = q.audio_tracks.iter().any(|t| t.item(v.id).is_some()); + if is_audio { + targets.push(v.id); + continue; + } + if !audio_only { + targets.push(v.id); + } + if audio_only || (follows && !video_only) { + targets.extend(audio_partners(&q, v)); + } + } + let n = s.edit_sequence("Switch Multi-Camera Angle", |seq, _, _| Ok(edit::multicam::set_angle(seq, &targets, angle)))?; + Ok(json!({"angle": angle, "changed": n})) +} + +fn enable(s: &mut Session, p: &Value) -> Result { + let q = s.active_sequence().ok_or(EngineError::NoSequence)?.clone(); + let clips = with_links(s, &clips_p(s, p)); + let nested: Vec = clips.iter().filter_map(|c| q.find_item(*c).map(|x| x.1.clone())).filter(|i| s.project.sequence(i.item).is_some()).collect(); + if nested.is_empty() { + return Err(EngineError::Other("select nested sequence or multi-camera clips".into())); + } + let on = bool_p(p, "enabled").unwrap_or_else(|| nested.iter().any(|i| !i.multicam.is_some_and(|m| m.enabled))); + let project = s.project.clone(); + let ids: Vec = nested.iter().map(|i| i.id).collect(); + let n = s.edit_sequence(if on { "Enable Multi-Camera" } else { "Disable Multi-Camera" }, |seq, _, _| { + let mut n = 0; + for t in seq.all_tracks_mut() { + for it in t.items.iter_mut().filter(|i| ids.contains(&i.id)) { + let first = project.sequence(it.item).and_then(|q| q.cameras().first_video_angle()).unwrap_or(0) as u32; + let sel = it.multicam.get_or_insert(MulticamSel { enabled: false, angle: first }); + if sel.enabled != on { + sel.enabled = on; + n += 1; + } + } + } + Ok(n) + })?; + Ok(json!({"enabled": on, "changed": n})) +} + +fn flatten(s: &mut Session, p: &Value) -> Result { + let seq_id = s.state.active_sequence.ok_or(EngineError::NoSequence)?; + let q = s.active_sequence().ok_or(EngineError::NoSequence)?.clone(); + let clips = with_links(s, &clips_p(s, p)); + let mut targets: Vec<(TrackKind, usize, TrackItem)> = Vec::new(); + for (k, ts) in [(TrackKind::Video, &q.video_tracks), (TrackKind::Audio, &q.audio_tracks)] { + for (ti, t) in ts.iter().enumerate() { + for it in t.items.iter().filter(|i| clips.contains(&i.id) && i.multicam.is_some_and(|m| m.enabled)) { + targets.push((k, ti, it.clone())); + } + } + } + if targets.is_empty() { + return Err(EngineError::Other("select multi-camera clips to flatten".into())); + } + let project = s.project.clone(); + let media = s.media.clone(); + let n = targets.len(); + let ids = s.edit("Flatten Multi-Camera", |pr, st| { + // extra audio tracks needed for sources whose audio spans several tracks + let need_a = targets + .iter() + .filter(|t| t.0 == TrackKind::Audio) + .map(|(_, ti, it)| { + let nq = project.sequence(it.item).map(|nq| audible(nq, it).len()).unwrap_or(1); + ti + nq.max(1) + }) + .max() + .unwrap_or(0); + let have = pr.sequence(seq_id).map(|q| q.audio_tracks.len()).unwrap_or(0); + for k in have..need_a { + let id = TrackId(pr.alloc_id()); + if let Some(q) = pr.sequence_mut(seq_id) { + q.audio_tracks.push(Track::new(id, TrackKind::Audio, format!("Audio {}", k + 1))); + } + } + let new_ids = seq_edit(pr, seq_id, &media, |seq, ctx| { + let mut links = HashMap::new(); + let mut out = Vec::new(); + for (kind, ti, it) in &targets { + let Some(nested) = project.sequence(it.item) else { continue }; + let mut pieces: Vec<(TrackId, TrackItem)> = Vec::new(); + match kind { + TrackKind::Video => { + let angle = it.multicam.map(|m| m.angle as usize).unwrap_or(0); + if let Some(nt) = nested.angle_video_track_index(angle).and_then(|i| nested.video_tracks.get(i)) { + let dest = seq.video_tracks[*ti].id; + pieces.extend(edit::multicam::flatten_items(it, nt, ctx, &mut links)?.into_iter().map(|x| (dest, x))); + } + } + TrackKind::Audio => { + for (k, at) in audible(nested, it).iter().enumerate() { + let Some(nt) = nested.audio_tracks.iter().find(|x| x.id == *at) else { continue }; + let Some(dest) = seq.audio_tracks.get(ti + k).map(|t| t.id) else { continue }; + pieces.extend(edit::multicam::flatten_items(it, nt, ctx, &mut links)?.into_iter().map(|x| (dest, x))); + } + } + } + out.extend(edit::multicam::replace_with(seq, it.id, pieces, ctx)?); + } + Ok(out) + })?; + st.selection = new_ids.clone(); + Ok(new_ids) + })?; + Ok(json!({"flattened": n, "clips": ids.iter().map(|c| c.0).collect::>()})) +} + +/// Audio tracks of `nested` a multi-camera audio clip plays. +fn audible(nested: &Sequence, it: &TrackItem) -> Vec { + let angle = it.multicam_angle(nested); + match &nested.multicam { + Some(mc) => mc.audible_tracks(if mc.audio == MulticamAudio::SwitchAudio { angle } else { None }), + None => nested.audio_tracks.iter().filter(|t| !t.muted).map(|t| t.id).collect(), + } +} + +// ---------------------------------------------------------------- live switching + +/// A live switching pass (Multi-Camera view, playing): every cut is applied at once, and the whole +/// pass is one undo step. +#[derive(Clone, Debug, Default)] +pub struct Recorder { + /// Sequence being recorded (None = no pass running). + pub seq: Option, + pub start: Tick, + /// Tracks the pass switches (fixed by the first cut). + pub tracks: Vec, + pub cuts: Vec, + /// The multi-camera source being switched (fixed by the first cut). + pub source: Option, + /// The project before the pass's first cut. + base: Option>, + key: String, + pass: u64, +} + +impl Recorder { + pub fn active(&self) -> bool { + self.seq.is_some() + } +} + +const RECORD_LABEL: &str = "Record Multi-Camera"; + +fn record_start(s: &mut Session, p: &Value) -> Result { + let t = time_p(s, p, "").unwrap_or(s.playhead()); + let pass = s.mcrec.pass + 1; + s.mcrec = Recorder { seq: s.state.active_sequence, start: t, pass, key: format!("multicam-record-{pass}"), ..Default::default() }; + Ok(json!({"recording": true, "start": t.0})) +} + +/// Re-apply the pass from its base project up to `end`. +fn apply_pass(s: &mut Session, end: Tick) -> Result { + let Some(seq_id) = s.mcrec.seq else { return Ok(0) }; + let merging = s.history.merge_key.as_deref() == Some(s.mcrec.key.as_str()) && s.history.undo.last().is_some_and(|u| u.0 == RECORD_LABEL); + if !merging || s.mcrec.base.is_none() { + s.mcrec.base = Some(s.project.clone()); + } + let base = s.mcrec.base.clone().expect("set above"); + let (tracks, cuts, key) = (s.mcrec.tracks.clone(), s.mcrec.cuts.clone(), s.mcrec.key.clone()); + let Some(source) = s.mcrec.source else { return Ok(0) }; + let media = s.media.clone(); + s.edit_merged(RECORD_LABEL, &key, |pr, _| { + *pr = (*base).clone(); + seq_edit(pr, seq_id, &media, |seq, ctx| Ok(edit::multicam::record(seq, &tracks, source, &cuts, end, ctx))) + }) +} + +fn cut(s: &mut Session, p: &Value) -> Result { + let cmd = "multicam.cut"; + let t = time_p(s, p, "").unwrap_or(s.playhead()); + if !s.mcrec.active() || s.mcrec.seq != s.state.active_sequence { + // stopped: switch the clip at the playhead + return switch_angle(s, p); + } + let (_, clip) = multicam_clip_at(s, t).ok_or_else(|| EngineError::Other("no multi-camera clip at the playhead".into()))?; + let angle = angle_p(s, p, clip.item, cmd)?; + if s.mcrec.tracks.is_empty() { + let q = s.active_sequence().ok_or(EngineError::NoSequence)?; + let mut tracks: Vec = + q.video_tracks.iter().filter(|tr| tr.item_at(t).is_some_and(|i| i.item == clip.item && i.multicam.is_some())).map(|tr| tr.id).collect(); + tracks.truncate(1); + if s.state.multicam_audio_follows_video && !bool_p(p, "videoOnly").unwrap_or(false) { + let partners = audio_partners(q, &clip); + tracks.extend(q.audio_tracks.iter().filter(|tr| tr.items.iter().any(|i| partners.contains(&i.id))).map(|tr| tr.id)); + } + s.mcrec.tracks = tracks; + } + s.mcrec.source = Some(clip.item); + s.mcrec.cuts.push(Cut { time: t, angle }); + // provisional: the new angle runs to the end of this multi-camera clip (its contiguous pieces) + // until the pass stops + let end = s + .active_sequence() + .and_then(|q| q.track(s.mcrec.tracks[0])) + .map(|tr| { + let mut end = clip.end(); + for it in tr.items.iter().filter(|i| i.start >= clip.end()) { + if it.start != end || it.item != clip.item { + break; + } + end = it.end(); + } + end + }) + .unwrap_or(clip.end()); + apply_pass(s, end.max(t))?; + Ok(json!({"angle": angle, "time": t.0, "cuts": s.mcrec.cuts.len()})) +} + +/// Ctrl+N: an edit at the playhead on the multi-camera clip (and, when audio follows video, its +/// linked audio), and camera N from there to the clip's end. +fn cut_to_camera(s: &mut Session, p: &Value) -> Result { + let cmd = "multicam.cutToCamera"; + if s.mcrec.active() && s.mcrec.seq == s.state.active_sequence { + return cut(s, p); + } + let t = time_p(s, p, "").unwrap_or(s.playhead()); + let (_, clip) = multicam_clip_at(s, t).ok_or_else(|| EngineError::Other("no multi-camera clip at the playhead".into()))?; + let angle = angle_p(s, p, clip.item, cmd)?; + let q = s.active_sequence().ok_or(EngineError::NoSequence)?; + let mut clips = vec![clip.id]; + if s.state.multicam_audio_follows_video && !bool_p(p, "videoOnly").unwrap_or(false) { + clips.extend(audio_partners(q, &clip)); + } + let n = s.edit_sequence("Cut to Camera", |seq, ctx, _| { + let new = edit::razor_items(seq, &clips, t, ctx); + // the pieces starting at t (new right halves, or the clips themselves at an edit point) + let right: Vec = + seq.all_tracks().flat_map(|tr| tr.items.iter()).filter(|i| i.start == t && (new.contains(&i.id) || clips.contains(&i.id))).map(|i| i.id).collect(); + Ok(edit::multicam::set_angle(seq, &right, angle)) + })?; + Ok(json!({"angle": angle, "changed": n})) +} + +fn record_stop(s: &mut Session, p: &Value) -> Result { + if !s.mcrec.active() { + return Ok(json!({"recording": false, "cuts": 0})); + } + let t = time_p(s, p, "").unwrap_or(s.playhead()); + let n = s.mcrec.cuts.len(); + let mut points = 0; + if n > 0 { + points = apply_pass(s, t)?; + } + let pass = s.mcrec.pass; + s.mcrec = Recorder { pass, ..Default::default() }; + s.history.merge_key = None; + Ok(json!({"recording": false, "cuts": n, "editPoints": points})) +} + +fn audio_follows(s: &mut Session, p: &Value) -> Result { + let on = bool_p(p, "enabled").unwrap_or(!s.state.multicam_audio_follows_video); + s.state.multicam_audio_follows_video = on; + Ok(json!({"enabled": on})) +} + +fn edit_cameras(s: &mut Session, p: &Value) -> Result { + let cmd = "multicam.editCameras"; + let id = match item_p(p, "sequence") { + Some(i) => i, + None => multicam_clip_at(s, s.playhead()).map(|c| c.1.item).ok_or_else(|| bad(cmd, "need `sequence` (or a multi-camera clip at the playhead)"))?, + }; + let q = s.project.sequence(id).ok_or_else(|| bad(cmd, "no such sequence"))?; + if q.multicam.is_none() { + return Err(bad(cmd, "not a multi-camera source sequence")); + } + let changes: Vec = p.get("cameras").and_then(Value::as_array).cloned().unwrap_or_default(); + let audio = match str_p(p, "audio") { + Some(a) => Some(MulticamAudio::from_name(a).ok_or_else(|| bad(cmd, format!("unknown audio mode `{a}`")))?), + None => None, + }; + s.edit("Edit Cameras", |pr, _| { + let q = pr.sequence_mut(id).ok_or_else(|| bad(cmd, "no such sequence"))?; + let mc = q.multicam.as_mut().ok_or_else(|| bad(cmd, "not a multi-camera source"))?; + for c in &changes { + let a = c.get("angle").and_then(Value::as_u64).ok_or_else(|| bad(cmd, "each camera needs `angle`"))? as usize; + let cam = mc.cameras.get_mut(a).ok_or_else(|| bad(cmd, format!("no angle {a}")))?; + if let Some(n) = c.get("name").and_then(Value::as_str) { + cam.name = n.to_string(); + } + if let Some(e) = c.get("enabled").and_then(Value::as_bool) { + cam.enabled = e; + } + } + if let Some(a) = audio { + mc.audio = a; + } + Ok(()) + })?; + Ok(json!({"sequence": id.0})) +} + +/// The multi-camera clip at the playhead (or `time`), for the Multi-Camera view and agents. +pub fn inspect_at(s: &Session, t: Tick) -> Value { + let rec = json!({"recording": s.mcrec.active(), "cuts": s.mcrec.cuts.iter().map(|c| json!({"time": c.time.0, "angle": c.angle})).collect::>()}); + let Some((track, clip)) = multicam_clip_at(s, t) else { + return json!({"clip": null, "recorder": rec, "audioFollowsVideo": s.state.multicam_audio_follows_video}); + }; + let q = s.project.sequence(clip.item); + let cams = q.map(|q| q.cameras()).unwrap_or_default(); + let audio_angle = s + .active_sequence() + .map(|seq| audio_partners(seq, &clip)) + .and_then(|ps| ps.first().and_then(|c| s.active_sequence().and_then(|seq| seq.find_item(*c)).and_then(|x| x.1.multicam.map(|m| m.angle)))); + json!({ + "clip": clip.id.0, + "track": track.0, + "source": clip.item.0, + "sourceName": s.project.item(clip.item).map(|i| i.name.clone()), + "sourceTime": clip.source_time_at(t).0, + "angle": clip.multicam.map(|m| m.angle), + "audioAngle": audio_angle, + "audio": cams.audio.name(), + "shown": cams.shown_angles(), + "cameras": cams.cameras.iter().enumerate().map(|(i, c)| json!({"angle": i, "name": c.name, "enabled": c.enabled, "video": c.video_track.is_some()})).collect::>(), + "recorder": rec, + "audioFollowsVideo": s.state.multicam_audio_follows_video, + }) +} + +fn inspect(s: &mut Session, p: &Value) -> Result { + let t = time_p(s, p, "").unwrap_or(s.playhead()); + Ok(inspect_at(s, t)) +} diff --git a/crates/engine/src/multicam_tests.rs b/crates/engine/src/multicam_tests.rs new file mode 100644 index 00000000..54e058e8 --- /dev/null +++ b/crates/engine/src/multicam_tests.rs @@ -0,0 +1,677 @@ +//! Synchronisation and multi-camera editing end to end. +//! +//! Fixtures (ffmpeg, `target/fixtures/multicam/`): one "event" (pink noise gated into bursts, so +//! it is not periodic) recorded by three devices that started at different times — two cameras +//! (MJPEG + PCM `.mov` with QuickTime timecode) and an external recorder (`.wav`) — each with its +//! own gain and independent noise. The true offsets are known to the sample. +//! +//! The editing tests use movies made with our own encoders (no ffmpeg). + +use std::path::{Path, PathBuf}; +use std::process::Command; + +use filmcraft_media::DemoScene; +use filmcraft_project::{ItemId, Marker, MarkerId, MarkerKind, MulticamAudio, Sequence, TrackItem}; +use filmcraft_time::{FrameRate, TICKS_PER_SECOND, Tick}; +use serde_json::{Value, json}; + +use crate::Session; +use crate::media_test_util::{make_movie, psnr, tmp_dir}; + +const SR: i64 = 48_000; +/// Event samples at which each recording starts (camera A, camera B, recorder). +const START_A: i64 = 96_000; +const START_B: i64 = 96_000 + 121_234; +const START_REC: i64 = 48_000; +const SAMPLE: Tick = Tick(TICKS_PER_SECOND / SR); + +fn ff(ffmpeg: &Path, args: &[&str]) -> bool { + Command::new(ffmpeg).args(["-v", "error", "-y"]).args(args).status().map(|s| s.success()).unwrap_or(false) +} + +/// The event as heard by one device: starts at event sample `start`, `secs` long, `gain`, white +/// noise of amplitude `noise` (seed `seed`). +fn mic_filter(start: i64, secs: f64, gain: f64, noise: f64, seed: u32) -> String { + let len = (secs * SR as f64) as i64; + format!( + "anoisesrc=d=60:c=pink:r=48000:a=0.35:s=7,volume='if(lt(mod(t*2.7\\,1)\\,0.55)\\,1\\,0.08)':eval=frame,atrim=start_sample={start},asetpts=N/SR/TB,volume={gain}[s];\ + anoisesrc=d=60:c=white:r=48000:a={noise}:s={seed}[n];[s][n]amix=inputs=2:normalize=0:duration=first,atrim=end_sample={len}[a]" + ) +} + +/// (camera A .mov, camera B .mov, recorder .wav), or None without ffmpeg. +fn fixtures() -> Option<(PathBuf, PathBuf, PathBuf)> { + let ffmpeg = filmcraft_testkit::ffmpeg_or_skip("multicam fixtures")?; + let dir = filmcraft_testkit::fixtures_dir("multicam"); + let cam = |name: &str, src: &str, start: i64, secs: f64, gain: f64, noise: f64, seed: u32, tc: &str| { + filmcraft_testkit::fixtures::generate(&dir.join(name), |tmp| { + let t = tmp.to_string_lossy(); + let video = format!("{src}=s=320x180:r=24:d={secs}"); + ff( + &ffmpeg, + &[ + "-f", + "lavfi", + "-i", + &video, + "-filter_complex", + &mic_filter(start, secs, gain, noise, seed), + "-map", + "0:v", + "-map", + "[a]", + "-c:v", + "mjpeg", + "-q:v", + "8", + "-c:a", + "pcm_s16le", + "-ar", + "48000", + "-ac", + "1", + "-timecode", + tc, + "-shortest", + &t, + ], + ) + }) + }; + let a = cam("camA.mov", "testsrc2", START_A, 12.0, 1.0, 0.02, 11, "01:00:00:00")?; + let b = cam("camB.mov", "smptebars", START_B, 10.0, 0.25, 0.05, 12, "01:00:02:12")?; + let rec = filmcraft_testkit::fixtures::generate(&dir.join("recorder.wav"), |tmp| { + let t = tmp.to_string_lossy(); + ff( + &ffmpeg, + &[ + "-f", + "lavfi", + "-i", + "anullsrc=r=48000", + "-filter_complex", + &mic_filter(START_REC, 16.0, 1.5, 0.003, 13), + "-map", + "[a]", + "-c:a", + "pcm_s16le", + "-ac", + "1", + &t, + ], + ) + })?; + Some((a, b, rec)) +} + +fn import(s: &mut Session, files: &[&Path]) -> Vec { + let paths: Vec = files.iter().map(|p| p.to_string_lossy().into_owned()).collect(); + let r = s.execute("file.import", json!({"paths": paths})).unwrap(); + r["items"].as_array().unwrap().iter().map(|v| ItemId(v.as_u64().unwrap())).collect() +} + +fn ids(v: &[ItemId]) -> Value { + json!(v.iter().map(|i| i.0).collect::>()) +} + +/// Run a Project-panel command on `items` (selected first, as in the UI). +fn on_items(s: &mut Session, cmd: &str, items: &[ItemId], mut p: Value) -> crate::Result { + s.execute("project.select", json!({"items": ids(items)})).unwrap(); + p["items"] = ids(items); + s.execute(cmd, p) +} + +/// Timeline time at which event sample `m` plays in clip `ti` (whose media starts at event sample +/// `start`). +fn event_time(ti: &TrackItem, start: i64, m: i64) -> Tick { + let media = Tick::from_units(m - start, SR); + ti.start + (media - ti.source_in) +} + +fn find_clip(q: &Sequence, item: ItemId, video: bool) -> &TrackItem { + let tracks = if video { &q.video_tracks } else { &q.audio_tracks }; + tracks.iter().flat_map(|t| t.items.iter()).find(|i| i.item == item).unwrap_or_else(|| panic!("no clip of {item:?}")) +} + +#[test] +fn audio_sync_multicam_sample_accurate() { + let Some((a, b, rec)) = fixtures() else { return }; + let mut s = Session::default(); + let items = import(&mut s, &[&a, &b, &rec]); + let t0 = std::time::Instant::now(); + let r = + on_items(&mut s, "clip.createMulticam", &items, json!({"method": "audio", "audio": "switch", "name": "Interview MC", "processedBin": true})).unwrap(); + let ms = t0.elapsed().as_secs_f64() * 1000.0; + let seq_id = ItemId(r["sequence"].as_u64().unwrap()); + let q = s.project.sequence(seq_id).unwrap().clone(); + // structure: two cameras with video, an audio-only recorder, switching audio + let mc = q.multicam.as_ref().unwrap(); + assert_eq!(mc.cameras.len(), 3); + assert_eq!(q.video_tracks.len(), 2); + assert_eq!(q.audio_tracks.len(), 3); + assert_eq!(mc.audio, MulticamAudio::SwitchAudio); + assert_eq!(mc.cameras[0].name, "camA.mov"); + assert!(mc.cameras[2].video_track.is_none() && mc.cameras[2].audio_tracks.len() == 1, "the recorder is an audio-only source"); + assert_eq!(q.video_tracks[0].name, "camA.mov"); + assert_eq!(s.project.item(seq_id).unwrap().type_label(), "Multi-Camera Source Sequence"); + // processed clips moved into their bin + let bin = s.project.root.children.iter().find_map(|c| match c { + filmcraft_project::BinEntry::Bin(b) if b.name == "Processed Clips" => Some(b.clone()), + _ => None, + }); + assert_eq!(bin.map(|b| b.children.len()), Some(3)); + // every recording plays event sample m at the same timeline time (±1 sample) + let clips = [(find_clip(&q, items[0], false), START_A), (find_clip(&q, items[1], false), START_B), (find_clip(&q, items[2], false), START_REC)]; + let m = 300_000; + let t_ref = event_time(clips[0].0, clips[0].1, m); + let mut worst = Tick::ZERO; + for (c, start) in &clips[1..] { + let d = (event_time(c, *start, m) - t_ref).abs(); + worst = worst.max(d); + } + // video clips sit on frame boundaries and keep the same alignment as their audio + for (k, item) in items[..2].iter().enumerate() { + let v = find_clip(&q, *item, true); + assert_eq!(v.start, q.settings.frame_rate.snap(v.start), "frame-aligned"); + assert_eq!((v.start, v.source_in), (clips[k].0.start, clips[k].0.source_in)); + } + let report = &r["clips"]; + eprintln!( + "audio sync: worst error {:.2} samples over 3 recordings; B lag {} (truth {}), recorder lag {} (truth {}); {ms:.0} ms", + worst.0 as f64 / SAMPLE.0 as f64, + report[1]["audio"]["lagSamples"], + START_B - START_A, + report[2]["audio"]["lagSamples"], + START_REC - START_A + ); + assert!(worst <= SAMPLE, "worst {} samples", worst.0 as f64 / SAMPLE.0 as f64); + assert!(report[1]["audio"]["reliable"].as_bool().unwrap() && report[2]["audio"]["reliable"].as_bool().unwrap(), "{report}"); + // undo removes the sequence and restores the bins + s.execute("edit.undo", json!({})).unwrap(); + assert!(s.project.sequence(seq_id).is_none()); +} + +#[test] +fn timecode_marker_and_in_point_sync_within_one_frame() { + let Some((a, b, _)) = fixtures() else { return }; + let mut s = Session::default(); + let items = import(&mut s, &[&a, &b]); + let truth = Tick::from_units(START_B - START_A, SR); + let rate = FrameRate::FPS_24; + let offset_of = |s: &Session, r: &Value| -> Tick { + let q = s.project.sequence(ItemId(r["sequence"].as_u64().unwrap())).unwrap(); + let (va, vb) = (find_clip(q, items[0], true), find_clip(q, items[1], true)); + // timeline time of media 0 of each camera; B starts `truth` after A in the event + (vb.start - vb.source_in) - (va.start - va.source_in) + }; + // timecode: A 01:00:00:00, B 01:00:02:12 = 2.5 s later (the true offset is 2.5257 s) + let r = on_items(&mut s, "clip.createMulticam", &items, json!({"method": "timecode"})).unwrap(); + let off = offset_of(&s, &r); + assert_eq!(off, rate.tick_of(60), "timecode offset is exactly 60 frames"); + assert!((off - truth).abs() <= rate.frame_duration(), "within a frame of the truth"); + // ignoring hours gives the same result here + let r = on_items(&mut s, "clip.createMulticam", &items, json!({"method": "timecode", "ignoreHours": true})).unwrap(); + assert_eq!(offset_of(&s, &r), rate.tick_of(60)); + // clip markers: the same event moment marked in both clips (frame accuracy) + let event_frame_a = 72; // 3 s into camera A + let at_b = rate.tick_of(event_frame_a) - truth; + let mut p = (*s.project).clone(); + for (it, t) in [(items[0], rate.tick_of(event_frame_a)), (items[1], rate.snap(at_b))] { + let m = p.item_mut(it).unwrap().as_media_mut().unwrap(); + m.markers.push(Marker { + id: MarkerId(900 + it.0), + start: t, + duration: Tick::ZERO, + name: "clap".into(), + comment: String::new(), + kind: MarkerKind::Comment, + color: filmcraft_project::Label::Rose, + }); + } + s.project = std::sync::Arc::new(p); + let r = on_items(&mut s, "clip.createMulticam", &items, json!({"method": "marker", "marker": "clap"})).unwrap(); + let off = offset_of(&s, &r); + assert!((off - truth).abs() <= rate.frame_duration(), "marker sync {} vs {}", off.0, truth.0); + // In points: A's In at 2.5 s, B's at 0 → B starts 2.5 s after A + let mut p = (*s.project).clone(); + p.item_mut(items[0]).unwrap().as_media_mut().unwrap().mark_in = Some(rate.tick_of(60)); + s.project = std::sync::Arc::new(p); + let r = on_items(&mut s, "clip.createMulticam", &items, json!({"method": "in"})).unwrap(); + assert_eq!(offset_of(&s, &r), rate.tick_of(60)); + // Out points: A ends at 12 s, B at 10 s, both Outs unset → ends aligned: B media 0 at 2 s + let r = on_items(&mut s, "clip.createMulticam", &items, json!({"method": "out"})).unwrap(); + assert_eq!(offset_of(&s, &r), rate.tick_of(48)); + // a missing marker is an error, not a guess + assert!(on_items(&mut s, "clip.createMulticam", &items, json!({"method": "marker", "marker": "nope"})).is_err()); +} + +#[test] +fn merge_clips_by_audio() { + let Some((a, _, rec)) = fixtures() else { return }; + let mut s = Session::default(); + let items = import(&mut s, &[&a, &rec]); + let r = on_items(&mut s, "clip.mergeClips", &items, json!({"method": "audio", "removeVideoAudio": true})).unwrap(); + let id = ItemId(r["item"].as_u64().unwrap()); + let q = s.project.sequence(id).unwrap(); + assert_eq!(s.project.item(id).unwrap().type_label(), "Merged Clip"); + assert_eq!(s.project.item(id).unwrap().name, "camA.mov - Merged"); + assert_eq!((q.video_tracks.len(), q.audio_tracks.len()), (1, 1), "camera audio removed"); + let v = find_clip(q, items[0], true); + let ar = find_clip(q, items[1], false); + assert_eq!(v.link, ar.link, "merged parts are linked"); + // the recorder's event sample m plays with the camera's + let m = 400_000; + let d = (event_time(ar, START_REC, m) - (v.start + (Tick::from_units(m - START_A, SR) - v.source_in))).abs(); + assert!(d <= SAMPLE, "merge offset error {} samples", d.0 as f64 / SAMPLE.0 as f64); + assert!(on_items(&mut s, "clip.mergeClips", &items[..1], json!({})).is_err(), "needs audio clips"); +} + +#[test] +fn timeline_synchronize_by_audio_and_timecode() { + let Some((a, b, rec)) = fixtures() else { return }; + let mut s = Session::default(); + let items = import(&mut s, &[&a, &b, &rec]); + let mut p = (*s.project).clone(); + let rate = FrameRate::FPS_24; + let seq = p.new_sequence("Edit", filmcraft_project::SequenceSettings { width: 320, height: 180, frame_rate: rate, ..Default::default() }, 2, 3, None); + // camera B on V2/A2 at 20 s, recorder on A3 at 1 s, camera A on V1/A1 at 5 s + let place = |p: &mut filmcraft_project::Project, it: ItemId, at: i64, v: Option, a: usize| { + let d = p.item(it).unwrap().duration(); + let range = filmcraft_time::TimeRange::new(Tick::ZERO, d); + let link = p.alloc_id(); + if let Some(vt) = v { + let mut vi = p.make_track_item(it, filmcraft_project::TrackKind::Video, rate.tick_of(at), range, rate).unwrap(); + vi.link = Some(link); + p.sequence_mut(seq).unwrap().video_tracks[vt].items.push(vi); + } + let mut ai = p.make_track_item(it, filmcraft_project::TrackKind::Audio, rate.tick_of(at), range, rate).unwrap(); + ai.link = Some(link); + p.sequence_mut(seq).unwrap().audio_tracks[a].items.push(ai); + }; + place(&mut p, items[0], 120, Some(0), 0); + place(&mut p, items[1], 480, Some(1), 1); + place(&mut p, items[2], 24, None, 2); + s.project = std::sync::Arc::new(p); + s.state.active_sequence = Some(seq); + let all: Vec = s.project.sequence(seq).unwrap().all_tracks().flat_map(|t| t.items.iter().map(|i| i.id.0)).collect(); + s.execute("timeline.select", json!({"clips": all})).unwrap(); + let r = s.execute("clip.synchronize", json!({"method": "audio"})).unwrap(); + assert_eq!(r["moved"], 2, "{r}"); + let q = s.active_sequence().unwrap().clone(); + let ca = find_clip(&q, items[0], false); + assert_eq!(ca.start, rate.tick_of(120), "the reference (V1) stays put"); + let m = 350_000; + for (it, start) in [(items[1], START_B), (items[2], START_REC)] { + let c = find_clip(&q, it, false); + let d = (event_time(c, start, m) - event_time(ca, START_A, m)).abs(); + assert!(d <= SAMPLE, "timeline sync error {} samples", d.0 as f64 / SAMPLE.0 as f64); + assert_eq!(c.start, rate.snap(c.start), "moved clips stay on frames"); + } + let vb = find_clip(&q, items[1], true); + let ab = find_clip(&q, items[1], false); + assert_eq!((vb.start, vb.source_in), (ab.start, ab.source_in), "linked partners moved together"); + s.execute("edit.undo", json!({})).unwrap(); + assert_eq!(find_clip(s.active_sequence().unwrap(), items[1], false).start, rate.tick_of(480)); + // timecode: B lands exactly 60 frames after A + s.execute("timeline.select", json!({"clips": all})).unwrap(); + let r = s.execute("clip.synchronize", json!({"method": "timecode", "track": "V1"})).unwrap(); + assert!(r["moved"].as_u64().unwrap() >= 1); + let q = s.active_sequence().unwrap(); + assert_eq!(find_clip(q, items[1], true).start - find_clip(q, items[0], true).start, rate.tick_of(60)); +} + +// ------------------------------------------------------------------ editing (our own media) + +const W: u32 = 160; +const H: u32 = 90; + +/// Three 4-second cameras (different scenes, own audio), a multi-camera source sequence by In +/// points, and a sequence holding one multi-camera clip on V1/A1. Returns (session, camera items, +/// source sequence, edit sequence). +fn mc_session(audio: &str) -> (Session, Vec, ItemId, ItemId) { + let dir = tmp_dir("mc-edit"); + let files: Vec = ["a.mov", "b.mov", "c.mov"].iter().map(|f| dir.join(f)).collect(); + for (f, sc) in files.iter().zip([DemoScene::OceanSunset, DemoScene::CityNight, DemoScene::Aurora]) { + make_movie(f, sc, W, H, 96); + } + let mut s = Session::default(); + let refs: Vec<&Path> = files.iter().map(|p| p.as_path()).collect(); + let items = import(&mut s, &refs); + let r = on_items(&mut s, "clip.createMulticam", &items, json!({"method": "in", "audio": audio, "cameraNames": "track"})).unwrap(); + let src = ItemId(r["sequence"].as_u64().unwrap()); + let mut p = (*s.project).clone(); + let edit = + p.new_sequence("Cut", filmcraft_project::SequenceSettings { width: W, height: H, frame_rate: FrameRate::FPS_24, ..Default::default() }, 2, 2, None); + s.project = std::sync::Arc::new(p); + s.state.active_sequence = Some(edit); + s.execute("source.open", json!({"item": src.0})).unwrap(); + s.execute("source.overwrite", json!({})).unwrap(); + s.set_playhead(Tick::ZERO); + (s, items, src, edit) +} + +fn v1(s: &Session) -> Vec<(i64, u32)> { + let q = s.active_sequence().unwrap(); + let rate = q.settings.frame_rate; + q.video_tracks[0].items.iter().map(|i| (rate.frame_at(i.start), i.multicam.unwrap().angle)).collect() +} + +fn a1(s: &Session) -> Vec<(i64, u32)> { + let q = s.active_sequence().unwrap(); + let rate = q.settings.frame_rate; + q.audio_tracks[0].items.iter().map(|i| (rate.frame_at(i.start), i.multicam.unwrap().angle)).collect() +} + +fn frame(s: &mut Session, f: i64) -> Vec { + let t = s.sequence_rate().tick_of(f); + s.set_playhead(t); + s.render_program(1.0).unwrap().over_black_rgba8() +} + +/// The frame of camera `item` at its media frame `f`, rendered alone. +fn camera_frame(s: &Session, item: ItemId, f: i64) -> Vec { + let provider = s.media.provider(s.project.clone(), s.services.clone()); + filmcraft_render::render_item(&s.project, item, FrameRate::FPS_24.tick_of(f), 1.0, &provider).unwrap().over_black_rgba8() +} + +#[test] +fn multicam_clip_structure_and_render() { + let (mut s, items, src, _) = mc_session("camera1"); + let q = s.project.sequence(src).unwrap(); + let mc = q.multicam.as_ref().unwrap(); + assert_eq!(mc.cameras.iter().map(|c| c.name.as_str()).collect::>(), ["Camera 1", "Camera 2", "Camera 3"]); + assert_eq!(mc.audio, MulticamAudio::Camera1); + assert_eq!((q.video_tracks.len(), q.audio_tracks.len()), (3, 3)); + // edited in as an enabled multi-camera clip on angle 0 (video and audio) + assert_eq!(v1(&s), [(0, 0)]); + assert_eq!(a1(&s), [(0, 0)]); + let inspect = s.execute("multicam.inspect", json!({"frame": 10})).unwrap(); + assert_eq!(inspect["source"], src.0); + assert_eq!(inspect["shown"], json!([0, 1, 2])); + // the clip shows angle 0 only (not the composite, whose top track is camera 3) + assert!(psnr(&frame(&mut s, 10), &camera_frame(&s, items[0], 10)) > 40.0); + s.execute("multicam.switchAngle", json!({"camera": 3})).unwrap(); + assert_eq!(v1(&s), [(0, 2)]); + assert!(psnr(&frame(&mut s, 10), &camera_frame(&s, items[2], 10)) > 40.0); + // disabling multi-camera shows the plain nest (top track = camera 3 here, so pick camera 2) + s.execute("multicam.switchAngle", json!({"camera": 2})).unwrap(); + let clip = s.active_sequence().unwrap().video_tracks[0].items[0].id; + s.execute("timeline.select", json!({"clips": [clip.0]})).unwrap(); + s.execute("clip.multicamEnable", json!({"enabled": false})).unwrap(); + assert!(psnr(&frame(&mut s, 10), &camera_frame(&s, items[2], 10)) > 40.0, "plain nest: the top track"); + s.execute("clip.multicamEnable", json!({})).unwrap(); + assert!(psnr(&frame(&mut s, 10), &camera_frame(&s, items[1], 10)) > 40.0, "re-enabled, angle remembered"); + // the GPU plan draws the angle's media frame directly (one layer, no CPU pass) and matches + let provider = s.media.provider(s.project.clone(), s.services.clone()); + let edit_id = s.state.active_sequence.unwrap(); + let t10 = FrameRate::FPS_24.tick_of(10); + let plan = filmcraft_render::plan::plan_frame(&s.project, edit_id, t10, filmcraft_render::RenderOptions::default(), &provider); + match &plan { + filmcraft_render::plan::FramePlan::Layers { layers, .. } => { + assert_eq!(layers.len(), 1); + assert!(!matches!(layers[0].frame.data, filmcraft_frame::PixelData::RgbaF32(_)), "a decoded frame, not a CPU render"); + } + _ => panic!("expected layers"), + } + let via_plan = filmcraft_render::plan::execute_cpu(&plan); + s.set_playhead(t10); + let reference = s.render_program(1.0).unwrap(); + assert!(psnr(&via_plan.over_black_rgba8(), &reference.over_black_rgba8()) > 40.0); + // grid of the angles + let provider = s.media.provider(s.project.clone(), s.services.clone()); + let (grid, angles) = filmcraft_render::multicam::render_grid(&s.project, src, Tick::ZERO, 0.25, &provider).unwrap(); + assert_eq!(angles, [0, 1, 2]); + assert_eq!((grid.w, grid.h), (2 * 40, 2 * 23)); +} + +#[test] +fn switch_audio_follows_video() { + let (mut s, _, src, _) = mc_session("switch"); + // audio follows video off (Premiere's default): only the picture switches + s.execute("multicam.switchAngle", json!({"camera": 2, "frame": 5})).unwrap(); + assert_eq!((v1(&s), a1(&s)), (vec![(0, 1)], vec![(0, 0)])); + s.execute("multicam.audioFollowsVideo", json!({"enabled": true})).unwrap(); + s.execute("multicam.switchAngle", json!({"camera": 3})).unwrap(); + assert_eq!((v1(&s), a1(&s)), (vec![(0, 2)], vec![(0, 2)])); + // video only (Ctrl-click) leaves the audio alone; audio only switches just the audio + s.execute("multicam.switchAngle", json!({"camera": 1, "videoOnly": true})).unwrap(); + assert_eq!((v1(&s), a1(&s)), (vec![(0, 0)], vec![(0, 2)])); + s.execute("multicam.switchAngle", json!({"camera": 2, "audioOnly": true})).unwrap(); + assert_eq!((v1(&s), a1(&s)), (vec![(0, 0)], vec![(0, 1)])); + // switching audio plays the selected angle's track of the source + let q = s.project.sequence(src).unwrap(); + let on = q.multicam.as_ref().unwrap().audible_tracks(Some(1)); + assert_eq!(on, q.multicam.as_ref().unwrap().cameras[1].audio_tracks); + let provider = s.media.provider(s.project.clone(), s.services.clone()); + let edit_seq = s.active_sequence().unwrap().clone(); + let mix = filmcraft_render::audio::mix_sequence(&s.project, &edit_seq, 24_000, 4_800, &provider); + let only_b = filmcraft_render::audio::mix_sequence(&s.project, &q.with_angle_audio(Some(1)), 24_000, 4_800, &provider); + let err = mix.channels[0].iter().zip(&only_b.channels[0]).map(|(a, b)| (a - b).abs()).fold(0.0f32, f32::max); + assert!(err < 1e-4, "multi-camera audio = camera 2's track ({err})"); + assert!(mix.channels[0].iter().any(|v| v.abs() > 1e-3), "audible"); + // undo steps back one switch at a time + s.execute("edit.undo", json!({})).unwrap(); + assert_eq!(a1(&s), [(0, 2)]); +} + +/// Live switching from a simulated playback clock: play 0 → 84 at 24 fps, press 2 at frame 24, +/// 3 at frame 48, 3 again at 60 (no new edit), 1 at 72; stop at 84. +#[test] +fn live_switch_recording_from_playback_clock() { + let (mut s, items, _, _) = mc_session("switch"); + s.execute("multicam.audioFollowsVideo", json!({"enabled": true})).unwrap(); + let undo_before = s.history.undo.len(); + let presses: [(i64, u64); 4] = [(24, 2), (48, 3), (60, 3), (72, 1)]; + let rate = s.sequence_rate(); + s.execute("multicam.recordStart", json!({"frame": 0})).unwrap(); + for f in 0..=84i64 { + // the clock: one tick per frame + s.set_playhead(rate.tick_of(f)); + if let Some((_, cam)) = presses.iter().find(|(pf, _)| *pf == f) { + s.execute("multicam.cut", json!({"camera": cam})).unwrap(); + // the cut is live: the program shows the new angle from this frame on + let want = items[*cam as usize - 1]; + assert!(psnr(&frame(&mut s, f + 2), &camera_frame(&s, want, f + 2)) > 40.0, "live cut at {f}"); + s.set_playhead(rate.tick_of(f)); + } + } + let r = s.execute("multicam.recordStop", json!({"frame": 84})).unwrap(); + assert_eq!(r["cuts"], 4); + // 0–24 cam 1, 24–48 cam 2, 48–72 cam 3 (the second press of 3 adds no edit), 72–96 cam 1 + // (healed with the untouched rest of the clip after the stop point) + let expect = vec![(0, 0), (24, 1), (48, 2), (72, 0)]; + assert_eq!(v1(&s), expect); + assert_eq!(a1(&s), expect, "audio followed the video"); + assert_eq!(s.history.undo.len(), undo_before + 1, "one undo step for the whole pass"); + assert_eq!(s.history.undo.last().unwrap().0, "Record Multi-Camera"); + // pieces are continuous in source time + let q = s.active_sequence().unwrap(); + for w in q.video_tracks[0].items.windows(2) { + assert_eq!(w[0].source_out(), w[1].source_in); + assert!(w[0].link.is_some()); + } + // a second pass changes only its own range and is its own undo step + s.execute("multicam.recordStart", json!({"frame": 30})).unwrap(); + s.execute("multicam.cut", json!({"camera": 3, "frame": 30})).unwrap(); + s.execute("multicam.recordStop", json!({"frame": 40})).unwrap(); + assert_eq!(v1(&s), vec![(0, 0), (24, 1), (30, 2), (40, 1), (48, 2), (72, 0)]); + s.execute("edit.undo", json!({})).unwrap(); + assert_eq!(v1(&s), expect); + s.execute("edit.undo", json!({})).unwrap(); + assert_eq!(v1(&s), vec![(0, 0)]); + // stopped: a key press switches the clip at the playhead instead of cutting + s.execute("multicam.cut", json!({"camera": 2, "frame": 10})).unwrap(); + assert_eq!(v1(&s), vec![(0, 1)]); +} + +#[test] +fn flatten_replaces_with_camera_clips() { + let (mut s, items, _, _) = mc_session("switch"); + s.execute("multicam.audioFollowsVideo", json!({"enabled": true})).unwrap(); + s.execute("multicam.recordStart", json!({"frame": 0})).unwrap(); + s.execute("multicam.cut", json!({"camera": 2, "frame": 30})).unwrap(); + s.execute("multicam.cut", json!({"camera": 3, "frame": 60})).unwrap(); + s.execute("multicam.recordStop", json!({"frame": 96})).unwrap(); + let before: Vec> = [5, 35, 70].iter().map(|f| frame(&mut s, *f)).collect(); + let all: Vec = s.active_sequence().unwrap().all_tracks().flat_map(|t| t.items.iter().map(|i| i.id.0)).collect(); + s.execute("timeline.select", json!({"clips": all})).unwrap(); + let r = s.execute("clip.multicamFlatten", json!({})).unwrap(); + assert_eq!(r["flattened"], 6); + let q = s.active_sequence().unwrap().clone(); + let v: Vec<(ItemId, i64, i64)> = q.video_tracks[0].items.iter().map(|i| (i.item, i.start.0, i.source_in.0)).collect(); + let rate = q.settings.frame_rate; + assert_eq!(v, vec![(items[0], 0, 0), (items[1], rate.tick_of(30).0, rate.tick_of(30).0), (items[2], rate.tick_of(60).0, rate.tick_of(60).0)]); + assert!(q.all_tracks().flat_map(|t| t.items.iter()).all(|i| i.multicam.is_none())); + let a: Vec = q.audio_tracks[0].items.iter().map(|i| i.item).collect(); + assert_eq!(a, vec![items[0], items[1], items[2]], "audio flattened to the switched cameras"); + for (i, f) in [5, 35, 70].iter().enumerate() { + assert!(psnr(&before[i], &frame(&mut s, *f)) > 45.0, "flattened frame {f} looks the same"); + } + // linked camera video/audio stay linked + assert_eq!(q.video_tracks[0].items[1].link, q.audio_tracks[0].items[1].link); + s.execute("edit.undo", json!({})).unwrap(); + assert_eq!(v1(&s).len(), 3); +} + +#[test] +fn edit_cameras_and_save_load() { + let (mut s, _, src, _) = mc_session("all"); + s.execute("multicam.editCameras", json!({"sequence": src.0, "cameras": [{"angle": 1, "name": "Wide", "enabled": false}], "audio": "switch"})).unwrap(); + let mc = s.project.sequence(src).unwrap().multicam.clone().unwrap(); + assert_eq!((mc.cameras[1].name.as_str(), mc.cameras[1].enabled, mc.audio), ("Wide", false, MulticamAudio::SwitchAudio)); + assert_eq!(mc.shown_angles(), [0, 2]); + // keys pick from the shown cameras: camera 2 is angle 2 now + s.execute("multicam.switchAngle", json!({"camera": 2})).unwrap(); + assert_eq!(v1(&s), [(0, 2)]); + assert!(s.execute("multicam.switchAngle", json!({"camera": 3})).is_err()); + let dir = tmp_dir("mc-save"); + let path = dir.join("mc.fcproj").to_string_lossy().into_owned(); + s.execute("file.save", json!({"path": path})).unwrap(); + let bytes = std::fs::read(&path).unwrap(); + let doc: Value = serde_json::from_slice(&bytes).unwrap(); + assert_eq!(doc["schema_version"], filmcraft_format::SCHEMA_VERSION); + let mut t = Session::default(); + t.execute("file.open", json!({"path": path})).unwrap(); + assert_eq!(*t.project, *s.project, "multi-camera data survives save/load"); + assert_eq!(t.project.sequence(src).unwrap().multicam.as_ref().unwrap().cameras[1].name, "Wide"); +} + +#[test] +fn any_nest_can_be_multicam() { + // a plain nested sequence with two video tracks: angles are its tracks + let (mut s, items, _, _) = mc_session("camera1"); + let mut p = (*s.project).clone(); + let rate = FrameRate::FPS_24; + let nest = p.new_sequence("Nest", filmcraft_project::SequenceSettings { width: W, height: H, frame_rate: rate, ..Default::default() }, 2, 0, None); + for (k, it) in items[..2].iter().enumerate() { + let mut v = p + .make_track_item(*it, filmcraft_project::TrackKind::Video, Tick::ZERO, filmcraft_time::TimeRange::new(Tick::ZERO, rate.tick_of(48)), rate) + .unwrap(); + for e in &mut v.effects { + filmcraft_project::resolve_auto_points(e, (W, H), (W, H)); + } + p.sequence_mut(nest).unwrap().video_tracks[k].items.push(v); + } + let edit = s.state.active_sequence.unwrap(); + let mut v = + p.make_track_item(nest, filmcraft_project::TrackKind::Video, Tick::ZERO, filmcraft_time::TimeRange::new(Tick::ZERO, rate.tick_of(48)), rate).unwrap(); + assert!(v.multicam.is_none(), "plain nests start as plain nests"); + for e in &mut v.effects { + filmcraft_project::resolve_auto_points(e, (W, H), (W, H)); + } + let clip = v.id; + let q = p.sequence_mut(edit).unwrap(); + q.video_tracks[1].items.push(v); + s.project = std::sync::Arc::new(p); + assert!(psnr(&frame(&mut s, 3), &camera_frame(&s, items[1], 3)) > 40.0, "nest: top track"); + s.execute("timeline.select", json!({"clips": [clip.0]})).unwrap(); + s.execute("clip.multicamEnable", json!({})).unwrap(); + assert!(psnr(&frame(&mut s, 3), &camera_frame(&s, items[0], 3)) > 40.0, "angle 0 = track V1 of the nest"); + s.execute("multicam.switchAngle", json!({"clips": [clip.0], "angle": 1})).unwrap(); + assert!(psnr(&frame(&mut s, 3), &camera_frame(&s, items[1], 3)) > 40.0); +} + +#[test] +fn camera_keys_cut_and_select() { + let (mut s, _, _, _) = mc_session("camera1"); + // Ctrl+2 at frame 30: an edit there, camera 2 after it + s.set_playhead(FrameRate::FPS_24.tick_of(30)); + s.execute("multicam.cutToCamera2", json!({})).unwrap(); + assert_eq!(v1(&s), [(0, 0), (30, 1)]); + // Ctrl+3 at the existing edit: no new edit, the right piece switches + s.execute("multicam.cutToCamera3", json!({})).unwrap(); + assert_eq!(v1(&s), [(0, 0), (30, 2)]); + // 2 while stopped switches the clip under the playhead + s.set_playhead(FrameRate::FPS_24.tick_of(10)); + s.execute("multicam.selectCamera2", json!({})).unwrap(); + assert_eq!(v1(&s), [(0, 1), (30, 2)]); + // the keys are bound by default + let keys: Vec<(String, String)> = + s.shortcuts.bindings.iter().filter(|b| b.command.starts_with("multicam.")).map(|b| (b.command.clone(), b.keys.clone())).collect(); + assert!(keys.contains(&("multicam.selectCamera1".into(), "1".into())), "{keys:?}"); + assert!(keys.contains(&("multicam.cutToCamera9".into(), "Ctrl+9".into())), "{keys:?}"); +} + +/// Process CPU time in seconds (user + system), for the perf test. +fn process_cpu() -> f64 { + let out = Command::new("ps").args(["-o", "cputime=", "-p", &std::process::id().to_string()]).output().ok(); + let txt = out.map(|o| String::from_utf8_lossy(&o.stdout).trim().to_string()).unwrap_or_default(); + // [[dd-]hh:]mm:ss(.ss) + txt.split(':').fold(0.0, |acc, p| acc * 60.0 + p.trim().parse::().unwrap_or(0.0)) +} + +/// Multi-Camera view cost: a 2×2 grid of four 1080p H.264 angles at ¼ scale per cell, played +/// sequentially, against the program (one angle at ½ scale). Run with +/// `cargo test --release -p filmcraft-engine --lib multicam_tests::perf -- --ignored --nocapture`. +#[test] +#[ignore] +fn perf_grid_of_four_1080p_angles() { + let Some(ffmpeg) = filmcraft_testkit::ffmpeg_or_skip("multicam perf") else { return }; + let dir = filmcraft_testkit::fixtures_dir("multicam"); + let srcs = ["testsrc2", "smptehdbars", "mandelbrot", "rgbtestsrc"]; + let files: Vec = srcs + .iter() + .map(|src| { + filmcraft_testkit::fixtures::generate(&dir.join(format!("perf-{src}.mp4")), |tmp| { + let v = format!("{src}=s=1920x1080:r=24"); + ff( + &ffmpeg, + &["-f", "lavfi", "-i", &v, "-t", "4", "-c:v", "libx264", "-preset", "veryfast", "-g", "48", "-pix_fmt", "yuv420p", &tmp.to_string_lossy()], + ) + }) + .expect("fixture") + }) + .collect(); + // a fresh session (empty decoder caches) per measurement + let setup = || { + let mut s = Session::default(); + let refs: Vec<&Path> = files.iter().map(|p| p.as_path()).collect(); + let items = import(&mut s, &refs); + let r = on_items(&mut s, "clip.createMulticam", &items, json!({"method": "in"})).unwrap(); + (s, ItemId(r["sequence"].as_u64().unwrap())) + }; + let rate = FrameRate::FPS_24; + let frames = 72; + let (s, src) = setup(); + let provider = s.media.provider(s.project.clone(), s.services.clone()); + let q = s.project.sequence(src).unwrap().clone(); + let t0 = std::time::Instant::now(); + for f in 0..frames { + let opts = filmcraft_render::RenderOptions { scale: 0.5, ..Default::default() }; + let _ = filmcraft_render::multicam::render_angle(&s.project, &q, 0, rate.tick_of(f), opts, &provider); + } + let one_ms = t0.elapsed().as_secs_f64() * 1000.0 / frames as f64; + let (s, src) = setup(); + let provider = s.media.provider(s.project.clone(), s.services.clone()); + let cpu0 = crate::multicam_tests::process_cpu(); + let t0 = std::time::Instant::now(); + for f in 0..frames { + let (g, _) = filmcraft_render::multicam::render_grid(&s.project, src, rate.tick_of(f), 0.25, &provider).unwrap(); + assert_eq!((g.w, g.h), (960, 540)); + } + let grid_ms = t0.elapsed().as_secs_f64() * 1000.0 / frames as f64; + let grid_cpu = (crate::multicam_tests::process_cpu() - cpu0) * 1000.0 / frames as f64; + eprintln!( + "Multi-Camera grid (4 × 1080p H.264, ¼-scale cells): {grid_ms:.1} ms/frame wall, {grid_cpu:.1} ms/frame CPU; one angle at ½ scale: {one_ms:.1} ms/frame wall (24 fps budget 41.7 ms)" + ); +} diff --git a/crates/engine/src/shortcut_presets.rs b/crates/engine/src/shortcut_presets.rs index 4f4d36c5..fa4d1fdc 100644 --- a/crates/engine/src/shortcut_presets.rs +++ b/crates/engine/src/shortcut_presets.rs @@ -35,6 +35,27 @@ pub const PREMIERE: &[Entry] = &[ ("tool.hand", "H", ""), ("tool.zoom", "Z", ""), ("tool.type", "T", ""), + // Multi-camera + ("multicam.selectCamera1", "1", ""), + ("multicam.selectCamera2", "2", ""), + ("multicam.selectCamera3", "3", ""), + ("multicam.selectCamera4", "4", ""), + ("multicam.selectCamera5", "5", ""), + ("multicam.selectCamera6", "6", ""), + ("multicam.selectCamera7", "7", ""), + ("multicam.selectCamera8", "8", ""), + ("multicam.selectCamera9", "9", ""), + ("multicam.cutToCamera1", "Ctrl+1", ""), + ("multicam.cutToCamera2", "Ctrl+2", ""), + ("multicam.cutToCamera3", "Ctrl+3", ""), + ("multicam.cutToCamera4", "Ctrl+4", ""), + ("multicam.cutToCamera5", "Ctrl+5", ""), + ("multicam.cutToCamera6", "Ctrl+6", ""), + ("multicam.cutToCamera7", "Ctrl+7", ""), + ("multicam.cutToCamera8", "Ctrl+8", ""), + ("multicam.cutToCamera9", "Ctrl+9", ""), + ("multicam.recordToggle", "0", ""), + ("multicam.toggleView", "Shift+0", ""), // Application ("app.preferences.autoSave", "Cmd+,", ""), ("app.keyboardShortcuts", "Cmd+Alt+K", ""), diff --git a/crates/engine/src/sync.rs b/crates/engine/src/sync.rs new file mode 100644 index 00000000..b6b7ac10 --- /dev/null +++ b/crates/engine/src/sync.rs @@ -0,0 +1,234 @@ +//! Synchronising clips: Clip ▸ Synchronize (timeline clips), Merge Clips and Create +//! Multi-Camera Source Sequence (project items). +//! +//! Every method reduces a clip to an **anchor**: the media time of the clip that lines up with +//! one common instant (common time = media time − anchor). +//! +//! | method | anchor | +//! |---|---| +//! | `in` | the In point (project items: the source In mark, else the start; timeline clips: their source In) | +//! | `out` | the Out point (exclusive end) | +//! | `timecode` | −(start timecode), optionally ignoring hours | +//! | `marker` | the first clip marker (or the first one named `marker`) | +//! | `audio` | from the waveform cross-correlation against the reference clip ([`filmcraft_audio_dsp::sync`]) | +//! +//! `offset` (frames) shifts every clip but the reference by that much (a known delay to correct). + +use filmcraft_audio_dsp::sync::{SyncOptions, SyncResult, find_offset, mixdown}; +use filmcraft_project::{ItemId, ItemKind, Project, TrackItem}; +use filmcraft_time::{FrameRate, TICKS_PER_SECOND, Tick, TimeRange}; +use serde_json::{Value, json}; + +use crate::{EngineError, Result, Session}; + +/// Sample rate audio is analysed at. +pub const ANALYSIS_RATE: u32 = 48_000; +/// Longest stretch of a clip analysed for audio sync (from its start). +pub const MAX_ANALYSIS: Tick = Tick(30 * 60 * TICKS_PER_SECOND); + +#[derive(Clone, Debug, PartialEq)] +pub enum Method { + In, + Out, + Timecode { ignore_hours: bool }, + Marker { name: Option }, + Audio, +} + +impl Method { + pub fn from_params(p: &Value) -> std::result::Result { + let m = p.get("method").and_then(Value::as_str).unwrap_or("in").to_ascii_lowercase(); + Ok(match m.as_str() { + "in" | "inpoint" | "inpoints" | "start" | "clipstart" => Method::In, + "out" | "outpoint" | "outpoints" | "end" | "clipend" => Method::Out, + "timecode" | "tc" => Method::Timecode { ignore_hours: p.get("ignoreHours").and_then(Value::as_bool).unwrap_or(false) }, + "marker" | "clipmarker" | "markers" => Method::Marker { name: p.get("marker").and_then(Value::as_str).map(str::to_string) }, + "audio" | "sound" | "waveform" => Method::Audio, + o => return Err(format!("unknown sync method `{o}` (in, out, timecode, marker, audio)")), + }) + } + pub fn name(&self) -> &'static str { + match self { + Method::In => "in", + Method::Out => "out", + Method::Timecode { .. } => "timecode", + Method::Marker { .. } => "marker", + Method::Audio => "audio", + } + } +} + +/// What syncing needs to know about one clip. +#[derive(Clone, Debug)] +pub struct SyncClip { + pub item: ItemId, + pub name: String, + /// Media range the clip uses. + pub range: TimeRange, + /// In / Out points (media time, exclusive Out). + pub in_point: Tick, + pub out_point: Tick, + /// Start timecode in ticks (0 when the media has none). + pub timecode: Tick, + /// Clip markers: (media time, name). + pub markers: Vec<(Tick, String)>, +} + +/// The media item behind an item (subclips resolve to their parent) and its subclip range. +fn media_of(p: &Project, item: ItemId) -> Option<(ItemId, &filmcraft_project::MediaClip, Option)> { + let it = p.item(item)?; + match &it.kind { + ItemKind::Media(m) => Some((item, m, None)), + ItemKind::Subclip { parent, range } => media_of(p, *parent).map(|(i, m, _)| (i, m, Some(*range))), + _ => None, + } +} + +fn start_timecode(m: &filmcraft_project::MediaClip) -> Tick { + let rate = m.frame_rate(); + m.info.start_timecode.map(|f| rate.tick_of(f)).unwrap_or(Tick::ZERO) +} + +/// A project item as a sync clip (the whole media, or a subclip's range; In/Out marks). +pub fn project_clip(p: &Project, item: ItemId) -> Option { + let it = p.item(item)?; + let (_, m, sub) = media_of(p, item)?; + let range = sub.unwrap_or(TimeRange::new(Tick::ZERO, m.duration())); + let rate = m.frame_rate(); + let in_point = m.mark_in.filter(|_| sub.is_none()).unwrap_or(range.start); + let out_point = m.mark_out.filter(|_| sub.is_none()).map(|o| o + rate.frame_duration()).unwrap_or(range.end()); + Some(SyncClip { + item, + name: it.name.clone(), + range, + in_point, + out_point, + timecode: start_timecode(m), + markers: m.markers.iter().map(|k| (k.start, k.name.clone())).collect(), + }) +} + +/// A timeline clip as a sync clip (its source range; markers of the clip and of its media). +pub fn timeline_clip(p: &Project, ti: &TrackItem) -> SyncClip { + let media = media_of(p, ti.item); + let range = TimeRange::from_bounds(ti.source_in, ti.source_out()); + let mut markers: Vec<(Tick, String)> = ti.markers.iter().map(|k| (k.start, k.name.clone())).collect(); + if let Some((_, m, _)) = media { + markers.extend(m.markers.iter().map(|k| (k.start, k.name.clone()))); + } + SyncClip { + item: ti.item, + name: ti.name.clone(), + range, + in_point: range.start, + out_point: range.end(), + timecode: media.map(|(_, m, _)| start_timecode(m)).unwrap_or(Tick::ZERO), + markers, + } +} + +/// Mono audio of a clip's media range at [`ANALYSIS_RATE`] (None: no audio). +pub fn clip_audio(s: &Session, c: &SyncClip) -> Option> { + let src = s.source(c.item)?; + if !src.info().has_audio() { + return None; + } + let sr = ANALYSIS_RATE as i64; + let start = c.range.start.to_units_floor(sr); + let len = c.range.duration.min(MAX_ANALYSIS).to_units_floor(sr).max(0) as usize; + let buf = src.audio(start, len, ANALYSIS_RATE).ok()?; + let chans: Vec<&[f32]> = buf.channels.iter().map(|c| c.as_slice()).collect(); + Some(mixdown(&chans)) +} + +/// The outcome for one clip: its anchor, and for audio the correlation figures. +#[derive(Clone, Debug)] +pub struct Anchor { + pub anchor: Tick, + pub audio: Option, +} + +/// Compute the anchors of `clips` (index `reference` is the reference for audio and `offset`). +/// `offset` (ticks) moves every other clip later by that much. +pub fn anchors(s: &Session, clips: &[SyncClip], reference: usize, method: &Method, offset: Tick) -> Result> { + if clips.is_empty() { + return Err(EngineError::Other("nothing to synchronize".into())); + } + let hour = Tick(3600 * TICKS_PER_SECOND); + let mut out: Vec = Vec::with_capacity(clips.len()); + let ref_audio = if *method == Method::Audio { + Some(clip_audio(s, &clips[reference]).ok_or_else(|| EngineError::Other(format!("{} has no audio to synchronize by", clips[reference].name)))?) + } else { + None + }; + for (i, c) in clips.iter().enumerate() { + let a = match method { + Method::In => Anchor { anchor: c.in_point, audio: None }, + Method::Out => Anchor { anchor: c.out_point, audio: None }, + Method::Timecode { ignore_hours } => { + let tc = if *ignore_hours { Tick(c.timecode.0.rem_euclid(hour.0)) } else { c.timecode }; + Anchor { anchor: Tick::ZERO - tc, audio: None } + } + Method::Marker { name } => { + let m = + c.markers.iter().filter(|(_, n)| name.as_deref().is_none_or(|w| n.eq_ignore_ascii_case(w))).map(|(t, _)| *t).min().ok_or_else(|| { + EngineError::Other(format!("{} has no clip marker{}", c.name, name.as_ref().map(|n| format!(" named “{n}”")).unwrap_or_default())) + })?; + Anchor { anchor: m, audio: None } + } + Method::Audio => { + if i == reference { + Anchor { anchor: c.range.start, audio: None } + } else { + let a = ref_audio.as_deref().unwrap_or_default(); + let b = clip_audio(s, c).ok_or_else(|| EngineError::Other(format!("{} has no audio to synchronize by", c.name)))?; + let r = find_offset(a, &b, ANALYSIS_RATE, &SyncOptions::default()) + .ok_or_else(|| EngineError::Other(format!("{} and {} have no common audio", clips[reference].name, c.name)))?; + // b[n] (media r0_i + n) lines up with a[n + L] (media r0_ref + n + L); with the + // reference anchored at r0_ref, equal common times give anchor_i = r0_i − L + let lag = Tick::from_units(r.lag, ANALYSIS_RATE as i64); + Anchor { anchor: c.range.start - lag, audio: Some(r) } + } + } + }; + out.push(a); + } + if offset != Tick::ZERO { + for (i, a) in out.iter_mut().enumerate() { + if i != reference { + // later on the common timeline = smaller anchor + a.anchor -= offset; + } + } + } + Ok(out) +} + +/// Where each clip's range starts on a common timeline that begins at 0 (the earliest clip), at +/// the given frame rate (snapped to frames when `snap`). +pub fn placements(clips: &[SyncClip], anchors: &[Anchor], rate: FrameRate, snap: bool) -> Vec { + let raw: Vec = clips.iter().zip(anchors).map(|(c, a)| c.range.start - a.anchor).collect(); + let min = raw.iter().copied().min().unwrap_or(Tick::ZERO); + raw.into_iter().map(|t| if snap { rate.snap_nearest(t - min) } else { t - min }).collect() +} + +/// JSON report of one sync for agents. +pub fn report(clips: &[SyncClip], anchors: &[Anchor], starts: &[Tick], rate: FrameRate) -> Value { + Value::Array( + clips + .iter() + .zip(anchors) + .zip(starts) + .map(|((c, a), st)| { + json!({ + "item": c.item.0, + "name": c.name, + "start": st.0, + "startFrame": rate.frame_at(*st), + "anchor": a.anchor.0, + "audio": a.audio.map(|r| json!({"lagSamples": r.lag, "lagFine": r.lag_fine, "confidence": r.confidence, "distinct": r.distinct, "reliable": r.reliable(), "inverted": r.inverted})), + }) + }) + .collect(), + ) +} diff --git a/crates/format/src/lib.rs b/crates/format/src/lib.rs index dc18729f..565b73d2 100644 --- a/crates/format/src/lib.rs +++ b/crates/format/src/lib.rs @@ -27,6 +27,9 @@ //! - **v5** (M8.8): colour management (sequence working space, Interpret Footage colour space, //! project LUT library). No-op step, for the same reason. //! - **v6** (M11.7/M11.9): media identity fingerprints and ingest settings. No-op step. +//! - **v7** (M5.5): effect masks (keyframable Bézier mask paths on effect instances). No-op step. +//! - **v8** (M12): multi-camera source sequences, multi-camera clips and merged clips. No-op step; +//! older builds would drop the camera data when saving. pub mod atomic; pub mod autosave; @@ -45,7 +48,7 @@ pub type Migration = fn(Value) -> Result; /// `MIGRATIONS[i]` upgrades schema `i + 1` to `i + 2`. Append one function per schema bump; never /// edit a shipped one. -pub const MIGRATIONS: &[Migration] = &[v1_to_v2, v2_to_v3, v3_to_v4, v4_to_v5, v5_to_v6, v6_to_v7]; +pub const MIGRATIONS: &[Migration] = &[v1_to_v2, v2_to_v3, v3_to_v4, v4_to_v5, v5_to_v6, v6_to_v7, v7_to_v8]; /// The schema version this build writes (and the newest it reads). pub const SCHEMA_VERSION: u32 = MIGRATIONS.len() as u32 + 1; @@ -244,13 +247,19 @@ fn v6_to_v7(doc: Value) -> Result { Ok(doc) } +/// v7 → v8: multi-camera source sequences / clips and merged clips added; existing data needs no +/// change. +fn v7_to_v8(doc: Value) -> Result { + Ok(doc) +} + #[cfg(test)] mod tests { use super::*; #[test] fn schema_version_matches_table() { - assert_eq!(SCHEMA_VERSION, 7); + assert_eq!(SCHEMA_VERSION, 8); } #[test] diff --git a/crates/interchange/src/common.rs b/crates/interchange/src/common.rs index 5231ea4a..f7332252 100644 --- a/crates/interchange/src/common.rs +++ b/crates/interchange/src/common.rs @@ -425,6 +425,7 @@ impl Builder { frame_hold: None, scale_to_frame: false, essential: None, + multicam: None, } } @@ -483,6 +484,8 @@ pub(crate) fn empty_sequence(settings: SequenceSettings) -> Sequence { master_mixer: Default::default(), submix_tracks: Vec::new(), caption_tracks: Vec::new(), + multicam: None, + merged: None, } } diff --git a/crates/project/src/essential.rs b/crates/project/src/essential.rs index d3a3d821..9fd2bb1e 100644 --- a/crates/project/src/essential.rs +++ b/crates/project/src/essential.rs @@ -778,6 +778,7 @@ mod tests { frame_hold: None, scale_to_frame: false, essential: None, + multicam: None, } } diff --git a/crates/project/src/lib.rs b/crates/project/src/lib.rs index 08945953..5dc1476f 100644 --- a/crates/project/src/lib.rs +++ b/crates/project/src/lib.rs @@ -14,6 +14,7 @@ pub mod graphic; pub mod keyframe; pub mod mask; pub mod mixer; +pub mod multicam; use std::collections::BTreeMap; @@ -28,6 +29,7 @@ pub use essential::{AudioType, EssentialSound}; pub use keyframe::{Interpolation, Keyframe, Param, ParamValue}; pub use mask::{Mask, MaskMode, MaskPath, MaskVertex, TrackMethod}; pub use mixer::{AutomationMode, InputMap, MixerStrip, TrackSend}; +pub use multicam::{Camera, MergedClip, MulticamAudio, MulticamSel, MulticamSource}; macro_rules! id_type { ($name:ident) => { @@ -320,6 +322,8 @@ impl ProjectItem { filmcraft_media::MediaKind::ImageSequence => "Image Sequence", filmcraft_media::MediaKind::Synthetic => "Synthetic", }, + ItemKind::Sequence(s) if s.multicam.is_some() => "Multi-Camera Source Sequence", + ItemKind::Sequence(s) if s.merged.is_some() => "Merged Clip", ItemKind::Sequence(_) => "Sequence", ItemKind::Subclip { .. } => "Subclip", ItemKind::AdjustmentLayer { .. } => "Adjustment Layer", @@ -470,6 +474,9 @@ pub struct TrackItem { /// Essential Sound audio type and settings (audio clips). #[serde(default, skip_serializing_if = "Option::is_none")] pub essential: Option, + /// Multi-camera clip (a nested multi-camera source sequence): enabled flag and angle. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub multicam: Option, } impl TrackItem { @@ -727,6 +734,12 @@ pub struct Sequence { /// Caption tracks (drawn above the video tracks; first = top). Older files have none (serde default). #[serde(default)] pub caption_tracks: Vec, + /// Set on multi-camera source sequences (cameras, audio mode). + #[serde(default, skip_serializing_if = "Option::is_none")] + pub multicam: Option, + /// Set on merged clips (the merged video and audio items). + #[serde(default, skip_serializing_if = "Option::is_none")] + pub merged: Option, } impl Sequence { @@ -961,6 +974,8 @@ impl Project { master_mixer: MixerStrip::default(), submix_tracks: Vec::new(), caption_tracks: Vec::new(), + multicam: None, + merged: None, }; for i in 0..v { let id = TrackId(self.alloc_id()); @@ -1025,6 +1040,9 @@ impl Project { TrackKind::Video => effect::intrinsic_video(), TrackKind::Audio => effect::intrinsic_audio(), }; + // a multi-camera source sequence edits in as a multi-camera clip showing its first angle + let multicam = + it.as_sequence().and_then(|q| q.multicam.as_ref()).map(|m| MulticamSel { enabled: true, angle: m.first_video_angle().unwrap_or(0) as u32 }); let id = ClipId(self.alloc_id()); let dur = seq_rate.snap_nearest(source.duration).max(seq_rate.frame_duration()); Some(TrackItem { @@ -1046,6 +1064,7 @@ impl Project { frame_hold: None, scale_to_frame: false, essential: None, + multicam, }) } diff --git a/crates/project/src/multicam.rs b/crates/project/src/multicam.rs new file mode 100644 index 00000000..8b4e4566 --- /dev/null +++ b/crates/project/src/multicam.rs @@ -0,0 +1,201 @@ +//! Multi-camera source sequences and merged clips. +//! +//! A **multi-camera source sequence** is an ordinary [`Sequence`](crate::Sequence) whose +//! [`Sequence::multicam`](crate::Sequence::multicam) describes its cameras: each camera (angle) is +//! one video track plus its audio tracks, with the clips already synchronised. Edited into another +//! sequence it is a nested clip; with [`TrackItem::multicam`](crate::TrackItem::multicam) enabled +//! the clip shows only the selected angle's video track (video clips) or plays the selected +//! angle's audio (audio clips, when the source switches audio), instead of the composite. +//! +//! A **merged clip** is a sequence too ([`Sequence::merged`](crate::Sequence::merged)): one video +//! clip and up to 16 separately recorded audio clips, synchronised, used like a single clip. + +use serde::{Deserialize, Serialize}; + +use crate::{ItemId, Sequence, TrackId}; + +/// How a multi-camera source sequence plays audio (Create Multi-Camera Source Sequence ▸ Audio). +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Serialize, Deserialize)] +pub enum MulticamAudio { + /// The audio of camera 1 (plus audio-only clips); the other cameras' audio tracks are muted. + #[default] + Camera1, + /// Every camera's audio, mixed. + AllCameras, + /// The audio of the selected angle (plus audio-only clips): multi-camera audio clips switch + /// with their angle. + SwitchAudio, +} + +impl MulticamAudio { + pub fn from_name(s: &str) -> Option { + match s.to_ascii_lowercase().replace([' ', '_', '-'], "").as_str() { + "camera1" | "cam1" => Some(Self::Camera1), + "all" | "allcameras" => Some(Self::AllCameras), + "switch" | "switchaudio" => Some(Self::SwitchAudio), + _ => None, + } + } + pub fn name(self) -> &'static str { + match self { + Self::Camera1 => "camera1", + Self::AllCameras => "all", + Self::SwitchAudio => "switch", + } + } +} + +/// One camera angle of a multi-camera source sequence. +#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] +pub struct Camera { + pub name: String, + /// The angle's video track (None for an audio-only source, which is never an angle of the + /// grid but is mixed in). + pub video_track: Option, + /// The angle's audio tracks. + #[serde(default)] + pub audio_tracks: Vec, + /// Shown in the Multi-Camera view (Edit Cameras). + #[serde(default = "yes")] + pub enabled: bool, + /// The project item the angle was made from. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub source: Option, +} + +fn yes() -> bool { + true +} + +/// [`Sequence::multicam`](crate::Sequence::multicam): what makes a sequence a multi-camera source. +#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)] +pub struct MulticamSource { + /// Angles in camera order (angle `n` = `cameras[n]`). Audio-only sources come last and have no + /// video track. + pub cameras: Vec, + pub audio: MulticamAudio, + /// How the clips were synchronised (`in`, `out`, `timecode`, `marker`, `audio`), informational. + #[serde(default)] + pub sync: String, +} + +impl MulticamSource { + /// Angles that have a video track (the grid), in camera order: `(angle, camera)`. + pub fn video_angles(&self) -> impl Iterator { + self.cameras.iter().enumerate().filter(|(_, c)| c.video_track.is_some()) + } + /// Enabled video angles: the cameras shown in the Multi-Camera view (keys 1–9 pick from these). + pub fn shown_angles(&self) -> Vec { + self.video_angles().filter(|(_, c)| c.enabled).map(|(i, _)| i).collect() + } + /// Angle whose video is on `track`. + pub fn angle_of_video_track(&self, track: TrackId) -> Option { + self.cameras.iter().position(|c| c.video_track == Some(track)) + } + /// Whether an audio track belongs to a camera with video (switchable) rather than to an + /// audio-only source. + pub fn camera_of_audio_track(&self, track: TrackId) -> Option { + self.cameras.iter().position(|c| c.audio_tracks.contains(&track)) + } + /// The audio tracks that play for `angle` (None = all of them, as in a plain nest). + /// Audio-only sources always play; camera audio follows [`MulticamAudio`]. + pub fn audible_tracks(&self, angle: Option) -> Vec { + let mut v = Vec::new(); + for (i, c) in self.cameras.iter().enumerate() { + let audio_only = c.video_track.is_none(); + let on = audio_only + || match self.audio { + MulticamAudio::Camera1 => i == self.first_video_angle().unwrap_or(0), + MulticamAudio::AllCameras => true, + MulticamAudio::SwitchAudio => Some(i) == angle.or(self.first_video_angle()), + }; + if on { + v.extend(c.audio_tracks.iter().copied()); + } + } + v + } + pub fn first_video_angle(&self) -> Option { + self.video_angles().map(|(i, _)| i).next() + } +} + +/// [`TrackItem::multicam`](crate::TrackItem::multicam): the angle a multi-camera clip shows. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Serialize, Deserialize)] +pub struct MulticamSel { + /// Multi-Camera ▸ Enable. Off: the clip plays like any nested sequence. + pub enabled: bool, + /// Camera index into [`MulticamSource::cameras`] (video clips: the picture; audio clips: the + /// sound when the source switches audio). + pub angle: u32, +} + +/// [`Sequence::merged`](crate::Sequence::merged): what makes a sequence a merged clip. +#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)] +pub struct MergedClip { + /// The video clip (None: audio-only merge). + pub video: Option, + pub audio: Vec, + /// How the clips were synchronised. + #[serde(default)] + pub sync: String, +} + +impl crate::TrackItem { + /// The angle this clip shows when it is an enabled multi-camera clip, else None (it plays as an + /// ordinary nest). Any nested sequence can be a multi-camera clip: without camera data its + /// video tracks are the angles. + pub fn multicam_angle(&self, _nested: &Sequence) -> Option { + self.multicam.filter(|m| m.enabled).map(|m| m.angle as usize) + } +} + +impl Sequence { + /// The cameras of this sequence used as a multi-camera source: its camera data, or for a plain + /// sequence one camera per video track (named after the track) with all audio mixed. + pub fn cameras(&self) -> MulticamSource { + if let Some(m) = &self.multicam { + return m.clone(); + } + MulticamSource { + cameras: self + .video_tracks + .iter() + .enumerate() + .map(|(i, t)| Camera { + name: t.name.clone(), + video_track: Some(t.id), + audio_tracks: self.audio_tracks.get(i).map(|a| vec![a.id]).unwrap_or_default(), + enabled: true, + source: None, + }) + .collect(), + audio: MulticamAudio::AllCameras, + sync: String::new(), + } + } + /// Video track index of `angle` when this sequence is used as a multi-camera source. + pub fn angle_video_track_index(&self, angle: usize) -> Option { + match &self.multicam { + Some(mc) => { + let id = mc.cameras.get(angle)?.video_track?; + self.video_tracks.iter().position(|t| t.id == id) + } + None => (angle < self.video_tracks.len()).then_some(angle), + } + } + /// A copy whose audio tracks are muted except those audible for `angle` (None = the source's + /// default: camera 1, all, or the first angle when switching). Plain sequences are unchanged. + pub fn with_angle_audio(&self, angle: Option) -> Sequence { + let mut q = self.clone(); + if let Some(mc) = &self.multicam { + let on = mc.audible_tracks(angle); + for t in &mut q.audio_tracks { + if !on.contains(&t.id) { + t.muted = true; + } + } + } + q + } +} diff --git a/crates/render/src/audio.rs b/crates/render/src/audio.rs index 81ab684e..d05d59e7 100644 --- a/crates/render/src/audio.rs +++ b/crates/render/src/audio.rs @@ -76,16 +76,6 @@ pub fn pan_gains(pan: f32) -> (f32, f32) { #[allow(clippy::too_many_arguments)] fn mix_item(project: &Project, item: &TrackItem, start: i64, frames: usize, sr: u32, sources: &dyn SourceProvider, out: &mut AudioBuffer, extra: f32) { - let Some(src) = sources.source(item.item) else { return }; - if !src.info().has_audio() { - // nested sequences - if let Some(nested) = project.sequence(item.item) { - let rel0 = start - item.start.to_units_floor(sr as i64) + item.source_in.to_units_floor(sr as i64); - let b = mix_sequence(project, nested, rel0, frames, sources); - out.mix_from(&b, &[extra, extra]); - } - return; - } let item_s0 = item.start.to_units_floor(sr as i64); let item_s1 = item.end().to_units_floor(sr as i64); let a0 = start.max(item_s0); @@ -97,7 +87,15 @@ fn mix_item(project: &Project, item: &TrackItem, start: i64, frames: usize, sr: return; } let n = (a1 - a0) as usize; - let buf = effected(item, src.as_ref(), a0, n, sr); + let buf = if let Some(nested) = project.sequence(item.item) { + nested_audio(project, item, nested, a0, n, sr, sources) + } else { + let Some(src) = sources.source(item.item) else { return }; + if !src.info().has_audio() { + return; + } + effected(item, src.as_ref(), a0, n, sr) + }; // gains: clip gain × Volume (keyframed, per 64-sample block) × channel volume × panner let clip_gain = db_to_gain(item.gain_db); let vol = item.effect("volume").filter(|e| e.enabled && !e.param("bypass").and_then(|p| p.value.as_bool()).unwrap_or(false)); @@ -124,6 +122,23 @@ fn mix_item(project: &Project, item: &TrackItem, start: i64, frames: usize, sr: } } +/// A nested sequence's mix for timeline samples `[a0, a0 + n)` of the clip that shows it. A +/// multi-camera clip plays the audio of its source's audio setting: camera 1, all cameras, or (when +/// switching audio) the angle the clip selects. +fn nested_audio(project: &Project, item: &TrackItem, nested: &Sequence, a0: i64, n: usize, sr: u32, sources: &dyn SourceProvider) -> [Vec; 2] { + let rel0 = a0 - item.start.to_units_floor(sr as i64) + item.source_in.to_units_floor(sr as i64); + let b = if nested.multicam.is_some() { + let q = nested.with_angle_audio(item.multicam_angle(nested)); + mix_sequence(project, &q, rel0, n, sources) + } else { + mix_sequence(project, nested, rel0, n, sources) + }; + let mut it = b.channels.into_iter(); + let l = it.next().unwrap_or_else(|| vec![0.0; n]); + let r = it.next().unwrap_or_else(|| l.clone()); + [l, r] +} + /// The clip's audio after speed/reverse and clip effects (before clip gain, Volume and Panner) for /// timeline samples `[a0, a0 + n)`. fn effected(item: &TrackItem, src: &dyn filmcraft_media::MediaSource, a0: i64, n: usize, sr: u32) -> [Vec; 2] { diff --git a/crates/render/src/lib.rs b/crates/render/src/lib.rs index 0559be03..e31d1a67 100644 --- a/crates/render/src/lib.rs +++ b/crates/render/src/lib.rs @@ -19,6 +19,7 @@ pub mod image; pub mod luts; pub mod mask; pub mod mixer; +pub mod multicam; pub mod offline; pub mod plan; pub mod preview; @@ -80,18 +81,24 @@ pub fn render_sequence(project: &Project, seq_id: ItemId, t: Tick, opts: RenderO } fn render_seq(project: &Project, seq: &Sequence, t: Tick, opts: RenderOptions, sources: &dyn SourceProvider) -> Image { + render_seq_tracks(project, seq, t, opts, sources, None) +} + +/// Render a sequence, or only its video track `only` (a multi-camera angle; drawn even when the +/// track's output is off). +pub(crate) fn render_seq_tracks(project: &Project, seq: &Sequence, t: Tick, opts: RenderOptions, sources: &dyn SourceProvider, only: Option) -> Image { let (w, h) = output_size(seq, opts.scale); let mut canvas = Image::new(w, h); if opts.depth > 8 { return canvas; } let tc = format_time(t, seq.settings.frame_rate, seq.settings.drop_frame, TimeDisplay::Timecode, seq.settings.sample_rate as i64); - for track in &seq.video_tracks { + for (ti, track) in seq.video_tracks.iter().enumerate() { // A cancelled frame job (playback moved on) stops here; its result is discarded. if filmcraft_media::cancel::cancelled() { return canvas; } - if !track.enabled { + if only.is_some_and(|o| o != ti) || (!track.enabled && only.is_none()) { continue; } // transition covering t? @@ -285,7 +292,17 @@ pub(crate) fn item_layer( } ItemKind::Sequence(nested) => { let sub = RenderOptions { scale: want, effects: opts.effects, depth: opts.depth + 1, captions: false, working_output: true }; - render_seq(project, nested, mt, sub, sources) + match item.multicam_angle(nested) { + // a multi-camera clip shows its angle's track only (nothing for an audio-only angle) + Some(angle) => match nested.angle_video_track_index(angle) { + Some(ti) => render_seq_tracks(project, nested, mt, sub, sources, Some(ti)), + None => { + let (nw, nh) = output_size(nested, want); + Image::new(nw, nh) + } + }, + None => render_seq(project, nested, mt, sub, sources), + } } ItemKind::AdjustmentLayer { .. } => return None, ItemKind::Graphic { .. } => { diff --git a/crates/render/src/multicam.rs b/crates/render/src/multicam.rs new file mode 100644 index 00000000..a9b624cd --- /dev/null +++ b/crates/render/src/multicam.rs @@ -0,0 +1,61 @@ +//! Multi-camera rendering: one angle of a multi-camera source sequence, and the Multi-Camera +//! view's grid of angles. +//! +//! The grid renders every shown angle at a reduced scale (sources are asked for small frames, so +//! YUV→RGB conversion and compositing run at the cell size) in parallel, and tiles them into one +//! image. A 2×2 grid of 1080p angles at ¼ scale costs about as much as one ½-resolution frame of +//! compositing plus the four decodes. + +use filmcraft_project::{ItemId, Project, Sequence}; +use filmcraft_time::Tick; +use rayon::prelude::*; + +use crate::{Image, RenderOptions, SourceProvider, output_size}; + +/// Grid layout for `n` angles: (columns, rows). 1 → 1×1, 2–4 → 2×2, 5–9 → 3×3, 10–16 → 4×4. +pub fn grid_dims(n: usize) -> (usize, usize) { + let c = (1..=4).find(|c| c * c >= n).unwrap_or(4); + let r = n.div_ceil(c).max(1); + (c, r.min(c)) +} + +/// Render angle `angle` of the multi-camera source sequence `seq` at its time `t` (transparent +/// for an audio-only angle). The image is in the sequence's working space unless `opts.depth == 0` +/// and `!opts.working_output` (then display-ready like [`crate::render_sequence`]). +pub fn render_angle(project: &Project, seq: &Sequence, angle: usize, t: Tick, opts: RenderOptions, sources: &dyn SourceProvider) -> Image { + match seq.angle_video_track_index(angle) { + Some(ti) => crate::render_seq_tracks(project, seq, t, opts, sources, Some(ti)), + None => { + let (w, h) = output_size(seq, opts.scale); + Image::new(w, h) + } + } +} + +/// The Multi-Camera view's angle grid of the sequence `item` used as a multi-camera source at its time `t`: the shown +/// angles (Edit Cameras), each at `cell_scale` of the sequence frame size, tiled left to right, +/// top to bottom over black. Returns the image and the angles in grid order. +pub fn render_grid(project: &Project, item: ItemId, t: Tick, cell_scale: f32, sources: &dyn SourceProvider) -> Option<(Image, Vec)> { + let seq = project.sequence(item)?; + let angles = seq.cameras().shown_angles(); + let (cols, rows) = grid_dims(angles.len().max(1)); + let (cw, ch) = output_size(seq, cell_scale); + let opts = RenderOptions { scale: cell_scale, ..Default::default() }; + let cells: Vec = angles.par_iter().map(|&a| render_angle(project, seq, a, t, opts, sources)).collect(); + let mut out = Image::filled(cw * cols, ch * rows, [0.0, 0.0, 0.0, 1.0]); + for (k, cell) in cells.iter().enumerate().take(cols * rows) { + let (x0, y0) = ((k % cols) * cw, (k / cols) * ch); + for y in 0..ch.min(cell.h) { + for x in 0..cw.min(cell.w) { + let s = (y * cell.w + x) * 4; + let d = ((y0 + y) * out.w + x0 + x) * 4; + // cells over black: premultiplied over an opaque background + let a = cell.px[s + 3]; + for c in 0..3 { + out.px[d + c] = cell.px[s + c] + out.px[d + c] * (1.0 - a); + } + } + } + } + Some((out, angles)) +} diff --git a/crates/render/src/plan.rs b/crates/render/src/plan.rs index aba9dec8..bdaf1f7b 100644 --- a/crates/render/src/plan.rs +++ b/crates/render/src/plan.rs @@ -145,6 +145,22 @@ fn push_item( ) { let mt = item.source_time_at(t); let (op, bl) = crate::opacity_blend(item, mt); + // A multi-camera clip that only shows its angle (no effects, untransformed, same frame size) + // draws the angle's clip directly: no CPU pass over the nested sequence. + if let Some(ItemKind::Sequence(nested)) = project.item(item.item).map(|p| &p.kind) + && let Some(angle) = item.multicam_angle(nested) + && bl == Blend::Normal + && !(opts.effects && item.has_standard_effects()) + && (nested.settings.width, nested.settings.height) == (seq.settings.width, seq.settings.height) + && near_identity(&motion_matrix(seq, item, (nested.settings.width, nested.settings.height), mt)) + && let Some(tr) = nested.angle_video_track_index(angle).and_then(|i| nested.video_tracks.get(i)) + && !tr.transitions.iter().any(|x| x.range().contains(mt)) + { + if let Some(inner) = tr.item_at(mt).filter(|i| i.enabled) { + push_item(project, nested, inner, mt, opts, sources, extra_opacity * op, out); + } + return; + } // Graphic clips without standard effects: the layers are rasterised (cached) into one tight // image the GPU places as a layer. if bl == Blend::Normal @@ -183,6 +199,10 @@ fn push_item( } } +fn near_identity(m: &Affine) -> bool { + (m.a - 1.0).abs() < 1e-9 && (m.d - 1.0).abs() < 1e-9 && m.b.abs() < 1e-9 && m.c.abs() < 1e-9 && m.e.abs() < 1e-6 && m.f.abs() < 1e-6 +} + /// Execute a plan on the CPU (reference for the GPU compositor). pub fn execute_cpu(plan: &FramePlan) -> crate::Image { match plan { diff --git a/crates/ui-egui/src/frames.rs b/crates/ui-egui/src/frames.rs index 99874419..fcdc471a 100644 --- a/crates/ui-egui/src/frames.rs +++ b/crates/ui-egui/src/frames.rs @@ -26,6 +26,9 @@ pub enum Target { Item(ItemId), /// A GPU frame plan of a sequence (decoded layers + transforms), composited on the GPU. SequencePlan(ItemId), + /// The Multi-Camera view's angle grid of a multi-camera source at its time (all shown angles, + /// each at the job's scale). + MulticamGrid(ItemId), } #[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] @@ -707,6 +710,7 @@ fn render_job(job: &Job, pool: &Arc, services: &Arc, pr let img = match job.key.target { Target::Sequence(seq) | Target::SequencePlan(seq) => Some(filmcraft_render::render_sequence(&job.project, seq, job.time, opts, &provider)), Target::Item(item) => filmcraft_render::render_item(&job.project, item, job.time, job.scale, &provider), + Target::MulticamGrid(item) => filmcraft_render::multicam::render_grid(&job.project, item, job.time, job.scale, &provider).map(|(img, _)| img), }; let rgba = match img { Some(img) => Rgba { w: img.w, h: img.h, px: img.over_black_rgba8() }, diff --git a/crates/ui-egui/src/lib.rs b/crates/ui-egui/src/lib.rs index 530fdd64..fd786563 100644 --- a/crates/ui-egui/src/lib.rs +++ b/crates/ui-egui/src/lib.rs @@ -338,6 +338,8 @@ impl FilmcraftApp { let t = self.session.playhead(); let _ = self.session.execute("mixer.recordStart", json!({"time": t.0})); } + // Multi-Camera view: playing records live cuts (keys 1–9 / clicking angles) + panels::multicam::on_play(self); } pub fn stop(&mut self) { @@ -355,6 +357,7 @@ impl FilmcraftApp { self.ui.status = e.to_string(); } } + panels::multicam::on_stop(self); } fn start_audio(&mut self) { diff --git a/crates/ui-egui/src/menus.rs b/crates/ui-egui/src/menus.rs index af8e71f5..490d8c6f 100644 --- a/crates/ui-egui/src/menus.rs +++ b/crates/ui-egui/src/menus.rs @@ -39,6 +39,8 @@ pub const UI_COMMANDS: &[UiCommand] = &[ uic!("view.playbackRes.quarter", "1/4", ["View", "Playback Resolution"], None), uic!("view.playbackRes.eighth", "1/8", ["View", "Playback Resolution"], None), uic!("view.safeMargins", "Safe Margins", ["View"], None), + uic!("multicam.toggleView", "Multi-Camera View", ["View"], Some("Shift+0")), + uic!("multicam.recordToggle", "Multi-Camera Record On/Off Toggle", [], Some("0")), uic!("view.theme.dark", "Darkest", ["View", "Appearance"], None), uic!("view.theme.medium", "Medium", ["View", "Appearance"], None), uic!("view.theme.light", "Light", ["View", "Appearance"], None), @@ -100,6 +102,12 @@ pub fn invoke(app: &mut FilmcraftApp, ctx: &egui::Context, id: &str, params: Val if let Some(r) = crate::panels::trim_monitor::route_transport(app, ctx, id) { return r; } + if let Some(r) = crate::panels::multicam::route(app, id, ¶ms) { + if let Err(e) = &r { + app.ui.status = e.clone(); + } + return r; + } if let Some(r) = crate::panels::media_dialogs::route(app, id, ¶ms) { if let Err(e) = &r { app.ui.status = e.clone(); diff --git a/crates/ui-egui/src/panels/dialogs.rs b/crates/ui-egui/src/panels/dialogs.rs index 382b054d..dc18d4ac 100644 --- a/crates/ui-egui/src/panels/dialogs.rs +++ b/crates/ui-egui/src/panels/dialogs.rs @@ -7,6 +7,7 @@ pub fn show(app: &mut FilmcraftApp, ctx: &egui::Context) { crate::panels::media_dialogs::show(app, ctx); crate::panels::color_dialogs::show(app, ctx); crate::panels::presets::save_dialog(app, ctx); + crate::panels::multicam::show_dialog(app, ctx); let Some(d) = app.dialog else { return }; if let Some(still_open) = crate::panels::file_dialogs::show(app, ctx, d) { if !still_open && app.dialog == Some(d) { diff --git a/crates/ui-egui/src/panels/mod.rs b/crates/ui-egui/src/panels/mod.rs index 6b6f9cc7..2ad4d6f4 100644 --- a/crates/ui-egui/src/panels/mod.rs +++ b/crates/ui-egui/src/panels/mod.rs @@ -18,6 +18,7 @@ pub mod misc; pub mod mixer; pub mod monitor; pub mod presets; +pub mod multicam; pub mod project; pub mod shortcuts_dialog; pub mod text; diff --git a/crates/ui-egui/src/panels/monitor.rs b/crates/ui-egui/src/panels/monitor.rs index a7584ffa..ef5cb3c9 100644 --- a/crates/ui-egui/src/panels/monitor.rs +++ b/crates/ui-egui/src/panels/monitor.rs @@ -66,6 +66,17 @@ pub fn show(app: &mut FilmcraftApp, ui: &mut egui::Ui, rect: Rect, which: Which) } }; let prefix = if which == Which::Program { "program" } else { "source" }; + // Multi-Camera view: the angle grid on the left, the program on the right + let multicam = which == Which::Program && app.ui.program.multicam; + let (grid_area, video_area) = if multicam { + let mid = video_area.center().x; + (Some(Rect::from_min_max(video_area.min, pos2(mid - 2.0, video_area.max.y))), Rect::from_min_max(pos2(mid + 2.0, video_area.min.y), video_area.max)) + } else { + (None, video_area) + }; + if let Some(g) = grid_area { + crate::panels::multicam::grid(app, ui, g); + } ui.painter().rect_filled(video_area, 0.0, t.panel_bg); // ---- picture let has_video = frame_size.0 > 0; @@ -203,7 +214,14 @@ pub fn show(app: &mut FilmcraftApp, ui: &mut egui::Ui, rect: Rect, which: Which) if which == Which::Program { ui.separator(); ui.checkbox(&mut app.ui.show_scopes, "Lumetri Scopes"); + ui.checkbox(&mut app.ui.program.multicam, "Multi-Camera"); ui.checkbox(&mut app.playback.looping, "Loop"); + ui.separator(); + let mut follows = app.session.state.multicam_audio_follows_video; + if ui.checkbox(&mut follows, "Multi-Camera Audio Follows Video").changed() { + let _ = app.session.execute("multicam.audioFollowsVideo", json!({"enabled": follows})); + } + ui.checkbox(&mut app.ui.multicam_record, "Multi-Camera Record"); } }); @@ -216,7 +234,7 @@ pub fn show(app: &mut FilmcraftApp, ui: &mut egui::Ui, rect: Rect, which: Which) transport(app, ui, row3, which); } -fn quantize_scale(s: f32) -> f32 { +pub fn quantize_scale(s: f32) -> f32 { // Buckets keep the cache effective while resizing the panel. let buckets = [1.0 / 32.0, 1.0 / 16.0, 1.0 / 8.0, 0.1875, 0.25, 0.375, 0.5, 0.75, 1.0]; *buckets.iter().find(|b| **b >= s - 1e-4).unwrap_or(&1.0) diff --git a/crates/ui-egui/src/panels/multicam.rs b/crates/ui-egui/src/panels/multicam.rs new file mode 100644 index 00000000..51a0c933 --- /dev/null +++ b/crates/ui-egui/src/panels/multicam.rs @@ -0,0 +1,381 @@ +//! Multi-camera UI: the Program monitor's Multi-Camera view (angle grid next to the program) and +//! the Synchronize, Merge Clips and Create Multi-Camera Source Sequence dialogs. +//! +//! The grid is one frame job ([`Target::MulticamGrid`]): every shown angle rendered at the grid +//! cell's size (reduced resolution) and tiled, prefetched like the program while playing. +//! Clicking an angle (or keys 1–9) cuts to it while recording (playing with Multi-Camera Record +//! on) and switches the clip at the playhead when stopped; Ctrl/⌘-click switches the video only. +//! +//! Automation ids: +//! - Multi-Camera view: `program.multicam.grid`, `program.multicam.angle.` (n = 1-based camera +//! in grid order), `program.multicam.record`. +//! - Synchronize: `sync.method.`, `sync.ignoreHours`, `sync.marker`, +//! `sync.offset`, `sync.track`, `sync.ok`, `sync.cancel`. +//! - Merge Clips: `merge.name`, `merge.method.<…>`, `merge.ignoreHours`, `merge.marker`, +//! `merge.offset`, `merge.removeVideoAudio`, `merge.ok`, `merge.cancel`. +//! - Create Multi-Camera Source Sequence: `mcam.name`, `mcam.method.<…>`, `mcam.ignoreHours`, +//! `mcam.marker`, `mcam.offset`, `mcam.audio.`, +//! `mcam.cameraNames.`, `mcam.processedBin`, `mcam.ok`, `mcam.cancel`. + +use egui::{Align2, Color32, Rect, RichText, Sense, Stroke, StrokeKind, pos2, vec2}; +use filmcraft_project::ItemId; +use filmcraft_time::Tick; +use serde_json::{Value, json}; + +use crate::FilmcraftApp; +use crate::frames::{FrameKey, Target}; +use crate::state::SyncDraft; +use crate::theme::Tokens; + +type Elems = Vec<(String, Rect, String)>; + +/// Menu / shortcut entry points: the three dialogs (without params) and the view toggles. +pub fn route(app: &mut FilmcraftApp, id: &str, params: &Value) -> Option> { + match id { + "multicam.toggleView" => { + app.ui.program.multicam = params.get("enabled").and_then(Value::as_bool).unwrap_or(!app.ui.program.multicam); + return Some(Ok(json!({"multicam": app.ui.program.multicam}))); + } + "multicam.recordToggle" => { + app.ui.multicam_record = params.get("enabled").and_then(Value::as_bool).unwrap_or(!app.ui.multicam_record); + app.ui.status = format!("Multi-Camera Record {}", if app.ui.multicam_record { "on" } else { "off" }); + return Some(Ok(json!({"record": app.ui.multicam_record}))); + } + _ => {} + } + if !params.as_object().is_none_or(|m| m.is_empty()) { + return None; + } + let kind = match id { + "clip.synchronize" => "synchronize", + "clip.mergeClips" => "merge", + "clip.createMulticam" => "multicam", + _ => return None, + }; + if let Err(e) = filmcraft_engine::find_command(id).map_or(Ok(()), |c| (c.enabled)(&app.session)) { + return Some(Err(e)); + } + let items: Vec = app.session.state.project_selection.iter().map(|i| i.0).collect(); + let first = app.session.state.project_selection.first().and_then(|i| app.session.project.item(*i)).map(|i| i.name.clone()).unwrap_or_default(); + let name = match kind { + "merge" => format!("{first} - Merged"), + "multicam" => format!("{first} Multicam"), + _ => String::new(), + }; + app.ui.sync_dialog = Some(SyncDraft { kind: kind.into(), items, name, ..Default::default() }); + Some(Ok(json!({"dialog": kind}))) +} + +fn push(elems: &mut Elems, id: impl Into, r: &egui::Response, label: impl Into) { + elems.push((id.into(), r.rect, label.into())); +} + +const METHODS: [(&str, &str); 5] = [("in", "In Points"), ("out", "Out Points"), ("timecode", "Timecode"), ("marker", "Clip Marker"), ("audio", "Audio")]; + +/// Draw the open Synchronize / Merge Clips / Create Multi-Camera dialog. +pub fn show_dialog(app: &mut FilmcraftApp, ctx: &egui::Context) { + let Some(mut d) = app.ui.sync_dialog.clone() else { return }; + let (title, pre) = match d.kind.as_str() { + "merge" => ("Merge Clips", "merge"), + "multicam" => ("Create Multi-Camera Source Sequence", "mcam"), + _ => ("Synchronize Clips", "sync"), + }; + let mut elems: Elems = Vec::new(); + let mut action: Option<&str> = None; + let accent = app.tokens.accent; + let names: Vec = d.items.iter().filter_map(|i| app.session.project.item(ItemId(*i)).map(|it| it.name.clone())).collect(); + egui::Window::new(title).collapsible(false).resizable(false).default_width(440.0).anchor(Align2::CENTER_CENTER, [0.0, 0.0]).show(ctx, |ui| { + if pre != "sync" { + ui.horizontal(|ui| { + ui.label(if pre == "merge" { "Clip Name:" } else { "Sequence Name:" }); + let r = ui.add(egui::TextEdit::singleline(&mut d.name).desired_width(300.0)); + push(&mut elems, format!("{pre}.name"), &r, "name"); + }); + ui.label(RichText::new(format!("{} clip(s): {}", names.len(), names.join(", "))).weak().small()); + ui.add_space(4.0); + } else { + let n = app.session.state.selection.len(); + ui.label(RichText::new(format!("{n} clip(s) selected; the clip on the reference track stays put.")).weak().small()); + } + ui.group(|ui| { + ui.set_min_width(400.0); + ui.label(RichText::new("Synchronize Point").strong()); + for (m, label) in METHODS { + ui.horizontal(|ui| { + let r = ui.radio(d.method == m, label); + push(&mut elems, format!("{pre}.method.{m}"), &r, label); + if r.clicked() { + d.method = m.into(); + } + match m { + "timecode" => { + let r = ui.add_enabled(d.method == "timecode", egui::Checkbox::new(&mut d.ignore_hours, "Ignore Hours")); + push(&mut elems, format!("{pre}.ignoreHours"), &r, "Ignore Hours"); + } + "marker" => { + let r = + ui.add_enabled(d.method == "marker", egui::TextEdit::singleline(&mut d.marker).hint_text("first marker").desired_width(140.0)); + push(&mut elems, format!("{pre}.marker"), &r, "marker name"); + } + _ => {} + } + }); + } + ui.horizontal(|ui| { + ui.label(if pre == "merge" { "Offset Audio by:" } else { "Offset:" }); + let r = ui.add(egui::DragValue::new(&mut d.offset).range(-100_000..=100_000).suffix(" frames")); + push(&mut elems, format!("{pre}.offset"), &r, "offset"); + }); + }); + match pre { + "sync" => { + ui.horizontal(|ui| { + ui.label("Reference track:"); + let r = ui.add(egui::TextEdit::singleline(&mut d.track).hint_text("lowest (V1…)").desired_width(90.0)); + push(&mut elems, "sync.track", &r, "reference track"); + }); + } + "merge" => { + let r = ui.checkbox(&mut d.remove_video_audio, "Remove Audio from AV Clip"); + push(&mut elems, "merge.removeVideoAudio", &r, "Remove Audio from AV Clip"); + } + _ => { + ui.group(|ui| { + ui.set_min_width(400.0); + ui.label(RichText::new("Audio").strong()); + ui.horizontal(|ui| { + for (a, label) in [("camera1", "Camera 1"), ("all", "All Cameras"), ("switch", "Switch Audio")] { + let r = ui.radio(d.audio == a, label); + push(&mut elems, format!("mcam.audio.{a}"), &r, label); + if r.clicked() { + d.audio = a.into(); + } + } + }); + ui.label(RichText::new("Camera Names").strong()); + ui.horizontal(|ui| { + for (a, label) in [("clip", "Clip Names"), ("track", "Enumerate Cameras"), ("metadata", "Camera Angle Metadata")] { + let r = ui.radio(d.camera_names == a, label); + push(&mut elems, format!("mcam.cameraNames.{a}"), &r, label); + if r.clicked() { + d.camera_names = a.into(); + } + } + }); + }); + let r = ui.checkbox(&mut d.processed_bin, "Move source clips to Processed Clips bin"); + push(&mut elems, "mcam.processedBin", &r, "Move source clips to Processed Clips bin"); + } + } + if !d.message.is_empty() { + ui.colored_label(Color32::from_rgb(0xe0, 0x8a, 0x6a), &d.message); + } + ui.add_space(6.0); + ui.horizontal(|ui| { + let r = ui.button("Cancel"); + push(&mut elems, format!("{pre}.cancel"), &r, "Cancel"); + if r.clicked() { + action = Some("cancel"); + } + let r = ui.add(egui::Button::new(RichText::new("OK").color(Color32::WHITE)).fill(accent)); + push(&mut elems, format!("{pre}.ok"), &r, "OK"); + if r.clicked() { + action = Some("ok"); + } + }); + }); + for (id, r, l) in elems { + app.auto.add(&id, r, &l); + } + if ctx.input(|i| i.key_pressed(egui::Key::Escape)) { + action = Some("cancel"); + } + match action { + Some("cancel") => { + app.ui.sync_dialog = None; + return; + } + Some("ok") => { + let mut p = json!({"method": d.method, "offset": d.offset, "ignoreHours": d.ignore_hours}); + if !d.marker.is_empty() { + p["marker"] = json!(d.marker); + } + let cmd = match pre { + "merge" => { + p["items"] = json!(d.items); + p["removeVideoAudio"] = json!(d.remove_video_audio); + if !d.name.is_empty() { + p["name"] = json!(d.name); + } + "clip.mergeClips" + } + "mcam" => { + p["items"] = json!(d.items); + p["audio"] = json!(d.audio); + p["cameraNames"] = json!(d.camera_names); + p["processedBin"] = json!(d.processed_bin); + if !d.name.is_empty() { + p["name"] = json!(d.name); + } + "clip.createMulticam" + } + _ => { + if !d.track.is_empty() { + p["track"] = json!(d.track); + } + "clip.synchronize" + } + }; + match app.session.execute(cmd, p) { + Ok(v) => { + app.ui.status = match pre { + "sync" => format!("Synchronized {} clip group(s)", v["moved"]), + "merge" => "Merged clip created".into(), + _ => format!("Multi-camera source sequence with {} camera(s)", v["cameras"].as_array().map_or(0, Vec::len)), + }; + app.ui.sync_dialog = None; + return; + } + Err(e) => d.message = e.to_string(), + } + } + _ => {} + } + app.ui.sync_dialog = Some(d); +} + +/// A revision for grid frames: changes when the multi-camera source (or its cameras' media) +/// changes, not with every edit of the cut (cuts recorded live must not throw away the grid). +fn grid_revision(app: &FilmcraftApp, src: ItemId) -> u64 { + use std::hash::{Hash, Hasher}; + let mut h = std::collections::hash_map::DefaultHasher::new(); + if let Some(q) = app.session.project.sequence(src) { + serde_json::to_vec(q).unwrap_or_default().hash(&mut h); + for t in &q.video_tracks { + for it in &t.items { + app.item_revision(it.item).hash(&mut h); + } + } + } + h.finish() +} + +/// The angle grid of the Multi-Camera view in `area`. +pub fn grid(app: &mut FilmcraftApp, ui: &mut egui::Ui, area: Rect) { + let t = app.tokens; + let ctx = ui.ctx().clone(); + ui.painter().rect_filled(area, 0.0, t.panel_bg); + app.auto.add("program.multicam.grid", area, "Multi-Camera"); + let playhead = app.session.playhead(); + let info = filmcraft_engine::multicam::inspect_at(&app.session, playhead); + let Some(src) = info["source"].as_u64().map(ItemId) else { + ui.painter().text(area.center(), Align2::CENTER_CENTER, "No multi-camera clip at the playhead", Tokens::ui(12.0), t.text_dim); + return; + }; + let Some(q) = app.session.project.sequence(src) else { return }; + let cams = q.cameras(); + let shown = cams.shown_angles(); + let (cols, rows) = filmcraft_render::multicam::grid_dims(shown.len().max(1)); + let (sw, sh) = (q.settings.width as f32, q.settings.height as f32); + let rate = q.settings.frame_rate; + let pic = crate::panels::monitor::fit(area.shrink(2.0), sw * cols as f32, sh * rows as f32); + let (cw, ch) = (pic.width() / cols as f32, pic.height() / rows as f32); + // decode and composite at the cell's size (never more than the playback resolution) + let cell_px = cw * ctx.pixels_per_point(); + let scale = crate::panels::monitor::quantize_scale(app.ui.program.res.scale().min((cell_px / sw).max(1.0 / 32.0))); + let mt = Tick(info["sourceTime"].as_i64().unwrap_or(0)); + let frame = rate.frame_at(mt); + let key = FrameKey { target: Target::MulticamGrid(src), frame, size: (scale * 1000.0) as u32, revision: grid_revision(app, src) }; + let project = app.session.project.clone(); + if app.playback.playing { + app.frames.schedule_playback(key, rate, scale, &project, app.playback.speed, app.playback.preroll.is_some()); + } else { + app.frames.request(key, rate.tick_of(frame), scale, &project, 0); + } + let tex = match app.frames.get(&key) { + Some(img) => Some(app.texture_for(&ctx, "monitor-multicam", key, &img)), + None => match app.frames.nearest(key.target, frame, key.size, key.revision, 6) { + Some(img) => Some(app.texture_for(&ctx, "monitor-multicam", FrameKey { frame: frame - 1, ..key }, &img)), + None => app.texture_existing("monitor-multicam").map(|(id, _)| id), + }, + }; + ui.painter().rect_filled(pic, 0.0, t.monitor_bg); + if let Some(tex) = tex { + ui.painter().image(tex, pic, Rect::from_min_max(pos2(0.0, 0.0), pos2(1.0, 1.0)), Color32::WHITE); + } + let current = info["angle"].as_u64().map(|a| a as usize); + let recording = app.session.mcrec.active(); + let mut click: Option<(usize, bool)> = None; + for (k, &angle) in shown.iter().enumerate() { + let cell = Rect::from_min_size(pic.min + vec2((k % cols) as f32 * cw, (k / cols) as f32 * ch), vec2(cw, ch)); + let name = cams.cameras.get(angle).map(|c| c.name.clone()).unwrap_or_default(); + let resp = ui.interact(cell, egui::Id::new(("multicam-angle", k)), Sense::click()).on_hover_text(format!("Camera {} — {name}", k + 1)); + app.auto.add(&format!("program.multicam.angle.{}", k + 1), cell, &name); + if resp.clicked() { + // modifiers held now, or carried by the click event (synthetic input) + let video_only = ui.input(|i| { + let ev = i.events.iter().any(|e| matches!(e, egui::Event::PointerButton { modifiers: m, .. } if m.ctrl || m.command)); + i.modifiers.ctrl || i.modifiers.command || ev + }); + click = Some((k + 1, video_only)); + } + if resp.hovered() { + ui.painter().rect_stroke(cell.shrink(1.0), 0.0, Stroke::new(1.0, Color32::from_white_alpha(90)), StrokeKind::Inside); + } + if current == Some(angle) { + // the active angle: yellow, red while recording + let col = if recording { Color32::from_rgb(0xe0, 0x3a, 0x3a) } else { Color32::from_rgb(0xf2, 0xc9, 0x4c) }; + ui.painter().rect_stroke(cell.shrink(1.5), 0.0, Stroke::new(3.0, col), StrokeKind::Inside); + } + let label = format!("{} {name}", k + 1); + let galley = ui.painter().layout_no_wrap(label, Tokens::ui(11.0), Color32::WHITE); + let lr = Rect::from_min_size(cell.left_bottom() + vec2(6.0, -galley.size().y - 8.0), galley.size() + vec2(8.0, 4.0)); + ui.painter().rect_filled(lr, 3.0, Color32::from_black_alpha(160)); + ui.painter().galley(lr.min + vec2(4.0, 2.0), galley, Color32::WHITE); + } + // record indicator + let rr = Rect::from_min_size(area.min + vec2(8.0, 8.0), vec2(64.0, 18.0)); + let rec_on = app.ui.multicam_record; + let resp = ui.interact(rr, egui::Id::new("multicam-record"), Sense::click()).on_hover_text("Multi-Camera Record On/Off (0)"); + app.auto.add("program.multicam.record", rr, "Multi-Camera Record"); + let dot = if recording { + Color32::from_rgb(0xe0, 0x3a, 0x3a) + } else if rec_on { + Color32::from_rgb(0x9a, 0x3a, 0x3a) + } else { + t.text_faint + }; + ui.painter().rect_filled(rr, 3.0, Color32::from_black_alpha(140)); + ui.painter().circle_filled(rr.left_center() + vec2(9.0, 0.0), 4.5, dot); + ui.painter().text(rr.left_center() + vec2(18.0, 0.0), Align2::LEFT_CENTER, if recording { "REC" } else { "Record" }, Tokens::ui(10.5), Color32::WHITE); + if resp.clicked() { + app.ui.multicam_record = !app.ui.multicam_record; + } + if let Some((n, video_only)) = click + && let Err(e) = app.session.execute("multicam.cut", json!({"camera": n, "videoOnly": video_only})) + { + app.ui.status = e.to_string(); + } +} + +/// Start a recording pass when playback starts in the Multi-Camera view. +pub fn on_play(app: &mut FilmcraftApp) { + if app.ui.program.multicam && app.ui.multicam_record && !app.session.mcrec.active() && (app.playback.speed - 1.0).abs() < 1e-9 { + let t = app.session.playhead(); + if filmcraft_engine::multicam::multicam_clip_at(&app.session, t).is_some() { + let _ = app.session.execute("multicam.recordStart", json!({"time": t.0})); + } + } +} + +/// End the recording pass (one undo step) when playback stops. +pub fn on_stop(app: &mut FilmcraftApp) { + if app.session.mcrec.active() { + let t = app.session.playhead(); + match app.session.execute("multicam.recordStop", json!({"time": t.0})) { + Ok(v) if v["cuts"].as_u64().unwrap_or(0) > 0 => app.ui.status = format!("Recorded {} multi-camera cut(s)", v["cuts"]), + Ok(_) => {} + Err(e) => app.ui.status = e.to_string(), + } + } +} diff --git a/crates/ui-egui/src/panels/timeline.rs b/crates/ui-egui/src/panels/timeline.rs index ef5a8e46..679ac9da 100644 --- a/crates/ui-egui/src/panels/timeline.rs +++ b/crates/ui-egui/src/panels/timeline.rs @@ -428,7 +428,18 @@ fn draw_clip( if let Some(aspect) = aspect { let tw = (th.height() * aspect).min(th.width()); let mt = rate.snap(it.source_in); - if let Some((tex, _)) = app.thumbnail(ctx, it.item, mt, 160) { + // a multi-camera clip shows its angle's clip + let (thumb_item, mt) = app + .session + .project + .sequence(it.item) + .and_then(|q| { + let ti = q.angle_video_track_index(it.multicam_angle(q)?)?; + let inner = q.video_tracks[ti].item_at(mt)?; + Some((inner.item, inner.source_time_at(mt))) + }) + .unwrap_or((it.item, mt)); + if let Some((tex, _)) = app.thumbnail(ctx, thumb_item, mt, 160) { let r = Rect::from_min_size(th.min, vec2(tw, th.height())); let cp = p.with_clip_rect(th.intersect(p.clip_rect())); cp.image(tex, r, Rect::from_min_max(pos2(0.0, 0.0), pos2((tw / (th.height() * aspect)).min(1.0), 1.0)), Color32::WHITE); @@ -443,6 +454,10 @@ fn draw_clip( // name band if w > 24.0 { let mut label = it.name.clone(); + if let Some(m) = it.multicam.filter(|m| m.enabled) { + let cam = app.session.project.sequence(it.item).and_then(|q| q.cameras().cameras.get(m.angle as usize).map(|c| c.name.clone())); + label = format!("[MC{}] {}", m.angle + 1, cam.unwrap_or(label)); + } if (it.speed - 1.0).abs() > 1e-6 || it.reverse { label = format!("{label} [{}%]", ((if it.reverse { -1.0 } else { 1.0 }) * it.speed * 100.0).round()); } diff --git a/crates/ui-egui/src/state.rs b/crates/ui-egui/src/state.rs index 48bf90b6..3609f37c 100644 --- a/crates/ui-egui/src/state.rs +++ b/crates/ui-egui/src/state.rs @@ -220,11 +220,14 @@ pub struct MonitorView { pub zoom: Option, pub safe_margins: bool, pub show_transport: bool, + /// Program monitor display mode Multi-Camera (angle grid + program). + #[serde(default)] + pub multicam: bool, } impl Default for MonitorView { fn default() -> Self { - Self { res: PlaybackRes::Half, zoom: None, safe_margins: false, show_transport: true } + Self { res: PlaybackRes::Half, zoom: None, safe_margins: false, show_transport: true, multicam: false } } } @@ -316,6 +319,12 @@ pub struct UiState { /// Effect presets: the preset being renamed / the Save Preset dialog (open when Some). #[serde(default)] pub save_preset: Option, + /// Synchronize / Merge Clips / Create Multi-Camera Source Sequence dialog (open when Some). + #[serde(default)] + pub sync_dialog: Option, + /// Multi-Camera Record On/Off (key 0): playing in the Multi-Camera view records cuts. + #[serde(default = "yes")] + pub multicam_record: bool, } /// A pen mask in progress: vertices placed so far, in clip pixels (`[x, y, tangent x, tangent y]`). @@ -337,6 +346,54 @@ pub struct SavePresetDraft { pub keyframes: String, } +/// Clip ▸ Synchronize…, Merge Clips… and Create Multi-Camera Source Sequence… share one draft; +/// `kind` says which dialog it is (`synchronize`, `merge`, `multicam`). +#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] +#[serde(default)] +pub struct SyncDraft { + pub kind: String, + /// Project items (Merge Clips, Create Multi-Camera). + pub items: Vec, + pub name: String, + /// `in` | `out` | `timecode` | `marker` | `audio` + pub method: String, + pub ignore_hours: bool, + /// Clip marker name to sync on (empty = the first marker). + pub marker: String, + /// Frames added to every clip but the reference. + pub offset: i64, + /// Synchronize: reference track (`V1`, `A2`…; empty = the lowest track). + pub track: String, + /// Create Multi-Camera: `camera1` | `all` | `switch`. + pub audio: String, + /// Create Multi-Camera: `clip` | `track` | `metadata`. + pub camera_names: String, + pub processed_bin: bool, + /// Merge Clips: drop the video clip's own audio. + pub remove_video_audio: bool, + pub message: String, +} + +impl Default for SyncDraft { + fn default() -> Self { + Self { + kind: "synchronize".into(), + items: vec![], + name: String::new(), + method: "audio".into(), + ignore_hours: false, + marker: String::new(), + offset: 0, + track: String::new(), + audio: "camera1".into(), + camera_names: "clip".into(), + processed_bin: true, + remove_video_audio: false, + message: String::new(), + } + } +} + /// File ▸ Link Media… (shown automatically when a project opens with missing media). #[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] #[serde(default)] @@ -508,6 +565,8 @@ impl Default for UiState { color_dialog: None, mask_pen: None, save_preset: None, + sync_dialog: None, + multicam_record: true, } } } diff --git a/crates/ui-egui/tests/multicam_ui.rs b/crates/ui-egui/tests/multicam_ui.rs new file mode 100644 index 00000000..28c6e1c3 --- /dev/null +++ b/crates/ui-egui/tests/multicam_ui.rs @@ -0,0 +1,301 @@ +//! Headless UI tests of multi-camera editing: the Create Multi-Camera Source Sequence, Synchronize +//! and Merge Clips dialogs, the Program monitor's Multi-Camera view (angle grid, clicking an +//! angle, Ctrl-click for video only) and live switching with the 1–9 keys while playing. +//! +//! Set `FILMCRAFT_UI_SNAPSHOT_DIR=` to also render the window offscreen with wgpu and write PNGs +//! there (`multicam-*.png`). + +use std::path::{Path, PathBuf}; +use std::sync::Arc; +use std::sync::mpsc::{Sender, channel}; + +use egui_kittest::Harness; +use filmcraft_engine::Session; +use filmcraft_engine::project::{ItemKind, Label, MediaClip, MediaRef, Project, SequenceSettings, TrackKind}; +use filmcraft_media::generators::GeneratorSource; +use filmcraft_media::{DemoScene, Generator, MediaSource}; +use filmcraft_time::{FrameRate, Tick, TimeRange}; +use filmcraft_ui_egui::FilmcraftApp; +use filmcraft_ui_egui::control::ControlRequest; +use serde_json::{Value, json}; + +const W: u32 = 320; +const H: u32 = 180; + +fn tmp_dir(name: &str) -> PathBuf { + let nanos = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).unwrap().as_nanos(); + let d = std::env::temp_dir().join(format!("filmcraft-ui-{name}-{}-{nanos}", std::process::id())); + std::fs::create_dir_all(&d).unwrap(); + d +} + +/// A 24 fps ProRes movie of a demo scene (our own encoder). +fn make_movie(path: &Path, scene: DemoScene, frames: i64) { + let rate = FrameRate::FPS_24; + let dur = rate.tick_of(frames); + let src = GeneratorSource::new(Generator::Demo(scene), W, H, rate, dur); + let clip = MediaClip { + media: MediaRef::Generator(Generator::Demo(scene)), + info: src.info().clone(), + interpret: Default::default(), + mark_in: None, + mark_out: None, + markers: vec![], + offline: false, + proxy: None, + identity: None, + }; + let mut p = Project::new("fixture"); + let item = p.add_item("scene", Label::Iris, ItemKind::Media(clip), None); + let seq = p.new_sequence("s", SequenceSettings { width: W, height: H, frame_rate: rate, ..Default::default() }, 1, 0, None); + let mut v = p.make_track_item(item, TrackKind::Video, Tick::ZERO, TimeRange::new(Tick::ZERO, dur), rate).unwrap(); + for e in &mut v.effects { + filmcraft_engine::project::resolve_auto_points(e, (W, H), (W, H)); + } + p.sequence_mut(seq).unwrap().video_tracks[0].items.push(v); + let settings = filmcraft_engine::export::ExportSettings { + format: filmcraft_engine::export::Format::ProRes, + path: path.to_string_lossy().into_owned(), + include_audio: false, + ..Default::default() + }; + let shared: filmcraft_media::SharedSource = Arc::new(src); + let provider = move |id| (id == item).then(|| shared.clone()); + filmcraft_engine::export::export(&Arc::new(p), seq, &settings, &provider, &Default::default()).unwrap(); +} + +/// Four cameras imported into a session with an empty 320×180 sequence "Cut". +fn session() -> (Session, Vec) { + let dir = tmp_dir("multicam"); + let scenes = [DemoScene::OceanSunset, DemoScene::CityNight, DemoScene::Aurora, DemoScene::Forest]; + let files: Vec = (0..4).map(|i| dir.join(format!("Cam {}.mov", (b'A' + i as u8) as char))).collect(); + for (f, sc) in files.iter().zip(scenes) { + make_movie(f, sc, 96); + } + let mut s = Session::default(); + let r = s.execute("file.import", json!({"paths": files.iter().map(|f| f.to_string_lossy()).collect::>()})).unwrap(); + let items: Vec = r["items"].as_array().unwrap().iter().map(|v| v.as_u64().unwrap()).collect(); + let mut p = (*s.project).clone(); + let seq = p.new_sequence("Cut", SequenceSettings { width: W, height: H, frame_rate: FrameRate::FPS_24, ..Default::default() }, 2, 2, None); + s.project = Arc::new(p); + s.state.active_sequence = Some(seq); + s.state.open_sequences = vec![seq]; + (s, items) +} + +struct Driver { + harness: Harness<'static, FilmcraftApp>, + tx: Sender, + snapshots: Option, +} + +impl Driver { + fn new(session: Session) -> Self { + let (tx, rx) = channel(); + let app = FilmcraftApp::new(session).with_control(rx); + let snapshots = std::env::var_os("FILMCRAFT_UI_SNAPSHOT_DIR").map(PathBuf::from); + let mut b = Harness::builder().with_size(egui::vec2(1600.0, 980.0)).with_max_steps(10_000); + if snapshots.is_some() { + b = b.wgpu(); + } + let harness = b.build_eframe(move |_cc| app); + let mut d = Driver { harness, tx, snapshots }; + d.frames(4); + d + } + + fn frames(&mut self, n: usize) { + for _ in 0..n { + let ctx = self.harness.ctx.clone(); + let mut raw = std::mem::take(self.harness.input_mut()); + eframe::App::raw_input_hook(self.harness.state_mut(), &ctx, &mut raw); + *self.harness.input_mut() = raw; + self.harness.step(); + } + } + + /// Run frames (with short sleeps so frame workers finish) until `done`. + fn until(&mut self, what: &str, mut done: impl FnMut(&mut Self) -> bool) { + for _ in 0..600 { + if done(self) { + return; + } + self.frames(1); + std::thread::sleep(std::time::Duration::from_millis(10)); + } + panic!("timed out waiting for {what}"); + } + + fn call(&mut self, method: &str, params: Value) -> Value { + let (req, reply) = ControlRequest::new(method, params.clone()); + self.tx.send(req).unwrap(); + for _ in 0..600 { + self.frames(1); + if let Ok(v) = reply.try_recv() { + return v; + } + } + panic!("no reply to {method} {params}"); + } + + fn ok(&mut self, method: &str, params: Value) -> Value { + let v = self.call(method, params.clone()); + assert_eq!(v["ok"], json!(true), "{method} {params} failed: {v}"); + v["result"].clone() + } + + fn exec(&mut self, command: &str, params: Value) -> Value { + self.ok("engine.execute", json!({"command": command, "params": params})) + } + + fn click(&mut self, id: &str) { + self.ok("ui.click", json!({"id": id})); + self.frames(3); + } + + fn ids(&mut self, prefix: &str) -> Vec { + let v = self.ok("ui.elements", json!({"prefix": prefix})); + v.as_array().unwrap().iter().filter_map(|e| e["id"].as_str().map(str::to_string)).collect() + } + + fn app(&mut self) -> &mut FilmcraftApp { + self.harness.state_mut() + } + + /// Angles of the clips on V1 / A1 of the active sequence. + fn angles(&mut self, audio: bool) -> Vec { + let q = self.app().session.active_sequence().unwrap().clone(); + let t = if audio { &q.audio_tracks[0] } else { &q.video_tracks[0] }; + t.items.iter().map(|i| i.multicam.map_or(99, |m| m.angle)).collect() + } + + fn snapshot(&mut self, name: &str) { + let Some(dir) = self.snapshots.clone() else { return }; + self.frames(3); + let img = match self.harness.render() { + Ok(i) => i, + Err(e) => { + eprintln!("snapshot {name} skipped: {e}"); + return; + } + }; + std::fs::create_dir_all(&dir).unwrap(); + let path = dir.join(format!("{name}.png")); + img.save(&path).unwrap(); + eprintln!("snapshot: {}", path.display()); + } +} + +#[test] +fn create_multicam_dialog_view_and_live_switching() { + let (s, items) = session(); + let mut d = Driver::new(s); + // Create Multi-Camera Source Sequence from the Project panel selection + d.exec("project.select", json!({"items": items})); + d.ok("ui.menu.invoke", json!({"id": "clip.createMulticam"})); + d.frames(3); + assert_eq!(d.app().ui.sync_dialog.as_ref().map(|x| x.kind.clone()).as_deref(), Some("multicam")); + for id in ["mcam.name", "mcam.method.audio", "mcam.method.timecode", "mcam.audio.switch", "mcam.cameraNames.track", "mcam.processedBin", "mcam.ok"] { + assert!(!d.ids(id).is_empty(), "{id}"); + } + d.click("mcam.method.in"); + d.click("mcam.audio.switch"); + d.snapshot("multicam-create-dialog"); + d.click("mcam.ok"); + assert!(d.app().ui.sync_dialog.is_none(), "dialog closed: {}", d.app().ui.status); + let src = d.app().session.state.project_selection[0]; + let mc = d.app().session.project.sequence(src).unwrap().multicam.clone().unwrap(); + assert_eq!(mc.cameras.len(), 4); + assert_eq!(mc.cameras[0].name, "Cam A.mov"); + // edit it into the sequence, open the Multi-Camera view + d.exec("source.open", json!({"item": src.0})); + d.exec("playhead.set", json!({"frame": 0})); + d.exec("source.overwrite", json!({})); + d.exec("playhead.set", json!({"frame": 12})); + d.ok("ui.menu.invoke", json!({"id": "multicam.toggleView"})); + d.frames(2); + assert!(d.app().ui.program.multicam); + let angles = d.ids("program.multicam.angle."); + assert_eq!(angles.len(), 4, "{angles:?}"); + d.until("grid frame", |d| { + let k = d.app().frames.queue_len(); + k == 0 + }); + d.frames(6); + d.snapshot("multicam-view"); + // clicking an angle switches the clip at the playhead; audio follows video when asked + d.exec("multicam.audioFollowsVideo", json!({"enabled": true})); + d.click("program.multicam.angle.2"); + assert_eq!(d.angles(false), [1]); + assert_eq!(d.angles(true), [1], "audio followed"); + // Ctrl-click: video only + d.ok("ui.click", json!({"id": "program.multicam.angle.3", "modifiers": {"ctrl": true}})); + d.frames(3); + assert_eq!((d.angles(false), d.angles(true)), (vec![2], vec![1])); + d.until("program frame", |d| d.app().frames.queue_len() == 0); + d.frames(6); + d.snapshot("multicam-view-camera3"); + d.exec("playhead.set", json!({"frame": 0})); + d.click("program.multicam.angle.1"); + // live switching: play, press 2 then 4, stop → one undo step with the recorded cuts + let undo_before = d.app().session.history.undo.len(); + d.ok("ui.playback", json!({"action": "play"})); + d.until("recording", |d| d.app().session.mcrec.active()); + d.until("playhead past 0.3 s", |d| d.app().session.playhead() > FrameRate::FPS_24.tick_of(7)); + d.ok("ui.key", json!({"key": "2"})); + d.frames(2); + d.until("playhead past 0.8 s", |d| d.app().session.playhead() > FrameRate::FPS_24.tick_of(19)); + d.ok("ui.key", json!({"key": "4"})); + d.frames(4); + d.snapshot("multicam-recording"); + d.until("playhead past 1.2 s", |d| d.app().session.playhead() > FrameRate::FPS_24.tick_of(29)); + d.ok("ui.playback", json!({"action": "stop"})); + d.frames(2); + assert!(!d.app().session.mcrec.active()); + let v = d.angles(false); + assert_eq!(&v[..3], [0, 1, 3], "{v:?}"); + assert_eq!(d.angles(true)[..3], [0, 1, 3], "audio followed the recording"); + assert_eq!(d.app().session.history.undo.len(), undo_before + 1, "one undo step per pass"); + d.frames(4); + d.snapshot("multicam-after-recording"); + d.exec("edit.undo", json!({})); + assert_eq!(d.angles(false), [0]); +} + +#[test] +fn synchronize_and_merge_dialogs() { + let (s, items) = session(); + let mut d = Driver::new(s); + // two cameras on V1 and V2 at different times + d.exec("source.open", json!({"item": items[0]})); + d.exec("playhead.set", json!({"frame": 0})); + d.exec("source.overwrite", json!({})); + d.exec("timeline.setTargeting", json!({"track": "V2", "sourcePatch": true})); + d.exec("timeline.setTargeting", json!({"track": "A2", "sourcePatch": true})); + d.exec("source.open", json!({"item": items[1]})); + d.exec("playhead.set", json!({"frame": 30})); + d.exec("source.overwrite", json!({})); + assert_eq!(d.app().session.active_sequence().unwrap().video_tracks[1].items.len(), 1); + let all: Vec = d.app().session.active_sequence().unwrap().all_tracks().flat_map(|t| t.items.iter().map(|i| i.id.0)).collect(); + d.exec("timeline.select", json!({"clips": all})); + d.ok("ui.menu.invoke", json!({"id": "clip.synchronize"})); + d.frames(3); + assert_eq!(d.app().ui.sync_dialog.as_ref().map(|x| x.kind.clone()).as_deref(), Some("synchronize")); + d.click("sync.method.in"); + d.snapshot("multicam-sync-dialog"); + d.click("sync.ok"); + assert!(d.app().ui.sync_dialog.is_none(), "{}", d.app().ui.status); + let q = d.app().session.active_sequence().unwrap().clone(); + assert_eq!(q.video_tracks[0].items[0].start, q.video_tracks[1].items[0].start, "In points aligned"); + // Merge Clips needs audio clips: the dialog explains what is wrong + d.exec("project.select", json!({"items": [items[0], items[1]]})); + d.ok("ui.menu.invoke", json!({"id": "clip.mergeClips"})); + d.frames(3); + assert!(!d.ids("merge.removeVideoAudio").is_empty()); + d.snapshot("multicam-merge-dialog"); + d.click("merge.ok"); + let msg = d.app().ui.sync_dialog.as_ref().map(|x| x.message.clone()).unwrap_or_default(); + assert!(msg.contains("one video clip"), "{msg}"); + d.click("merge.cancel"); + assert!(d.app().ui.sync_dialog.is_none()); +} diff --git a/docs/architecture.md b/docs/architecture.md index d54484e7..411fcbe8 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -384,6 +384,47 @@ frame (Y'CbCr/RGB + metadata) ─► source colour space: Interpret Footage over conversions and looks). `filmcraft-gpu::GpuLut` is the WGSL tetrahedral counterpart, tested for parity. +### 5.4 Multi-camera and synchronisation + +```text +clips ──sync (in | out | timecode[±hours] | marker | audio)──► anchors (media time ↔ common instant) + ├─ clip.synchronize moves selected timeline clips (link groups together) onto the reference + ├─ clip.mergeClips video + ≤16 audio clips → a merged-clip sequence + └─ clip.createMulticam cameras → a multi-camera source sequence (one video track per angle) +multi-camera clip = nested source + TrackItem::multicam {enabled, angle} + render: only the angle's video track · audio: camera 1 | all | the angle (Switch Audio) +``` + +- **Model** (`project::multicam`). `Sequence::multicam` (`MulticamSource`: cameras with their video + track, audio tracks, name, shown flag and source item; audio mode) marks a multi-camera source; + `Sequence::merged` a merged clip. `TrackItem::multicam` (`MulticamSel`) makes a nested clip a + multi-camera clip; any nest can be one (its video tracks are then the angles, + `Sequence::cameras()`). Editing a multi-camera source into a sequence gives an enabled clip on the + first angle (`Project::make_track_item`). Project schema v7. +- **Sync** (`engine::sync`). Each method reduces a clip to an *anchor* (the media time that lines up + with the common instant). Audio uses `audio_dsp::sync::find_offset`: DC removal, windowed-sinc + decimation to ≤ 8 kHz, GCC-PHAT-β (β = 0.75) via one packed complex FFT for the coarse lag, then + the same at the full rate on the loudest common window (≤ 2.7 s) and parabolic interpolation. + Recordings with different gains, microphones (filtered), 0 dB SNR noise or a strong echo are + aligned to the sample; two 10-minute recordings take ~2.4 s (release). Clips are placed on frame + boundaries with the sub-frame remainder taken from their source In, so video stays on frames + while audio keeps sample accuracy. +- **Render.** `render::item_layer` renders a multi-camera clip's angle track only + (`render_seq_tracks`); `render::audio` mixes the nested source with only the audible tracks + (`Sequence::with_angle_audio`). Nested audio now plays at all (it was skipped when the nest had + no media source) and is limited to the clip's range. `render::multicam::render_grid` renders the + shown angles at the cell scale in parallel (rayon) and tiles them: the Multi-Camera view is one + frame job (`frames::Target::MulticamGrid`), prefetched while playing like the program. +- **Editing** (`edit::multicam`, `engine::multicam`). `multicam.switchAngle` (click an angle, + Ctrl/⌘-click for video only), `multicam.selectCamera1…9` (keys 1–9) and `cutToCamera1…9` + (Ctrl+1–9), Enable/Flatten, Edit Cameras, Audio Follows Video (`EditorState`). Live switching: + playback in the Multi-Camera view runs `multicam.recordStart`; each key/click is a + `multicam.cut` that is applied at once (the program shows it) by re-applying the whole pass to + the project from before the pass (`edit_merged`), so a pass is one undo step; Stop runs + `multicam.recordStop`, which ends the last angle at the stop point. Through edits inside the + recorded range are healed, so pressing the angle already showing adds no edit. Flatten replaces + a clip by the clip(s) its angle shows (outer effects carried over; linked pairs stay linked). + ## 6. Export jobs (`filmcraft-export`) ```text diff --git a/docs/project-files.md b/docs/project-files.md index c5d06fa2..cba4230d 100644 --- a/docs/project-files.md +++ b/docs/project-files.md @@ -8,7 +8,7 @@ writing them is `crates/format` (`filmcraft-format`); the engine's `file.*` comm ```json { "format": "filmcraft.project", - "schema_version": 7, + "schema_version": 8, "generator": "FilmCraft 0.1.0", "project": { "name": "…", "settings": { … }, "root": { … }, "items": { … }, "next_id": 48 } } @@ -31,6 +31,7 @@ writing them is `crates/format` (`filmcraft-format`); the engine's `file.*` comm | 5 | M8.8 | colour management: working space, Interpret Footage colour space, LUT library; no-op step | | 6 | M11.7/M11.9 | media identity fingerprints, ingest settings; no-op step | | 7 | M5.5 | effect masks: `masks` on effect instances with keyframable Bézier `Path` values; no-op step | +| 8 | M12 | multi-camera source sequences and clips, merged clips; no-op step | ### Migrations @@ -58,8 +59,8 @@ on disk is unchanged until you save. The **first save over it** keeps the origin A file with a `schema_version` newer than the build supports is **refused**, not half-read: -> this project was saved by a newer version of FilmCraft (project schema v7); this build reads up to -> v6. Update FilmCraft to open it. +> this project was saved by a newer version of FilmCraft (project schema v9); this build reads up to +> v8. Update FilmCraft to open it. Reading it partially and saving it back would silently drop whatever the newer version added. diff --git a/docs/testing.md b/docs/testing.md index 34216876..52b62782 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -20,6 +20,7 @@ FILMCRAFT_REQUIRE_ORACLES=1 cargo test --workspace # CI: missing ffmpeg fails | Golden images | `crates/golden/tests/golden.rs` | CPU renders vs committed PNGs; GPU vs CPU on the same scenes (§3) | | Engine / command | `crates/engine/src/tests.rs` | run commands on the demo project, assert the sequence, undo/redo, disabled cases | | Render | `crates/render/src/tests.rs` | compositing, opacity, Motion, cross dissolve midpoint, ½-res vs full, GPU plan vs reference, audio mix, audio-effect continuity | +| Multi-camera / sync | `crates/audio-dsp/src/sync.rs`, `crates/engine/src/multicam_tests.rs`, `crates/ui-egui/tests/multicam_ui.rs` | audio offsets within one sample: Rust-generated multi-mic speech (gains −20…+10 dB, SNR 40/25/10/3 dB, zero-phase colouring, fractional delays, a 0 dB-SNR room with a −4 dB echo) and ffmpeg-made camera/recorder files (pink-noise bursts, independent noise; exact lags recovered); timecode (± hours) and marker sync within a frame; Merge Clips; timeline Synchronize keeps frame-aligned starts; multicam structure, angle render (PSNR vs the camera), Switch Audio mix = the angle's track, live switching from a simulated playback clock (cuts, healing, one undo step), Ctrl+N cuts, flatten (frames identical), save/load; `perf_grid_of_four_1080p_angles` (ignored) prints the grid cost | | Mixer | `crates/render/src/mixer_tests.rs`, `crates/engine/src/mixer_tests.rs` | sample-exact fader gain and pan laws, automation at block boundaries (bit-identical however requests are cut), solo/mute/solo-safe, sends and submix routing, latency-compensation alignment, render-vs-playback identity, Touch ramp-back, recorder modes (Latch/Touch/Write), thinning, Audio Gain, transition curves; `perf_24_tracks_3_effects_realtime_factor` prints the realtime factor | | Oracle | `crates/*/tests/*oracle*.rs`, `conformance.rs` | compare with ffmpeg/ffprobe (§2) | | Robustness / fuzz | `*/tests/robustness.rs`, `*/tests/fuzz.rs` | seeded mutation and truncation of real and synthetic files; nothing may panic |