Merge M12 (multicam) into M5.5–7 (masks/presets): schema v7 = masks, v8 = multicam (WIP: CI not yet run)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Brandon Thomas
commit
a9278c81cc
|
|
@ -31,7 +31,7 @@ Status: ✅ done · 🟡 in progress · ⬜ not started. Estimates are remaining
|
|||
| M0 | Skeleton + visual shell | ✅ | Workspace, 20 crates, dock/workspaces, Premiere 26 look, native menus, control channel, MCP, xtask gates (layers, wasm) | — | — |
|
||||
| M1 | Media I/O | ✅ | MP4/MOV demux+mux, WAV, stills, MJPEG, symphonia audio (MP3/FLAC/ALAC/Vorbis), GOP seek + frame cache, import | Media Browser polish | 2 |
|
||||
| M2 | H.264 decoder | ✅ | Own decoder, bit-exact on 37+ streams, 500–600 fps 1080p | — | — |
|
||||
| M3 | Editing core | 🟡 | Edit algebra (insert/overwrite/razor/lift/extract/ripple/roll/slip/slide/rate-stretch/nest/paste), tools, markers, trim mode + Trim Monitor + dynamic J/K/L trimming, Keyboard Shortcuts editor with FilmCraft/Premiere/FCP/Avid presets | Multicam, subclip polish, ~220 Premiere default shortcuts whose commands don't exist yet | 6–10 |
|
||||
| M3 | Editing core | 🟡 | Edit algebra (insert/overwrite/razor/lift/extract/ripple/roll/slip/slide/rate-stretch/nest/paste), tools, markers, trim mode + Trim Monitor + dynamic J/K/L trimming, Keyboard Shortcuts editor with FilmCraft/Premiere/FCP/Avid presets; multicam (Create Multi-Camera Source Sequence, Multi-Camera view with live switching on 1–9, angle switching, Enable/Flatten, Edit Cameras) and sync (Synchronize, Merge Clips; In/Out/timecode/marker/audio — GCC-PHAT, sample-accurate) | Multicam paging >16 angles and grid thumbnails, subclip polish, ~220 Premiere default shortcuts whose commands don't exist yet | 6–10 |
|
||||
| M4 | Playback | 🟡 | Audio-clock master, prefetch, J/K/L, dropped-frame stats, playback resolution | Render previews, render bar, 4K/8K tuning | 8–12 |
|
||||
| M5 | Effects, keyframes, GPU | 🟡 | ~60 CPU effects, 30 transitions, keyframes + value/velocity graphs, wgpu compositor | Full ~150-effect catalogue, WGSL parity for all, masks + tracking, adjustment layers, presets, Warp Stabilizer, Morph Cut | 30–45 |
|
||||
| M6 | Export | ✅ | Own H.264 encoder (High/Main/Baseline, B-frames, VBR/CBR/2-pass) → MP4 + own AAC; ProRes, MJPEG, PNG, GIF, WAV; background jobs | Preset library, queue UI, smart render | 6–8 |
|
||||
|
|
@ -48,6 +48,7 @@ Status: ✅ done · 🟡 in progress · ⬜ not started. Estimates are remaining
|
|||
|
||||
## Log
|
||||
|
||||
- **2026-10-01 (M12 multicam):** Synchronize / Merge Clips / Create Multi-Camera Source Sequence (sync by In, Out, timecode ± hours, clip markers, or audio: GCC-PHAT cross-correlation, offsets recovered to the sample on noisy multi-mic recordings), multi-camera clips (angle render, Switch Audio), Program monitor Multi-Camera view with live switching (keys 1–9, one undo step per pass), Ctrl+1–9 cuts, Enable / Flatten / Edit Cameras, project schema v7. Fixed: nested sequence audio was silent.
|
||||
- **2026-10-01 (evening):** M11 done: offline media with our own slate and the Link Media dialog (fingerprint-checked relink, folder remap, search), proxies (create in background / attach / toggle; export stays full-res) with ingest settings, Project Manager (collect, consolidate + transcode). Fixed: GOP cache deadlock when a rayon decoder inside a parallel export re-entered the same source.
|
||||
- **2026-10-01 (M7.4):** Essential Sound panel: Dialogue/Music/SFX/Ambience types, Loudness auto-match (BS.1770, exact to the target), Repair (noise, rumble, hum, new DeEsser and spectral DeReverb), Clarity (dynamics, EQ presets, Enhance Speech DSP chain), Creative reverb / stereo width, ducking that writes Volume keyframes, presets; all `essentialSound.*` commands, effects visible and keyframable in Effect Controls.
|
||||
- **2026-10-01 (night):** M8 colour: camera log curves and gamuts from published specs, colour-managed pipeline (PQ/HLG working spaces, Interpret Footage, BT.2390 tone mapping), HDR export signalling verified with ffprobe, LUTs (.cube/.3dl, tetrahedral CPU + WGSL, project library, Lumetri Input LUT / Look), Colour Match, Lumetri section bypass, HDR scopes.
|
||||
|
|
|
|||
|
|
@ -89,6 +89,22 @@ Pure functions for Essential Sound auto-ducking:
|
|||
merge; clamped to the clip (a ramp cut by an edge gets the ramp's value at the edge); strictly
|
||||
increasing times.
|
||||
|
||||
## Audio synchronisation (`sync`)
|
||||
|
||||
`find_offset(a, b, sample_rate, &SyncOptions)` returns the lag `L` with `b[n] ≈ g·a[n + L]`, a
|
||||
sub-sample estimate, and two trust figures (`confidence` = peak / RMS of the correlation,
|
||||
`distinct` = peak / next peak > 50 ms away; `reliable()` = ≥ 8 and ≥ 1.5).
|
||||
|
||||
- Coarse: DC removal + normalisation, Blackman-windowed-sinc decimation to ≤ 8 kHz (further for
|
||||
long recordings, so the transform stays ≤ 4 M points), GCC-PHAT-β (`R = A·B*/|A·B*|^0.75`,
|
||||
40 Hz – 0.4·rate band) with both signals packed into one complex FFT.
|
||||
- Refine: the same correlation at the full rate on the loudest common window (≤ 2¹⁷ samples), ±2
|
||||
coarse steps around the coarse peak, parabolic interpolation.
|
||||
- Tested within one sample on speech-like multi-microphone signals (gains −20…+10 dB, SNR down to
|
||||
3 dB, coloured microphones, fractional delays: worst 0.4 samples integer / 0.11 samples
|
||||
sub-sample), a 0 dB SNR room with a strong echo, inverted polarity, unrelated signals
|
||||
(unreliable) and `max_lag`. Two 10-minute recordings: 2.4 s (release, one core).
|
||||
|
||||
## Known limits
|
||||
|
||||
- Effects assume ≤ the channel count they were built with; extra channels pass through
|
||||
|
|
|
|||
|
|
@ -6,6 +6,7 @@
|
|||
//! registry ([`effects()`]) describing every effect's parameters (range, unit, default) so the
|
||||
//! engine and UI can build controls generically.
|
||||
//! * [`biquad`] — RBJ-cookbook biquads (TDF-II) with analytic magnitude response.
|
||||
//! * [`sync`] — offset between two recordings of one event (GCC-PHAT, sample-accurate).
|
||||
//!
|
||||
//! Conventions: audio is **planar f32** (`&mut [&mut [f32]]`, one slice per channel, all the same
|
||||
//! length). `process` never allocates; all buffers are sized at construction. Parameters may be
|
||||
|
|
@ -21,6 +22,7 @@ pub mod fft;
|
|||
pub mod loudness;
|
||||
pub mod oversample;
|
||||
mod smooth;
|
||||
pub mod sync;
|
||||
|
||||
pub use effects::{Category, EffectInfo, create_effect, effect_info, effects};
|
||||
pub use loudness::{LoudnessMeter, normalize_gain_db};
|
||||
|
|
|
|||
|
|
@ -0,0 +1,505 @@
|
|||
//! Synchronising recordings of the same event by their audio (Synchronize, Merge Clips and
|
||||
//! Create Multi-Camera Source Sequence "Audio").
|
||||
//!
|
||||
//! [`find_offset`] returns the lag `L` (in samples) for which `b[n] ≈ g · a[n + L]`: when `b`
|
||||
//! started recording two seconds after `a`, `L = 2 · sample_rate`. It works in two passes:
|
||||
//!
|
||||
//! 1. **Coarse.** Both signals are DC-removed, normalised, low-pass filtered (windowed sinc) and
|
||||
//! decimated to ≤ 8 kHz (further for very long recordings, so the transform stays ≤ 4 M points).
|
||||
//! The generalised cross-correlation with partial phase transform weighting (GCC-PHAT-β:
|
||||
//! `R = A·B* / |A·B*|^β`) is computed with one complex FFT of both signals packed as `a + i·b`.
|
||||
//! PHAT weighting whitens the spectra, so the peak is sharp and independent of the microphones'
|
||||
//! gains and frequency responses; β < 1 keeps noise-only bins from dominating.
|
||||
//! 2. **Refine.** Around the coarse peak, the same correlation is computed at the full sample rate
|
||||
//! on the loudest common window (≤ 2.7 s at 48 kHz), and the peak is refined by parabolic
|
||||
//! interpolation to a fraction of a sample.
|
||||
//!
|
||||
//! Two figures say how much to trust the result: `confidence`, the coarse peak's height over the
|
||||
//! RMS of the correlation, and `distinct`, its height over the highest peak more than 50 ms away
|
||||
//! (periodic material such as music or a steady hum can match at several lags).
|
||||
//!
|
||||
//! References: Knapp & Carter, "The generalized correlation method for estimation of time delay",
|
||||
//! IEEE Trans. ASSP 24(4), 1976; the β-weighted PHAT variant is standard textbook practice.
|
||||
|
||||
use crate::fft::Fft;
|
||||
|
||||
/// Tuning of [`find_offset`].
|
||||
#[derive(Clone, Copy, Debug, PartialEq)]
|
||||
pub struct SyncOptions {
|
||||
/// Largest |lag| considered, in samples (None = anything with enough overlap).
|
||||
pub max_lag: Option<i64>,
|
||||
/// Upper bound of the coarse search's sample rate (Hz).
|
||||
pub coarse_rate: u32,
|
||||
/// Most samples (both signals together) the coarse transform may hold; longer recordings are
|
||||
/// decimated further.
|
||||
pub max_coarse_samples: usize,
|
||||
/// PHAT weighting exponent (1 = full whitening, 0 = plain cross-correlation).
|
||||
pub phat_beta: f32,
|
||||
/// Length of the full-rate refinement window (samples).
|
||||
pub refine_window: usize,
|
||||
}
|
||||
|
||||
impl Default for SyncOptions {
|
||||
fn default() -> Self {
|
||||
Self { max_lag: None, coarse_rate: 8000, max_coarse_samples: 1 << 22, phat_beta: 0.75, refine_window: 1 << 17 }
|
||||
}
|
||||
}
|
||||
|
||||
/// The offset found between two recordings.
|
||||
#[derive(Clone, Copy, Debug, PartialEq)]
|
||||
pub struct SyncResult {
|
||||
/// `b[n]` lines up with `a[n + lag]` (samples at the input rate).
|
||||
pub lag: i64,
|
||||
/// The same with sub-sample precision (parabolic interpolation of the refined peak).
|
||||
pub lag_fine: f64,
|
||||
/// Coarse peak height over the RMS of the correlation.
|
||||
pub confidence: f32,
|
||||
/// Coarse peak height over the next-highest peak more than 50 ms away.
|
||||
pub distinct: f32,
|
||||
/// True when the signals correlate negatively (one recording has inverted polarity).
|
||||
pub inverted: bool,
|
||||
}
|
||||
|
||||
impl SyncResult {
|
||||
/// Whether the match is clear enough to trust.
|
||||
pub fn reliable(&self) -> bool {
|
||||
self.confidence >= 8.0 && self.distinct >= 1.5
|
||||
}
|
||||
}
|
||||
|
||||
/// Average interleaved or planar channels into one mono signal.
|
||||
pub fn mixdown(channels: &[&[f32]]) -> Vec<f32> {
|
||||
let n = channels.iter().map(|c| c.len()).min().unwrap_or(0);
|
||||
let k = channels.len().max(1) as f32;
|
||||
(0..n).map(|i| channels.iter().map(|c| c[i]).sum::<f32>() / k).collect()
|
||||
}
|
||||
|
||||
/// Low-pass filter (Blackman-windowed sinc, cutoff at 0.42 × the new sample rate) and keep every
|
||||
/// `d`-th sample.
|
||||
pub fn decimate(x: &[f32], d: usize) -> Vec<f32> {
|
||||
if d <= 1 {
|
||||
return x.to_vec();
|
||||
}
|
||||
let half = 8 * d;
|
||||
let fc = 0.42 / d as f64; // cycles per input sample
|
||||
let taps: Vec<f32> = {
|
||||
let n = 2 * half + 1;
|
||||
let mut h: Vec<f64> = (0..n)
|
||||
.map(|k| {
|
||||
let t = k as f64 - half as f64;
|
||||
let sinc = if t == 0.0 { 2.0 * fc } else { (2.0 * std::f64::consts::PI * fc * t).sin() / (std::f64::consts::PI * t) };
|
||||
let w = 0.42 - 0.5 * (2.0 * std::f64::consts::PI * k as f64 / (n - 1) as f64).cos()
|
||||
+ 0.08 * (4.0 * std::f64::consts::PI * k as f64 / (n - 1) as f64).cos();
|
||||
sinc * w
|
||||
})
|
||||
.collect();
|
||||
let s: f64 = h.iter().sum();
|
||||
h.iter_mut().for_each(|v| *v /= s);
|
||||
h.into_iter().map(|v| v as f32).collect()
|
||||
};
|
||||
let out_len = x.len().div_ceil(d);
|
||||
let mut out = Vec::with_capacity(out_len);
|
||||
for i in 0..out_len {
|
||||
let c = (i * d) as isize;
|
||||
let lo = (c - half as isize).max(0) as usize;
|
||||
let hi = ((c + half as isize + 1) as usize).min(x.len());
|
||||
let k0 = (lo as isize - (c - half as isize)) as usize;
|
||||
let mut acc = 0.0f32;
|
||||
for (j, &v) in x[lo..hi].iter().enumerate() {
|
||||
acc += v * taps[k0 + j];
|
||||
}
|
||||
out.push(acc);
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// Remove the mean and scale to unit RMS (silence stays silence).
|
||||
fn normalise(x: &mut [f32]) {
|
||||
if x.is_empty() {
|
||||
return;
|
||||
}
|
||||
let mean = (x.iter().map(|&v| v as f64).sum::<f64>() / x.len() as f64) as f32;
|
||||
x.iter_mut().for_each(|v| *v -= mean);
|
||||
let rms = (x.iter().map(|&v| (v as f64) * (v as f64)).sum::<f64>() / x.len() as f64).sqrt();
|
||||
if rms > 1e-12 {
|
||||
let k = (1.0 / rms) as f32;
|
||||
x.iter_mut().for_each(|v| *v *= k);
|
||||
}
|
||||
}
|
||||
|
||||
/// GCC-PHAT-β of `a` and `b` (zero-padded to `n`, a power of two ≥ `a.len() + b.len()`):
|
||||
/// `r[L mod n] = Σ a[i + L]·b[i]` with whitened spectra. Only bins whose frequency (cycles per
|
||||
/// sample) lies in `band` contribute.
|
||||
fn gcc(a: &[f32], b: &[f32], n: usize, beta: f32, band: (f32, f32)) -> Vec<f32> {
|
||||
let fft = Fft::new(n);
|
||||
let mut re = vec![0.0f32; n];
|
||||
let mut im = vec![0.0f32; n];
|
||||
re[..a.len()].copy_from_slice(a);
|
||||
im[..b.len()].copy_from_slice(b);
|
||||
fft.forward(&mut re, &mut im);
|
||||
// X = A + iB for real a, b: A(k) = (X(k) + X*(n-k)) / 2, B(k) = (X(k) - X*(n-k)) / 2i.
|
||||
let weigh = |k: usize, xr: f32, xi: f32, yr: f32, yi: f32| -> (f32, f32) {
|
||||
// y = X*(n-k) given as (yr, yi) already conjugated
|
||||
let (ar, ai) = (0.5 * (xr + yr), 0.5 * (xi + yi));
|
||||
let (dr, di) = (xr - yr, xi - yi);
|
||||
let (br, bi) = (0.5 * di, -0.5 * dr);
|
||||
// R = A·conj(B)
|
||||
let rr = ar * br + ai * bi;
|
||||
let ri = ai * br - ar * bi;
|
||||
let f = k.min(n - k) as f32 / n as f32;
|
||||
if f < band.0 || f > band.1 {
|
||||
return (0.0, 0.0);
|
||||
}
|
||||
let mag = (rr * rr + ri * ri).sqrt();
|
||||
if mag <= 1e-30 {
|
||||
return (0.0, 0.0);
|
||||
}
|
||||
let w = mag.powf(-beta);
|
||||
(rr * w, ri * w)
|
||||
};
|
||||
for k in 0..=n / 2 {
|
||||
let j = (n - k) % n;
|
||||
let (xkr, xki, xjr, xji) = (re[k], im[k], re[j], im[j]);
|
||||
let rk = weigh(k, xkr, xki, xjr, -xji);
|
||||
let rj = weigh(j, xjr, xji, xkr, -xki);
|
||||
re[k] = rk.0;
|
||||
im[k] = rk.1;
|
||||
re[j] = rj.0;
|
||||
im[j] = rj.1;
|
||||
}
|
||||
fft.inverse(&mut re, &mut im);
|
||||
re
|
||||
}
|
||||
|
||||
fn corr_at(r: &[f32], lag: i64) -> f32 {
|
||||
let n = r.len() as i64;
|
||||
r[lag.rem_euclid(n) as usize]
|
||||
}
|
||||
|
||||
/// Find the lag `L` with `b[n] ≈ g·a[n + L]` (see the module docs). `None` when either signal is
|
||||
/// too short or silent, or no lag leaves enough overlap.
|
||||
pub fn find_offset(a: &[f32], b: &[f32], sample_rate: u32, opts: &SyncOptions) -> Option<SyncResult> {
|
||||
let sr = sample_rate.max(1) as usize;
|
||||
let min_len = (sr / 10).max(64);
|
||||
if a.len() < min_len || b.len() < min_len {
|
||||
return None;
|
||||
}
|
||||
let energy = |x: &[f32]| x.iter().map(|&v| (v as f64) * (v as f64)).sum::<f64>();
|
||||
if energy(a) < 1e-12 || energy(b) < 1e-12 {
|
||||
return None;
|
||||
}
|
||||
// ---- coarse
|
||||
let d = (sr.div_ceil(opts.coarse_rate.max(500) as usize)).max((a.len() + b.len()).div_ceil(opts.max_coarse_samples.max(1 << 12))).max(1);
|
||||
let mut ca = a.to_vec();
|
||||
let mut cb = b.to_vec();
|
||||
normalise(&mut ca);
|
||||
normalise(&mut cb);
|
||||
let (ca, cb) = (decimate(&ca, d), decimate(&cb, d));
|
||||
let (la, lb) = (ca.len() as i64, cb.len() as i64);
|
||||
let n = ((la + lb) as usize).next_power_of_two();
|
||||
let rate_c = sr as f32 / d as f32;
|
||||
let band = ((40.0 / rate_c).min(0.05), if d > 1 { 0.40 } else { 0.5 });
|
||||
let r = gcc(&ca, &cb, n, opts.phat_beta, band);
|
||||
// valid lags: b[0..lb) against a[L..L+lb) must overlap by a minimum amount
|
||||
let min_overlap = ((rate_c as i64) / 2).min(la.min(lb) / 4).max(8);
|
||||
let max_lag_c = opts.max_lag.map(|m| m.abs() / d as i64 + 1);
|
||||
let lo = -(lb - min_overlap);
|
||||
let hi = la - min_overlap;
|
||||
if hi < lo {
|
||||
return None;
|
||||
}
|
||||
let mut best = (0i64, 0.0f32);
|
||||
let mut sum2 = 0.0f64;
|
||||
let mut count = 0usize;
|
||||
for l in lo..=hi {
|
||||
if max_lag_c.is_some_and(|m| l.abs() > m) {
|
||||
continue;
|
||||
}
|
||||
let v = corr_at(&r, l);
|
||||
sum2 += (v as f64) * (v as f64);
|
||||
count += 1;
|
||||
if v.abs() > best.1.abs() {
|
||||
best = (l, v);
|
||||
}
|
||||
}
|
||||
if count == 0 || best.1 == 0.0 {
|
||||
return None;
|
||||
}
|
||||
let rms = (sum2 / count as f64).sqrt().max(1e-30) as f32;
|
||||
let confidence = best.1.abs() / rms;
|
||||
let guard = ((rate_c * 0.05) as i64).max(2);
|
||||
let mut second = 0.0f32;
|
||||
for l in lo..=hi {
|
||||
if (l - best.0).abs() <= guard || max_lag_c.is_some_and(|m| l.abs() > m) {
|
||||
continue;
|
||||
}
|
||||
second = second.max(corr_at(&r, l).abs());
|
||||
}
|
||||
let distinct = if second > 0.0 { best.1.abs() / second } else { f32::INFINITY };
|
||||
let inverted = best.1 < 0.0;
|
||||
let coarse = best.0 * d as i64;
|
||||
// ---- refine at the full rate
|
||||
let margin = (2 * d + 4) as i64;
|
||||
let (fa, fb) = (a.len() as i64, b.len() as i64);
|
||||
let ov0 = (-coarse).max(0);
|
||||
let ov1 = fb.min(fa - coarse);
|
||||
if ov1 - ov0 < 16 {
|
||||
return Some(SyncResult { lag: coarse, lag_fine: coarse as f64, confidence, distinct, inverted });
|
||||
}
|
||||
let w = ((ov1 - ov0) as usize).min(opts.refine_window.max(256)) as i64;
|
||||
// the window of b with the most energy (both sides: a lag-shifted)
|
||||
let start = {
|
||||
let hop = (w / 4).max(1);
|
||||
let pre = |x: &[f32]| {
|
||||
let mut p = Vec::with_capacity(x.len() + 1);
|
||||
p.push(0.0f64);
|
||||
let mut s = 0.0;
|
||||
for &v in x {
|
||||
s += (v as f64) * (v as f64);
|
||||
p.push(s);
|
||||
}
|
||||
p
|
||||
};
|
||||
let (pa, pb) = (pre(a), pre(b));
|
||||
let mut s = ov0;
|
||||
let mut best_s = ov0;
|
||||
let mut best_e = -1.0f64;
|
||||
while s + w <= ov1 {
|
||||
let eb = pb[(s + w) as usize] - pb[s as usize];
|
||||
let ea = pa[(s + coarse + w) as usize] - pa[(s + coarse) as usize];
|
||||
let e = eb.min(ea);
|
||||
if e > best_e {
|
||||
best_e = e;
|
||||
best_s = s;
|
||||
}
|
||||
s += hop;
|
||||
}
|
||||
best_s
|
||||
};
|
||||
let bw: Vec<f32> = b[start as usize..(start + w) as usize].to_vec();
|
||||
let aw: Vec<f32> = (start + coarse - margin..start + coarse + w + margin).map(|i| if i >= 0 && i < fa { a[i as usize] } else { 0.0 }).collect();
|
||||
let mut aw = aw;
|
||||
let mut bw = bw;
|
||||
normalise(&mut aw);
|
||||
normalise(&mut bw);
|
||||
let n2 = (aw.len() + bw.len()).next_power_of_two();
|
||||
let r2 = gcc(&aw, &bw, n2, opts.phat_beta, (20.0 / sr as f32, 0.5));
|
||||
// r2[L'] = Σ aw[i + L']·bw[i]; L' = margin + δ
|
||||
let sign = if inverted { -1.0 } else { 1.0 };
|
||||
let mut bl = 0i64;
|
||||
let mut bv = f32::MIN;
|
||||
for l in 0..=2 * margin {
|
||||
let v = corr_at(&r2, l) * sign;
|
||||
if v > bv {
|
||||
bv = v;
|
||||
bl = l;
|
||||
}
|
||||
}
|
||||
let (y0, y1, y2) = (corr_at(&r2, bl - 1) * sign, bv, corr_at(&r2, bl + 1) * sign);
|
||||
let den = y0 - 2.0 * y1 + y2;
|
||||
let frac = if den.abs() > 1e-20 { (0.5 * (y0 - y2) / den).clamp(-0.5, 0.5) } else { 0.0 };
|
||||
let delta = bl - margin;
|
||||
let lag = coarse + delta;
|
||||
Some(SyncResult { lag, lag_fine: lag as f64 + frac as f64, confidence, distinct, inverted })
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::testutil::Rng;
|
||||
|
||||
/// Speech-like test signal: voiced syllables (harmonics of a gliding pitch under an envelope),
|
||||
/// noise bursts (consonants) and pauses.
|
||||
fn speechy(sr: f64, secs: f64, seed: u64) -> Vec<f32> {
|
||||
let n = (sr * secs) as usize;
|
||||
let mut rng = Rng::new(seed);
|
||||
let mut out = vec![0.0f32; n];
|
||||
let mut i = 0usize;
|
||||
while i < n {
|
||||
let len = ((0.08 + 0.25 * (rng.uniform() * 0.5 + 0.5) as f64) * sr) as usize;
|
||||
let kind = rng.uniform();
|
||||
let f0 = 110.0 + 120.0 * (rng.uniform() as f64 * 0.5 + 0.5);
|
||||
let glide = 1.0 + 0.3 * rng.uniform() as f64;
|
||||
let amp = 0.2 + 0.3 * (rng.uniform() * 0.5 + 0.5);
|
||||
for k in 0..len.min(n - i) {
|
||||
let t = k as f64 / sr;
|
||||
let env = (std::f64::consts::PI * k as f64 / len as f64).sin() as f32;
|
||||
let v = if kind > 0.3 {
|
||||
let f = f0 * (1.0 + (glide - 1.0) * k as f64 / len as f64);
|
||||
(1..8).map(|h| ((2.0 * std::f64::consts::PI * f * h as f64 * t).sin() / h as f64) as f32).sum::<f32>()
|
||||
} else if kind > -0.4 {
|
||||
rng.uniform()
|
||||
} else {
|
||||
0.0
|
||||
};
|
||||
out[i + k] = v * env * amp;
|
||||
}
|
||||
i += len + ((rng.uniform() * 0.5 + 0.5) * 0.15 * sr as f32) as usize;
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// `x` delayed by `delay` samples (fractional: windowed-sinc interpolation), scaled by `gain`,
|
||||
/// optionally coloured by a zero-phase low-pass (`[k, 1-2k, k]`, so the delay stays exact), with
|
||||
/// white noise `snr_db` below the signal.
|
||||
fn mic(x: &[f32], delay: f64, gain: f32, lowpass: Option<f32>, snr_db: f32, len: usize, seed: u64) -> Vec<f32> {
|
||||
let mut out = vec![0.0f32; len];
|
||||
let di = delay.floor() as i64;
|
||||
let fr = delay - di as f64;
|
||||
for (n, o) in out.iter_mut().enumerate() {
|
||||
let src = n as i64 - di;
|
||||
if fr == 0.0 {
|
||||
if src >= 0 && (src as usize) < x.len() {
|
||||
*o = x[src as usize];
|
||||
}
|
||||
} else {
|
||||
let mut acc = 0.0f64;
|
||||
for k in -16i64..=16 {
|
||||
let j = src - k;
|
||||
if j < 0 || j as usize >= x.len() {
|
||||
continue;
|
||||
}
|
||||
let t = k as f64 - fr;
|
||||
let s = (std::f64::consts::PI * t).sin() / (std::f64::consts::PI * t);
|
||||
let w = 0.5 + 0.5 * (std::f64::consts::PI * t / 17.0).cos();
|
||||
acc += x[j as usize] as f64 * s * w;
|
||||
}
|
||||
*o = acc as f32;
|
||||
}
|
||||
}
|
||||
if let Some(k) = lowpass {
|
||||
let src = out.clone();
|
||||
for i in 1..len.saturating_sub(1) {
|
||||
out[i] = k * src[i - 1] + (1.0 - 2.0 * k) * src[i] + k * src[i + 1];
|
||||
}
|
||||
}
|
||||
let rms = crate::testutil::rms(&out) as f32;
|
||||
let nrms = rms * 10f32.powf(-snr_db / 20.0) * 3f32.sqrt();
|
||||
let mut rng = Rng::new(seed);
|
||||
for v in &mut out {
|
||||
*v = *v * gain + rng.uniform() * nrms * gain;
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn decimate_keeps_low_frequencies_and_removes_high_ones() {
|
||||
let sr = 48_000.0;
|
||||
let low = crate::testutil::sine(300.0, 1.0, sr, 48_000, 0.0);
|
||||
let high = crate::testutil::sine(7_000.0, 1.0, sr, 48_000, 0.0);
|
||||
let dl = decimate(&low, 6);
|
||||
let dh = decimate(&high, 6);
|
||||
assert_eq!(dl.len(), 8000);
|
||||
let rl = crate::testutil::rms(&dl[100..7900]);
|
||||
let rh = crate::testutil::rms(&dh[100..7900]);
|
||||
assert!((rl - std::f64::consts::FRAC_1_SQRT_2).abs() < 0.01, "{rl}");
|
||||
assert!(rh < 0.01, "{rh}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn pure_delay_exact_both_signs() {
|
||||
let sr = 48_000;
|
||||
let s = speechy(sr as f64, 12.0, 1);
|
||||
for delay in [0i64, 1, 7, 12_345, 96_001, -4_321, -50_000] {
|
||||
// b = s shifted: b[n] = s[n + delay]
|
||||
let b: Vec<f32> = (0..400_000i64).map(|n| if n + delay >= 0 && ((n + delay) as usize) < s.len() { s[(n + delay) as usize] } else { 0.0 }).collect();
|
||||
let r = find_offset(&s, &b, sr, &SyncOptions::default()).unwrap();
|
||||
assert_eq!(r.lag, delay, "{r:?}");
|
||||
assert!(r.reliable(), "{r:?}");
|
||||
assert!(!r.inverted);
|
||||
}
|
||||
}
|
||||
|
||||
/// Three microphones recording one talker: different gains (−20…+10 dB), noise (SNR 25, 10 and
|
||||
/// 3 dB), one coloured by a low-pass, fractional delays. Every offset within one sample.
|
||||
#[test]
|
||||
fn multi_mic_offsets_within_one_sample() {
|
||||
let sr = 48_000u32;
|
||||
let src = speechy(sr as f64, 30.0, 7);
|
||||
let len = 30 * sr as usize;
|
||||
// mic k records src delayed by d_k: mic_k[n] = src[n - d_k]
|
||||
let cases: [(f64, f32, Option<f32>, f32); 4] =
|
||||
[(0.0, 1.0, None, 40.0), (2_345.0, 0.1, None, 25.0), (-31_337.4, 3.0, Some(0.3), 10.0), (150_000.6, 0.5, None, 3.0)];
|
||||
let mics: Vec<Vec<f32>> = cases.iter().enumerate().map(|(i, c)| mic(&src, c.0, c.1, c.2, c.3, len, 100 + i as u64)).collect();
|
||||
let mut worst = 0.0f64;
|
||||
let mut worst_fine = 0.0f64;
|
||||
for (k, c) in cases.iter().enumerate().skip(1) {
|
||||
// b = mic_k: b[n] = src[n - d_k] = a[n - d_k + d_0] → lag = d_0 - d_k
|
||||
let expect = cases[0].0 - c.0;
|
||||
let r = find_offset(&mics[0], &mics[k], sr, &SyncOptions::default()).unwrap();
|
||||
let err = (r.lag as f64 - expect).abs();
|
||||
let err_fine = (r.lag_fine - expect).abs();
|
||||
worst = worst.max(err);
|
||||
worst_fine = worst_fine.max(err_fine);
|
||||
assert!(err <= 1.0, "mic {k}: lag {} vs {expect} ({r:?})", r.lag);
|
||||
assert!(err_fine < 0.75, "mic {k}: fine {} vs {expect}", r.lag_fine);
|
||||
assert!(r.reliable(), "{r:?}");
|
||||
}
|
||||
eprintln!("multi-mic worst error: {worst} samples (integer lag), {worst_fine:.3} samples (sub-sample estimate)");
|
||||
}
|
||||
|
||||
/// A room: the far microphone hears the direct sound plus a strong reflection 23 ms later, at
|
||||
/// 0 dB SNR. The direct path wins.
|
||||
#[test]
|
||||
fn echo_and_heavy_noise() {
|
||||
let sr = 48_000u32;
|
||||
let src = speechy(sr as f64, 20.0, 21);
|
||||
let len = src.len();
|
||||
let near = mic(&src, 0.0, 1.0, None, 35.0, len, 1);
|
||||
let direct = mic(&src, 777.0, 1.0, None, 60.0, len, 2);
|
||||
let echo = mic(&src, 777.0 + 0.023 * sr as f64, 0.6, None, 60.0, len, 3);
|
||||
let mut rng = Rng::new(9);
|
||||
let room: Vec<f32> = direct.iter().zip(&echo).map(|(a, b)| a + b).collect();
|
||||
let nrms = crate::testutil::rms(&room) as f32 * 3f32.sqrt();
|
||||
let far: Vec<f32> = room.iter().map(|v| (v + rng.uniform() * nrms) * 0.3).collect();
|
||||
let r = find_offset(&near, &far, sr, &SyncOptions::default()).unwrap();
|
||||
assert!((r.lag + 777).abs() <= 1, "{r:?}");
|
||||
assert!(r.reliable(), "{r:?}");
|
||||
}
|
||||
|
||||
/// Two 10-minute recordings (run with `--ignored --nocapture` in release for the timing).
|
||||
#[test]
|
||||
#[ignore]
|
||||
fn long_recordings_timing() {
|
||||
let sr = 48_000u32;
|
||||
let src = speechy(sr as f64, 600.0, 31);
|
||||
let a = mic(&src, 0.0, 1.0, None, 20.0, src.len(), 1);
|
||||
let b = mic(&src, -1_234_567.0, 0.5, None, 10.0, src.len() - 2_000_000, 2);
|
||||
let t = std::time::Instant::now();
|
||||
let r = find_offset(&a, &b, sr, &SyncOptions::default()).unwrap();
|
||||
eprintln!("10 min + 9.3 min at 48 kHz: {:?} in {:.2} s", r, t.elapsed().as_secs_f64());
|
||||
assert!((r.lag - 1_234_567).abs() <= 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn inverted_polarity_is_found() {
|
||||
let sr = 48_000;
|
||||
let s = speechy(sr as f64, 8.0, 3);
|
||||
let b: Vec<f32> = (0..300_000usize).map(|n| if n + 1000 < s.len() { -0.5 * s[n + 1000] } else { 0.0 }).collect();
|
||||
let r = find_offset(&s, &b, sr, &SyncOptions::default()).unwrap();
|
||||
assert_eq!(r.lag, 1000);
|
||||
assert!(r.inverted);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn unrelated_signals_are_not_reliable() {
|
||||
let sr = 48_000;
|
||||
let a = speechy(sr as f64, 10.0, 11);
|
||||
let b = speechy(sr as f64, 10.0, 12);
|
||||
let r = find_offset(&a, &b, sr, &SyncOptions::default()).unwrap();
|
||||
assert!(!r.reliable(), "{r:?}");
|
||||
assert!(find_offset(&a, &[0.0; 48_000], sr, &SyncOptions::default()).is_none());
|
||||
assert!(find_offset(&a[..100], &a, sr, &SyncOptions::default()).is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn max_lag_limits_the_search() {
|
||||
let sr = 48_000;
|
||||
let s = speechy(sr as f64, 10.0, 5);
|
||||
let b: Vec<f32> = s[48_000..].to_vec();
|
||||
let r = find_offset(&s, &b, sr, &SyncOptions { max_lag: Some(60_000), ..Default::default() }).unwrap();
|
||||
assert_eq!(r.lag, 48_000);
|
||||
let r = find_offset(&s, &b, sr, &SyncOptions { max_lag: Some(10_000), ..Default::default() }).unwrap();
|
||||
assert!(!r.reliable() || r.lag != 48_000);
|
||||
}
|
||||
}
|
||||
|
|
@ -16,6 +16,7 @@
|
|||
//! Keyframes are stored in media time, so trims and splits never need to move them.
|
||||
|
||||
pub mod captions;
|
||||
pub mod multicam;
|
||||
|
||||
use std::collections::HashMap;
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,213 @@
|
|||
//! Multi-camera editing: switching angles, recording cuts live, healing through edits and
|
||||
//! flattening multi-camera clips into the clips they show.
|
||||
//!
|
||||
//! A multi-camera clip is a track item whose [`TrackItem::multicam`] is set; it refers to a
|
||||
//! multi-camera source sequence. Its angle is a camera index of that source.
|
||||
|
||||
use std::collections::HashMap;
|
||||
|
||||
use filmcraft_project::{ClipId, ItemId, MulticamSel, Sequence, Track, TrackId, TrackItem};
|
||||
use filmcraft_time::{Tick, TimeRange};
|
||||
|
||||
use crate::{EditCtx, EditError, Result, remove_orphan_transitions, split_track_at};
|
||||
|
||||
/// Whether `it` is a multi-camera clip (enabled or not).
|
||||
pub fn is_multicam(it: &TrackItem) -> bool {
|
||||
it.multicam.is_some()
|
||||
}
|
||||
|
||||
/// Set the angle of multi-camera clips. Returns how many changed.
|
||||
pub fn set_angle(seq: &mut Sequence, clips: &[ClipId], angle: u32) -> usize {
|
||||
let mut n = 0;
|
||||
for t in seq.all_tracks_mut() {
|
||||
if t.locked {
|
||||
continue;
|
||||
}
|
||||
for it in t.items.iter_mut().filter(|i| clips.contains(&i.id)) {
|
||||
if let Some(m) = &mut it.multicam
|
||||
&& m.angle != angle
|
||||
{
|
||||
m.angle = angle;
|
||||
n += 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
n
|
||||
}
|
||||
|
||||
/// Enable or disable multi-camera behaviour of clips (Multi-Camera ▸ Enable). Clips that are not
|
||||
/// multi-camera clips are left alone. Returns how many changed.
|
||||
pub fn set_enabled(seq: &mut Sequence, clips: &[ClipId], enabled: bool) -> usize {
|
||||
let mut n = 0;
|
||||
for t in seq.all_tracks_mut() {
|
||||
for it in t.items.iter_mut().filter(|i| clips.contains(&i.id)) {
|
||||
if let Some(m) = &mut it.multicam
|
||||
&& m.enabled != enabled
|
||||
{
|
||||
m.enabled = enabled;
|
||||
n += 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
n
|
||||
}
|
||||
|
||||
/// Two adjacent pieces of one multi-camera clip that a cut between them would not change.
|
||||
fn continuous(a: &TrackItem, b: &TrackItem) -> bool {
|
||||
a.end() == b.start
|
||||
&& a.item == b.item
|
||||
&& a.multicam.is_some()
|
||||
&& a.multicam == b.multicam
|
||||
&& a.speed == b.speed
|
||||
&& !a.reverse
|
||||
&& !b.reverse
|
||||
&& a.frame_hold.is_none()
|
||||
&& b.frame_hold.is_none()
|
||||
&& a.source_out() == b.source_in
|
||||
&& a.enabled == b.enabled
|
||||
&& a.gain_db == b.gain_db
|
||||
&& a.effects == b.effects
|
||||
}
|
||||
|
||||
/// Remove through edits between pieces of the same multi-camera clip showing the same angle,
|
||||
/// for edit points in `range` (inclusive of its ends). Returns the ids of the removed pieces.
|
||||
pub fn heal_track(track: &mut Track, range: TimeRange) -> Vec<ClipId> {
|
||||
let mut removed = Vec::new();
|
||||
let mut i = 0;
|
||||
while i + 1 < track.items.len() {
|
||||
let (a, b) = (&track.items[i], &track.items[i + 1]);
|
||||
let at = b.start;
|
||||
if at >= range.start && at <= range.end() && continuous(a, b) {
|
||||
let (aid, bid) = (a.id, b.id);
|
||||
let dur = b.duration;
|
||||
track.items[i].duration += dur;
|
||||
track.items.remove(i + 1);
|
||||
track.transitions.retain(|tr| !(tr.from == Some(aid) && tr.to == Some(bid)));
|
||||
for tr in &mut track.transitions {
|
||||
if tr.from == Some(bid) {
|
||||
tr.from = Some(aid);
|
||||
}
|
||||
if tr.to == Some(bid) {
|
||||
tr.to = Some(aid);
|
||||
}
|
||||
}
|
||||
removed.push(bid);
|
||||
} else {
|
||||
i += 1;
|
||||
}
|
||||
}
|
||||
remove_orphan_transitions(track);
|
||||
removed
|
||||
}
|
||||
|
||||
/// One live switch: from `time` on, show `angle`.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub struct Cut {
|
||||
pub time: Tick,
|
||||
pub angle: u32,
|
||||
}
|
||||
|
||||
/// Apply a recording pass: on `tracks`, the multi-camera clips of `source` get angle `cuts[i].angle` from
|
||||
/// `cuts[i].time` until the next cut, the last one until `end`. Clips are split at each cut and at
|
||||
/// `end`; afterwards through edits inside the recorded range are healed, so pressing the angle that
|
||||
/// is already showing adds no edit. Locked tracks are skipped. Returns the number of edit points
|
||||
/// in the recorded range after the pass.
|
||||
pub fn record(seq: &mut Sequence, tracks: &[TrackId], source: ItemId, cuts: &[Cut], end: Tick, ctx: &mut EditCtx) -> usize {
|
||||
let mut cuts: Vec<Cut> = cuts.iter().copied().filter(|c| c.time < end).collect();
|
||||
cuts.sort_by_key(|c| c.time);
|
||||
// two presses at one time: the later wins
|
||||
let mut dedup: Vec<Cut> = Vec::new();
|
||||
for c in cuts {
|
||||
match dedup.last_mut() {
|
||||
Some(l) if l.time == c.time => *l = c,
|
||||
_ => dedup.push(c),
|
||||
}
|
||||
}
|
||||
let Some(first) = dedup.first().map(|c| c.time) else { return 0 };
|
||||
let mut links = HashMap::new();
|
||||
let mut points = 0;
|
||||
for tr in seq.all_tracks_mut() {
|
||||
if tr.locked || !tracks.contains(&tr.id) {
|
||||
continue;
|
||||
}
|
||||
let mc_at = |tr: &Track, t: Tick| tr.items.iter().any(|i| i.start < t && t < i.end() && is_multicam(i) && i.item == source);
|
||||
for t in dedup.iter().map(|c| c.time).chain([end]) {
|
||||
if mc_at(tr, t) {
|
||||
split_track_at(tr, t, ctx, &mut links);
|
||||
}
|
||||
}
|
||||
for (k, c) in dedup.iter().enumerate() {
|
||||
let r_end = dedup.get(k + 1).map_or(end, |n| n.time);
|
||||
for it in tr.items.iter_mut().filter(|i| i.start >= c.time && i.end() <= r_end && i.item == source) {
|
||||
if let Some(m) = &mut it.multicam {
|
||||
m.angle = c.angle;
|
||||
}
|
||||
}
|
||||
}
|
||||
heal_track(tr, TimeRange::from_bounds(first, end));
|
||||
points += tr.items.iter().filter(|i| i.start > first && i.start < end).count();
|
||||
}
|
||||
points
|
||||
}
|
||||
|
||||
/// The clips a multi-camera (or any nested) clip shows, as track items of the outer sequence:
|
||||
/// the parts of `source_track`'s items inside the clip's source range, moved to timeline time.
|
||||
/// Inner links are remapped through `links` (so a flattened video/audio pair stays linked).
|
||||
/// Fails for clips with a speed change, reverse or frame hold (no exact flattening).
|
||||
pub fn flatten_items(outer: &TrackItem, source_track: &Track, ctx: &mut EditCtx, links: &mut HashMap<u64, u64>) -> Result<Vec<TrackItem>> {
|
||||
if (outer.speed - 1.0).abs() > 1e-9 || outer.reverse || outer.frame_hold.is_some() {
|
||||
return Err(EditError::Other("can't flatten a clip with a speed change, reverse or frame hold".into()));
|
||||
}
|
||||
let (s0, s1) = (outer.source_in, outer.source_out());
|
||||
let mut out = Vec::new();
|
||||
for ni in source_track.items.iter().filter(|i| i.start < s1 && i.end() > s0) {
|
||||
let mut it = ni.clone();
|
||||
if it.start < s0 {
|
||||
let cut = s0 - it.start;
|
||||
it.source_in += Tick((cut.0 as f64 * it.speed.abs()).round() as i64);
|
||||
it.duration -= cut;
|
||||
it.start = s0;
|
||||
}
|
||||
if it.end() > s1 {
|
||||
it.duration = s1 - it.start;
|
||||
}
|
||||
if it.duration <= Tick::ZERO {
|
||||
continue;
|
||||
}
|
||||
it.start = outer.start + (it.start - s0);
|
||||
it.id = ClipId(ctx.alloc());
|
||||
it.link = it.link.map(|l| *links.entry(l).or_insert_with(|| ctx.alloc()));
|
||||
it.group = None;
|
||||
// the outer clip's look carries over: its standard effects follow the inner ones, and its
|
||||
// Motion/Opacity/Volume replace the inner ones when only the outer clip changed them
|
||||
let take_intrinsics = outer.has_modified_intrinsics() && !it.has_modified_intrinsics();
|
||||
for e in &outer.effects {
|
||||
let intrinsic = e.def().is_some_and(|d| d.intrinsic);
|
||||
if !intrinsic {
|
||||
it.effects.push(e.clone());
|
||||
} else if take_intrinsics && let Some(slot) = it.effects.iter_mut().find(|x| x.effect == e.effect) {
|
||||
*slot = e.clone();
|
||||
}
|
||||
}
|
||||
out.push(it);
|
||||
}
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
/// Replace a multi-camera clip on `track` by `pieces` (from [`flatten_items`]); pieces whose
|
||||
/// track is not `track` go to the tracks given with them. Returns the new ids.
|
||||
pub fn replace_with(seq: &mut Sequence, clip: ClipId, pieces: Vec<(TrackId, TrackItem)>, ctx: &mut EditCtx) -> Result<Vec<ClipId>> {
|
||||
let (tid, _) = seq.find_item(clip).ok_or(EditError::NoItem(clip))?;
|
||||
let t = seq.track_mut(tid).ok_or(EditError::NoTrack(tid))?;
|
||||
if t.locked {
|
||||
return Err(EditError::Locked);
|
||||
}
|
||||
t.items.retain(|i| i.id != clip);
|
||||
remove_orphan_transitions(t);
|
||||
crate::overwrite(seq, pieces, ctx)
|
||||
}
|
||||
|
||||
/// The multi-camera selection a freshly edited multi-camera clip gets.
|
||||
pub fn default_sel(angle: u32) -> MulticamSel {
|
||||
MulticamSel { enabled: true, angle }
|
||||
}
|
||||
|
|
@ -55,6 +55,7 @@ impl Fx {
|
|||
frame_hold: None,
|
||||
scale_to_frame: false,
|
||||
essential: None,
|
||||
multicam: None,
|
||||
}
|
||||
}
|
||||
fn put(&mut self, track: TrackId, start: i64, dur: i64, src_in: i64) -> ClipId {
|
||||
|
|
|
|||
|
|
@ -75,7 +75,7 @@ fn has_recovery(s: &Session) -> std::result::Result<(), String> {
|
|||
pub(crate) fn has_seq(s: &Session) -> std::result::Result<(), String> {
|
||||
s.active_sequence().map(|_| ()).ok_or_else(|| "no sequence is open".into())
|
||||
}
|
||||
fn has_selection(s: &Session) -> std::result::Result<(), String> {
|
||||
pub(crate) fn has_selection(s: &Session) -> std::result::Result<(), String> {
|
||||
has_seq(s)?;
|
||||
if s.state.selection.is_empty() { Err("no clips selected".into()) } else { Ok(()) }
|
||||
}
|
||||
|
|
@ -102,7 +102,7 @@ fn has_previews(s: &Session) -> std::result::Result<(), String> {
|
|||
has_seq(s)?;
|
||||
if s.previews.count() == 0 { Err("there are no render files".into()) } else { Ok(()) }
|
||||
}
|
||||
fn has_project_selection(s: &Session) -> std::result::Result<(), String> {
|
||||
pub(crate) fn has_project_selection(s: &Session) -> std::result::Result<(), String> {
|
||||
if s.state.project_selection.is_empty() { Err("select an item in the Project panel".into()) } else { Ok(()) }
|
||||
}
|
||||
fn has_clipboard(s: &Session) -> std::result::Result<(), String> {
|
||||
|
|
@ -152,13 +152,13 @@ pub(crate) fn time_p(s: &Session, p: &Value, prefix: &str) -> Option<Tick> {
|
|||
pub(crate) fn clip_p(p: &Value, k: &str) -> Option<ClipId> {
|
||||
u64_p(p, k).map(ClipId)
|
||||
}
|
||||
fn clips_p(s: &Session, p: &Value) -> Vec<ClipId> {
|
||||
pub(crate) fn clips_p(s: &Session, p: &Value) -> Vec<ClipId> {
|
||||
match p.get("clips").and_then(Value::as_array) {
|
||||
Some(a) => a.iter().filter_map(|v| v.as_u64().map(ClipId)).collect(),
|
||||
None => clip_p(p, "clip").map(|c| vec![c]).unwrap_or_else(|| s.state.selection.clone()),
|
||||
}
|
||||
}
|
||||
fn track_p(s: &Session, p: &Value, k: &str) -> Option<TrackId> {
|
||||
pub(crate) fn track_p(s: &Session, p: &Value, k: &str) -> Option<TrackId> {
|
||||
let v = p.get(k)?;
|
||||
if let Some(id) = v.as_u64() {
|
||||
return Some(TrackId(id));
|
||||
|
|
@ -192,7 +192,7 @@ pub fn with_links(s: &Session, clips: &[ClipId]) -> Vec<ClipId> {
|
|||
out
|
||||
}
|
||||
|
||||
fn default_seq_settings_for(info: &filmcraft_media::MediaInfo) -> SequenceSettings {
|
||||
pub(crate) fn default_seq_settings_for(info: &filmcraft_media::MediaInfo) -> SequenceSettings {
|
||||
let mut st = SequenceSettings::default();
|
||||
if let Some(v) = &info.video {
|
||||
st.width = v.width;
|
||||
|
|
@ -1876,6 +1876,7 @@ fn build() -> Vec<CommandSpec> {
|
|||
];
|
||||
v.extend(crate::captions::commands());
|
||||
v.extend(crate::mixer::commands());
|
||||
v.extend(crate::multicam::commands());
|
||||
v.extend(crate::essential_sound::commands());
|
||||
v.extend(crate::color::commands());
|
||||
v.extend(crate::graphics::commands());
|
||||
|
|
|
|||
|
|
@ -20,12 +20,14 @@ pub mod masks;
|
|||
pub mod media_pool;
|
||||
pub mod mixer;
|
||||
pub mod presets;
|
||||
pub mod multicam;
|
||||
pub mod previews;
|
||||
pub mod project_manager;
|
||||
pub mod proxies;
|
||||
pub mod relink;
|
||||
pub mod shortcut_presets;
|
||||
pub mod shortcuts;
|
||||
pub mod sync;
|
||||
pub mod trim;
|
||||
|
||||
use std::sync::Arc;
|
||||
|
|
@ -173,6 +175,10 @@ pub struct EditorState {
|
|||
/// The mask being edited on the Program monitor (Effect Controls selection).
|
||||
#[serde(default)]
|
||||
pub selected_mask: Option<masks::MaskSel>,
|
||||
/// Multi-Camera Audio Follows Video: switching a multi-camera clip's angle switches its linked
|
||||
/// audio clips too.
|
||||
#[serde(default)]
|
||||
pub multicam_audio_follows_video: bool,
|
||||
}
|
||||
|
||||
/// Events for frontends (drained each frame).
|
||||
|
|
@ -210,6 +216,8 @@ pub struct Session {
|
|||
pub previews: Arc<previews::PreviewStore>,
|
||||
/// Audio Track Mixer automation pass in progress.
|
||||
pub mixrec: mixer::Recorder,
|
||||
/// Multi-camera live switching pass in progress.
|
||||
pub mcrec: multicam::Recorder,
|
||||
/// Dynamic (J/K/L) trimming and trim-mode loop playback in progress.
|
||||
pub trim_play: trim::TrimPlayback,
|
||||
/// Keyboard shortcuts (active bindings, presets; `shortcuts.*` commands).
|
||||
|
|
@ -284,6 +292,7 @@ impl Session {
|
|||
loaded_schema: filmcraft_format::SCHEMA_VERSION,
|
||||
previews: Arc::new(previews::PreviewStore::temp()),
|
||||
mixrec: Default::default(),
|
||||
mcrec: Default::default(),
|
||||
trim_play: Default::default(),
|
||||
shortcuts: shortcuts::Shortcuts::new(),
|
||||
offline: Default::default(),
|
||||
|
|
@ -649,6 +658,8 @@ mod mixer_tests;
|
|||
#[cfg(test)]
|
||||
mod presets_tests;
|
||||
#[cfg(test)]
|
||||
mod multicam_tests;
|
||||
#[cfg(test)]
|
||||
mod previews_tests;
|
||||
#[cfg(test)]
|
||||
mod project_manager_tests;
|
||||
|
|
|
|||
|
|
@ -0,0 +1,968 @@
|
|||
//! Multi-camera editing and clip synchronisation commands.
|
||||
//!
|
||||
//! | Id | Menu | |
|
||||
//! |---|---|---|
|
||||
//! | `clip.synchronize` | Clip ▸ Synchronize… | align selected timeline clips to a reference by In, Out, timecode, clip marker or audio |
|
||||
//! | `clip.mergeClips` | Clip ▸ Merge Clips… | one video + audio clips → a merged clip |
|
||||
//! | `clip.createMulticam` | Clip ▸ Create Multi-Camera Source Sequence… | synchronised cameras → a multi-camera source sequence |
|
||||
//! | `clip.multicamEnable` | Clip ▸ Multi-Camera ▸ Enable | toggle multi-camera behaviour of nested clips |
|
||||
//! | `clip.multicamFlatten` | Clip ▸ Multi-Camera ▸ Flatten | replace multi-camera clips by the clips their angle shows |
|
||||
//! | `multicam.switchAngle` | | set the angle of clips (keys 1–9 when stopped, clicking an angle) |
|
||||
//! | `multicam.recordStart` / `multicam.cut` / `multicam.recordStop` | | live switching while playing: one undo step per pass |
|
||||
//! | `multicam.selectCamera1`…`9` | keys 1–9 | while recording: cut to camera N; stopped: switch the clip at the playhead |
|
||||
//! | `multicam.cutToCamera1`…`9` | Ctrl+1–9 | add an edit at the playhead and show camera N from there |
|
||||
//! | `multicam.audioFollowsVideo` | | switching video switches the linked audio clips too |
|
||||
//! | `multicam.editCameras` | | rename / show / hide cameras, change the audio mode |
|
||||
//! | `multicam.inspect` | | the multi-camera clip at the playhead, its cameras and the recorder state |
|
||||
//!
|
||||
//! Sync methods are described in [`crate::sync`].
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::sync::Arc;
|
||||
|
||||
use filmcraft_edit as edit;
|
||||
use filmcraft_edit::EditCtx;
|
||||
use filmcraft_edit::multicam::Cut;
|
||||
use filmcraft_project::{
|
||||
BinEntry, Camera, ClipId, ItemId, ItemKind, Label, MergedClip, MulticamAudio, MulticamSel, MulticamSource, Project, Sequence, SequenceSettings, Track,
|
||||
TrackId, TrackItem, TrackKind, resolve_auto_points,
|
||||
};
|
||||
use filmcraft_time::{FrameRate, Tick, TimeRange};
|
||||
use serde_json::{Value, json};
|
||||
|
||||
use crate::commands::{CommandSpec, always, bad, bool_p, clips_p, has_selection, has_seq, item_p, str_p, time_p, track_p, u64_p, with_links};
|
||||
use crate::sync::{self, Method, SyncClip};
|
||||
use crate::{EngineError, MediaPool, Result, Session};
|
||||
|
||||
type Run = fn(&mut Session, &Value) -> Result<Value>;
|
||||
type Enabled = fn(&Session) -> std::result::Result<(), String>;
|
||||
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
fn spec(id: &'static str, label: &'static str, menu: &'static [&'static str], params: &'static str, enabled: Enabled, run: Run, journal: bool) -> CommandSpec {
|
||||
CommandSpec { id, label, menu, shortcut: None, params, enabled, run, journal }
|
||||
}
|
||||
|
||||
macro_rules! camera_keys {
|
||||
($($n:literal),*) => {
|
||||
[$(
|
||||
CommandSpec {
|
||||
id: concat!("multicam.selectCamera", $n),
|
||||
label: concat!("Select Camera ", $n),
|
||||
menu: &[],
|
||||
shortcut: Some(concat!($n)),
|
||||
params: r#"{"videoOnly":bool?}"#,
|
||||
enabled: has_seq,
|
||||
run: |s, p| cut(s, &with_camera(p, $n)),
|
||||
journal: true,
|
||||
},
|
||||
CommandSpec {
|
||||
id: concat!("multicam.cutToCamera", $n),
|
||||
label: concat!("Cut to Camera ", $n),
|
||||
menu: &[],
|
||||
shortcut: Some(concat!("Ctrl+", $n)),
|
||||
params: r#"{"videoOnly":bool?}"#,
|
||||
enabled: has_seq,
|
||||
run: |s, p| cut_to_camera(s, &with_camera(p, $n)),
|
||||
journal: true,
|
||||
},
|
||||
)*]
|
||||
};
|
||||
}
|
||||
|
||||
fn with_camera(p: &Value, n: u64) -> Value {
|
||||
let mut v = if p.is_object() { p.clone() } else { json!({}) };
|
||||
v["camera"] = json!(n);
|
||||
v
|
||||
}
|
||||
|
||||
pub fn commands() -> Vec<CommandSpec> {
|
||||
let mut v = base_commands();
|
||||
v.extend(camera_keys!(1, 2, 3, 4, 5, 6, 7, 8, 9));
|
||||
v
|
||||
}
|
||||
|
||||
fn base_commands() -> Vec<CommandSpec> {
|
||||
vec![
|
||||
spec(
|
||||
"clip.synchronize",
|
||||
"Synchronize…",
|
||||
&["Clip"],
|
||||
r#"{"method":"in|out|timecode|marker|audio","clips":[id]?,"reference":clip?,"track":"V1"|"A1"?,"ignoreHours":bool?,"marker":str?,"offset":frames?}"#,
|
||||
can_synchronize,
|
||||
synchronize,
|
||||
true,
|
||||
),
|
||||
spec(
|
||||
"clip.mergeClips",
|
||||
"Merge Clips…",
|
||||
&["Clip"],
|
||||
r#"{"items":[id]?,"method":"in|out|timecode|marker|audio","name":str?,"removeVideoAudio":bool?,"ignoreHours":bool?,"marker":str?,"offset":frames?}"#,
|
||||
has_two_items,
|
||||
merge_clips,
|
||||
true,
|
||||
),
|
||||
spec(
|
||||
"clip.createMulticam",
|
||||
"Create Multi-Camera Source Sequence…",
|
||||
&["Clip"],
|
||||
r#"{"items":[id]?,"name":str?,"method":"in|out|timecode|marker|audio","ignoreHours":bool?,"marker":str?,"offset":frames?,"audio":"camera1|all|switch","cameraNames":"clip|track|metadata","processedBin":bool?,"reference":item?}"#,
|
||||
has_items,
|
||||
create_multicam,
|
||||
true,
|
||||
),
|
||||
spec("clip.multicamEnable", "Enable", &["Clip", "Multi-Camera"], r#"{"clips":[id]?,"enabled":bool?}"#, has_selection, enable, true),
|
||||
spec("clip.multicamFlatten", "Flatten", &["Clip", "Multi-Camera"], r#"{"clips":[id]?}"#, has_selection, flatten, true),
|
||||
spec(
|
||||
"multicam.switchAngle",
|
||||
"Switch Multi-Camera Angle",
|
||||
&[],
|
||||
r#"{"camera":1..16 (shown order)|"angle":0-based,"clips":[id]?,"time":ticks?,"videoOnly":bool?,"audioOnly":bool?}"#,
|
||||
has_seq,
|
||||
switch_angle,
|
||||
true,
|
||||
),
|
||||
spec("multicam.recordStart", "Start Multi-Camera Recording", &[], r#"{"time":ticks?}"#, has_seq, record_start, false),
|
||||
spec("multicam.cut", "Cut to Camera", &[], r#"{"camera":1..16|"angle":0-based,"time":ticks?,"videoOnly":bool?}"#, has_seq, cut, true),
|
||||
spec("multicam.recordStop", "Stop Multi-Camera Recording", &[], r#"{"time":ticks?}"#, always, record_stop, true),
|
||||
spec("multicam.audioFollowsVideo", "Multi-Camera Audio Follows Video", &[], r#"{"enabled":bool?}"#, always, audio_follows, true),
|
||||
spec(
|
||||
"multicam.editCameras",
|
||||
"Edit Cameras…",
|
||||
&[],
|
||||
r#"{"sequence":id?,"cameras":[{"angle":0-based,"name":str?,"enabled":bool?}]?,"audio":"camera1|all|switch"?}"#,
|
||||
always,
|
||||
edit_cameras,
|
||||
true,
|
||||
),
|
||||
spec("multicam.cutToCamera", "Cut to Camera", &[], r#"{"camera":1..16|"angle":0-based,"time":ticks?,"videoOnly":bool?}"#, has_seq, cut_to_camera, true),
|
||||
CommandSpec {
|
||||
id: "multicam.inspect",
|
||||
label: "Inspect Multi-Camera",
|
||||
menu: &[],
|
||||
shortcut: None,
|
||||
params: r#"{"time":ticks?}"#,
|
||||
enabled: always,
|
||||
run: inspect,
|
||||
journal: false,
|
||||
},
|
||||
]
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- enablement
|
||||
|
||||
fn can_synchronize(s: &Session) -> std::result::Result<(), String> {
|
||||
has_seq(s)?;
|
||||
let sel = with_links(s, &s.state.selection);
|
||||
if sel.len() < 2 { Err("select at least two clips to synchronize".into()) } else { Ok(()) }
|
||||
}
|
||||
fn has_items(s: &Session) -> std::result::Result<(), String> {
|
||||
if s.state.project_selection.iter().any(|i| sync::project_clip(&s.project, *i).is_some()) {
|
||||
Ok(())
|
||||
} else {
|
||||
Err("select clips in the Project panel".into())
|
||||
}
|
||||
}
|
||||
fn has_two_items(s: &Session) -> std::result::Result<(), String> {
|
||||
if s.state.project_selection.iter().filter(|i| sync::project_clip(&s.project, **i).is_some()).count() >= 2 {
|
||||
Ok(())
|
||||
} else {
|
||||
Err("select a video clip and audio clips in the Project panel".into())
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- helpers
|
||||
|
||||
fn items_p(s: &Session, p: &Value) -> Vec<ItemId> {
|
||||
match p.get("items").and_then(Value::as_array) {
|
||||
Some(a) => a.iter().filter_map(|v| v.as_u64().map(ItemId)).collect(),
|
||||
None => s.state.project_selection.clone(),
|
||||
}
|
||||
}
|
||||
|
||||
fn method_p(p: &Value, cmd: &str) -> Result<Method> {
|
||||
Method::from_params(p).map_err(|e| bad(cmd, e))
|
||||
}
|
||||
|
||||
fn offset_p(p: &Value, rate: FrameRate) -> Tick {
|
||||
p.get("offset").and_then(Value::as_i64).map(|f| rate.tick_of(f)).unwrap_or(Tick::ZERO)
|
||||
}
|
||||
|
||||
/// Run `f` on sequence `seq_id` of `p` with an edit context (ids, media durations), then check it.
|
||||
fn seq_edit<R>(p: &mut Project, seq_id: ItemId, media: &Arc<MediaPool>, f: impl FnOnce(&mut Sequence, &mut EditCtx) -> Result<R>) -> Result<R> {
|
||||
let snapshot = p.clone();
|
||||
let media = media.clone();
|
||||
let durations = move |id: ItemId| crate::media_duration(&snapshot, &media, id);
|
||||
let min = p.sequence(seq_id).map(|s| s.settings.frame_rate.frame_duration()).unwrap_or(Tick(1));
|
||||
let mut next = p.next_id;
|
||||
let r = {
|
||||
let seq = p.sequence_mut(seq_id).ok_or(EngineError::NoSequence)?;
|
||||
let mut ctx = EditCtx { next_id: &mut next, media_duration: &durations, min_duration: min };
|
||||
f(seq, &mut ctx)?
|
||||
};
|
||||
p.next_id = next;
|
||||
if let Some(q) = p.sequence(seq_id) {
|
||||
q.check().map_err(EngineError::Other)?;
|
||||
}
|
||||
Ok(r)
|
||||
}
|
||||
|
||||
/// Frame-aligned placement that keeps sub-frame (sample) accuracy: a clip whose media range should
|
||||
/// start at the exact timeline time `exact` starts at the next frame boundary instead, with its
|
||||
/// source In moved by the difference. Returns (start, source In shift).
|
||||
fn frame_align(exact: Tick, rate: FrameRate) -> (Tick, Tick) {
|
||||
let snapped = rate.snap(exact);
|
||||
let start = if snapped < exact { snapped + rate.frame_duration() } else { snapped };
|
||||
(start, start - exact)
|
||||
}
|
||||
|
||||
/// A track item showing `range` of `item` whose media time `range.start` sits at timeline `exact`.
|
||||
fn placed_item(p: &mut Project, item: ItemId, kind: TrackKind, exact: Tick, range: TimeRange, settings: &SequenceSettings) -> Option<TrackItem> {
|
||||
let rate = settings.frame_rate;
|
||||
let (start, shift) = frame_align(exact, rate);
|
||||
let src = TimeRange::from_bounds(range.start + shift, range.end());
|
||||
let mut ti = p.make_track_item(item, kind, start, src, rate)?;
|
||||
// whole frames that fit in the media
|
||||
let frames = rate.frame_at(src.duration).max(1);
|
||||
ti.duration = rate.tick_of(frames);
|
||||
if kind == TrackKind::Video {
|
||||
let size = sync_source_size(p, item).unwrap_or((settings.width, settings.height));
|
||||
for e in &mut ti.effects {
|
||||
resolve_auto_points(e, (settings.width, settings.height), size);
|
||||
}
|
||||
}
|
||||
Some(ti)
|
||||
}
|
||||
|
||||
fn sync_source_size(p: &Project, item: ItemId) -> Option<(u32, u32)> {
|
||||
match &p.item(item)?.kind {
|
||||
ItemKind::Media(m) => m.info.video.as_ref().map(|v| (v.width, v.height)),
|
||||
ItemKind::Subclip { parent, .. } => sync_source_size(p, *parent),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- Synchronize
|
||||
|
||||
/// A set of linked timeline clips that move together, with the clip used to sync it.
|
||||
struct Unit {
|
||||
members: Vec<ClipId>,
|
||||
rep: TrackItem,
|
||||
rep_track: (TrackKind, usize),
|
||||
}
|
||||
|
||||
fn synchronize(s: &mut Session, p: &Value) -> Result<Value> {
|
||||
let cmd = "clip.synchronize";
|
||||
let method = method_p(p, cmd)?;
|
||||
let seq_id = s.state.active_sequence.ok_or(EngineError::NoSequence)?;
|
||||
let q = s.active_sequence().ok_or(EngineError::NoSequence)?.clone();
|
||||
let rate = q.settings.frame_rate;
|
||||
let sel = with_links(s, &clips_p(s, p));
|
||||
// group into link units
|
||||
let mut units: Vec<Unit> = Vec::new();
|
||||
let mut seen: Vec<ClipId> = Vec::new();
|
||||
let pos = |c: ClipId| -> Option<(TrackKind, usize, TrackItem)> {
|
||||
for (k, ts) in [(TrackKind::Video, &q.video_tracks), (TrackKind::Audio, &q.audio_tracks)] {
|
||||
for (ti, t) in ts.iter().enumerate() {
|
||||
if let Some(it) = t.item(c) {
|
||||
return Some((k, ti, it.clone()));
|
||||
}
|
||||
}
|
||||
}
|
||||
None
|
||||
};
|
||||
for &c in &sel {
|
||||
if seen.contains(&c) {
|
||||
continue;
|
||||
}
|
||||
let Some((_, _, it)) = pos(c) else { continue };
|
||||
let members: Vec<ClipId> = match it.link {
|
||||
Some(l) => sel.iter().copied().filter(|o| pos(*o).is_some_and(|x| x.2.link == Some(l))).collect(),
|
||||
None => vec![c],
|
||||
};
|
||||
seen.extend(&members);
|
||||
let mut cands: Vec<(TrackKind, usize, TrackItem)> = members.iter().filter_map(|m| pos(*m)).collect();
|
||||
// the representative: audio for audio sync, else video; lowest track first
|
||||
let audio_first = method == Method::Audio;
|
||||
cands.sort_by_key(|(k, ti, _)| (if (*k == TrackKind::Audio) == audio_first { 0 } else { 1 }, *ti));
|
||||
let (k, ti, rep) = cands.remove(0);
|
||||
units.push(Unit { members, rep, rep_track: (k, ti) });
|
||||
}
|
||||
if units.len() < 2 {
|
||||
return Err(bad(cmd, "select clips from at least two different sources"));
|
||||
}
|
||||
// reference unit
|
||||
let reference = if let Some(c) = u64_p(p, "reference").map(ClipId) {
|
||||
units.iter().position(|u| u.members.contains(&c)).ok_or_else(|| bad(cmd, "`reference` is not among the clips"))?
|
||||
} else if let Some(t) = track_p(s, p, "track") {
|
||||
units
|
||||
.iter()
|
||||
.position(|u| u.members.iter().any(|m| q.track(t).is_some_and(|tr| tr.item(*m).is_some())))
|
||||
.ok_or_else(|| bad(cmd, "no selected clip on the reference track"))?
|
||||
} else {
|
||||
(0..units.len()).min_by_key(|&i| (if units[i].rep_track.0 == TrackKind::Video { 0 } else { 1 }, units[i].rep_track.1)).unwrap_or(0)
|
||||
};
|
||||
let clips: Vec<SyncClip> = units.iter().map(|u| sync::timeline_clip(&s.project, &u.rep)).collect();
|
||||
let anchors = sync::anchors(s, &clips, reference, &method, offset_p(p, rate))?;
|
||||
// timeline time of each unit's anchor must equal the reference's
|
||||
let at = |u: &Unit, a: Tick| u.rep.start + Tick(((a - u.rep.source_in).0 as f64 / u.rep.speed.abs().max(1e-9)).round() as i64);
|
||||
let t_ref = at(&units[reference], anchors[reference].anchor);
|
||||
let mut deltas: Vec<Tick> = units.iter().zip(&anchors).map(|(u, a)| t_ref - at(u, a.anchor)).collect();
|
||||
// nothing may move before the sequence start: shift everything (reference too) right
|
||||
let min_start = units
|
||||
.iter()
|
||||
.zip(&deltas)
|
||||
.flat_map(|(u, d)| u.members.iter().filter_map(|m| pos(*m)).map(move |x| x.2.start + *d).collect::<Vec<_>>())
|
||||
.min()
|
||||
.unwrap_or(Tick::ZERO);
|
||||
if min_start < Tick::ZERO {
|
||||
deltas.iter_mut().for_each(|d| *d -= min_start);
|
||||
}
|
||||
let mut moved = 0usize;
|
||||
let mut placements: Vec<(TrackId, TrackItem)> = Vec::new();
|
||||
let mut remove: Vec<ClipId> = Vec::new();
|
||||
for (u, d) in units.iter().zip(&deltas) {
|
||||
if *d == Tick::ZERO {
|
||||
continue;
|
||||
}
|
||||
moved += 1;
|
||||
// frame-aligned start, sub-frame remainder taken from the source In
|
||||
let (start, shift) = frame_align(u.rep.start + *d, rate);
|
||||
let dd = start - u.rep.start;
|
||||
for m in &u.members {
|
||||
let Some((tid, it)) = q.find_item(*m) else { continue };
|
||||
let mut it = it.clone();
|
||||
it.start += dd;
|
||||
if shift > Tick::ZERO {
|
||||
it.source_in += Tick((shift.0 as f64 * it.speed.abs()).round() as i64);
|
||||
it.duration = (it.duration - rate.frame_duration()).max(rate.frame_duration());
|
||||
}
|
||||
remove.push(*m);
|
||||
placements.push((tid, it));
|
||||
}
|
||||
}
|
||||
let starts: Vec<Tick> = units.iter().zip(&deltas).map(|(u, d)| u.rep.start + *d).collect();
|
||||
let rep = sync::report(&clips, &anchors, &starts, rate);
|
||||
if moved > 0 {
|
||||
let media = s.media.clone();
|
||||
s.edit(&format!("Synchronize ({})", method.name()), |pr, st| {
|
||||
seq_edit(pr, seq_id, &media, |seq, ctx| {
|
||||
edit::delete_items(seq, &remove);
|
||||
let ids = edit::overwrite(seq, placements, ctx)?;
|
||||
Ok(ids)
|
||||
})
|
||||
.map(|ids| {
|
||||
st.selection = ids;
|
||||
})
|
||||
})?;
|
||||
}
|
||||
Ok(json!({"moved": moved, "reference": reference, "clips": rep}))
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- Merge Clips / Create Multi-Camera
|
||||
|
||||
fn camera_name(p: &Project, item: ItemId, mode: &str, n: usize) -> String {
|
||||
let it = p.item(item);
|
||||
let clip = || it.map(|i| i.name.clone()).unwrap_or_else(|| format!("Camera {n}"));
|
||||
match mode {
|
||||
"track" | "tracks" | "trackname" | "trackNames" | "enumerate" => format!("Camera {n}"),
|
||||
"metadata" | "cameraAngle" | "label" => it
|
||||
.and_then(|i| {
|
||||
["Camera Angle", "Camera Label", "Camera", "camera", "cameraAngle"].iter().find_map(|k| i.metadata.get(*k).filter(|v| !v.is_empty()).cloned())
|
||||
})
|
||||
.unwrap_or_else(clip),
|
||||
_ => clip(),
|
||||
}
|
||||
}
|
||||
|
||||
/// Resolve and order the clips for Merge Clips / Create Multi-Camera.
|
||||
fn sync_items(s: &Session, items: &[ItemId], cmd: &str) -> Result<Vec<SyncClip>> {
|
||||
let mut out = Vec::new();
|
||||
for i in items {
|
||||
let c = sync::project_clip(&s.project, *i).ok_or_else(|| bad(cmd, format!("item {} is not a clip", i.0)))?;
|
||||
if !out.iter().any(|o: &SyncClip| o.item == c.item) {
|
||||
out.push(c);
|
||||
}
|
||||
}
|
||||
if out.is_empty() {
|
||||
return Err(bad(cmd, "no clips"));
|
||||
}
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
fn settings_for(s: &Session, item: ItemId) -> SequenceSettings {
|
||||
let info = match s.project.item(item).map(|i| &i.kind) {
|
||||
Some(ItemKind::Media(m)) => Some(m.info.clone()),
|
||||
Some(ItemKind::Subclip { parent, .. }) => s.project.item(*parent).and_then(|i| i.as_media()).map(|m| m.info.clone()),
|
||||
_ => None,
|
||||
};
|
||||
let mut st = info.as_ref().map(crate::commands::default_seq_settings_for).unwrap_or_default();
|
||||
if let Some(m) = s.project.item(item).and_then(|i| i.as_media())
|
||||
&& let Some(r) = m.interpret.frame_rate
|
||||
{
|
||||
st.frame_rate = r;
|
||||
}
|
||||
st
|
||||
}
|
||||
|
||||
fn processed_bin(p: &mut Project) -> filmcraft_project::BinId {
|
||||
if let Some(b) = p.root.children.iter().find_map(|c| match c {
|
||||
BinEntry::Bin(b) if b.name == "Processed Clips" => Some(b.id),
|
||||
_ => None,
|
||||
}) {
|
||||
return b;
|
||||
}
|
||||
p.add_bin("Processed Clips", None)
|
||||
}
|
||||
|
||||
fn merge_clips(s: &mut Session, p: &Value) -> Result<Value> {
|
||||
let cmd = "clip.mergeClips";
|
||||
let method = method_p(p, cmd)?;
|
||||
let clips = sync_items(s, &items_p(s, p), cmd)?;
|
||||
let has_v = |c: &SyncClip| s.project.item(c.item).is_some_and(|i| i.has_video());
|
||||
let has_a = |c: &SyncClip| s.project.item(c.item).is_some_and(|i| i.has_audio());
|
||||
let videos: Vec<usize> = (0..clips.len()).filter(|&i| has_v(&clips[i])).collect();
|
||||
if videos.len() > 1 {
|
||||
return Err(bad(cmd, "Merge Clips takes one video clip plus audio clips"));
|
||||
}
|
||||
let audio_n = clips.iter().filter(|c| !has_v(c)).count();
|
||||
if audio_n == 0 || audio_n > 16 {
|
||||
return Err(bad(cmd, "Merge Clips takes 1–16 audio clips"));
|
||||
}
|
||||
// the video clip (or the first audio clip) is the reference
|
||||
let reference = videos.first().copied().unwrap_or(0);
|
||||
let mut order: Vec<usize> = vec![reference];
|
||||
order.extend((0..clips.len()).filter(|&i| i != reference));
|
||||
let clips: Vec<SyncClip> = order.iter().map(|&i| clips[i].clone()).collect();
|
||||
let settings = settings_for(s, clips[0].item);
|
||||
let rate = settings.frame_rate;
|
||||
let anchors = sync::anchors(s, &clips, 0, &method, offset_p(p, rate))?;
|
||||
let starts = sync::placements(&clips, &anchors, rate, false);
|
||||
let remove_video_audio = bool_p(p, "removeVideoAudio").unwrap_or(false);
|
||||
let name = str_p(p, "name").map(str::to_string).unwrap_or_else(|| format!("{} - Merged", clips[0].name));
|
||||
let video = videos.first().map(|_| clips[0].item);
|
||||
let vid_audio = video.is_some() && has_a(&clips[0]) && !remove_video_audio;
|
||||
let audio_items: Vec<usize> = (1..clips.len()).collect();
|
||||
let n_audio = audio_items.len() + vid_audio as usize;
|
||||
let rep = sync::report(&clips, &anchors, &starts, rate);
|
||||
let bin = s.project.root.parent_of(clips[0].item).filter(|b| *b != s.project.root.id);
|
||||
let id = s.edit("Merge Clips", |pr, st| {
|
||||
let sid = pr.new_sequence(&name, settings.clone(), video.is_some() as usize, n_audio, bin);
|
||||
let mut v_items = Vec::new();
|
||||
let mut a_items = Vec::new();
|
||||
let link = pr.alloc_id();
|
||||
if video.is_some() {
|
||||
let mut v = placed_item(pr, clips[0].item, TrackKind::Video, starts[0], clips[0].range, &settings).ok_or_else(|| bad(cmd, "bad video clip"))?;
|
||||
v.link = Some(link);
|
||||
v_items.push(v);
|
||||
if vid_audio {
|
||||
let mut a = placed_item(pr, clips[0].item, TrackKind::Audio, starts[0], clips[0].range, &settings).ok_or_else(|| bad(cmd, "bad clip"))?;
|
||||
a.link = Some(link);
|
||||
a_items.push(a);
|
||||
}
|
||||
}
|
||||
for &i in &audio_items {
|
||||
let mut a = placed_item(pr, clips[i].item, TrackKind::Audio, starts[i], clips[i].range, &settings).ok_or_else(|| bad(cmd, "bad audio clip"))?;
|
||||
a.link = Some(link);
|
||||
a_items.push(a);
|
||||
}
|
||||
let q = pr.sequence_mut(sid).expect("new sequence");
|
||||
for (t, it) in q.video_tracks.iter_mut().zip(v_items) {
|
||||
t.items.push(it);
|
||||
}
|
||||
for (t, it) in q.audio_tracks.iter_mut().zip(a_items) {
|
||||
t.items.push(it);
|
||||
}
|
||||
q.merged = Some(MergedClip { video, audio: audio_items.iter().map(|&i| clips[i].item).collect(), sync: method.name().into() });
|
||||
q.check().map_err(EngineError::Other)?;
|
||||
if let Some(it) = pr.item_mut(sid) {
|
||||
it.label = Label::Iris;
|
||||
}
|
||||
st.project_selection = vec![sid];
|
||||
Ok(sid)
|
||||
})?;
|
||||
Ok(json!({"item": id.0, "clips": rep}))
|
||||
}
|
||||
|
||||
fn create_multicam(s: &mut Session, p: &Value) -> Result<Value> {
|
||||
let cmd = "clip.createMulticam";
|
||||
let method = method_p(p, cmd)?;
|
||||
let mut clips = sync_items(s, &items_p(s, p), cmd)?;
|
||||
let has_v = |s: &Session, c: &SyncClip| s.project.item(c.item).is_some_and(|i| i.has_video());
|
||||
let has_a = |s: &Session, c: &SyncClip| s.project.item(c.item).is_some_and(|i| i.has_audio());
|
||||
// cameras with video first (selection order), then audio-only sources
|
||||
clips.sort_by_key(|c| !has_v(s, c));
|
||||
if !clips.iter().any(|c| has_v(s, c)) {
|
||||
return Err(bad(cmd, "select at least one clip with video"));
|
||||
}
|
||||
let names_mode = str_p(p, "cameraNames").unwrap_or("clip").to_string();
|
||||
if names_mode == "metadata" {
|
||||
// cameras ordered by their Camera Angle metadata when present
|
||||
let key = |c: &SyncClip| s.project.item(c.item).and_then(|i| i.metadata.get("Camera Angle").cloned());
|
||||
let mut v: Vec<(bool, Option<String>, SyncClip)> = clips.iter().map(|c| (!has_v(s, c), key(c), c.clone())).collect();
|
||||
v.sort_by_key(|a| (a.0, a.1.is_none(), a.1.clone()));
|
||||
clips = v.into_iter().map(|x| x.2).collect();
|
||||
}
|
||||
let reference = match item_p(p, "reference") {
|
||||
Some(r) => clips.iter().position(|c| c.item == r).ok_or_else(|| bad(cmd, "`reference` is not among the items"))?,
|
||||
None => 0,
|
||||
};
|
||||
let audio = match str_p(p, "audio") {
|
||||
Some(a) => MulticamAudio::from_name(a).ok_or_else(|| bad(cmd, format!("unknown audio mode `{a}` (camera1, all, switch)")))?,
|
||||
None => MulticamAudio::Camera1,
|
||||
};
|
||||
let settings = settings_for(s, clips[reference].item);
|
||||
let rate = settings.frame_rate;
|
||||
let anchors = sync::anchors(s, &clips, reference, &method, offset_p(p, rate))?;
|
||||
let starts = sync::placements(&clips, &anchors, rate, false);
|
||||
let rep = sync::report(&clips, &anchors, &starts, rate);
|
||||
let name = str_p(p, "name").map(str::to_string).unwrap_or_else(|| format!("{} Multicam", clips[reference].name));
|
||||
let move_processed = bool_p(p, "processedBin").unwrap_or(false);
|
||||
let flags: Vec<(bool, bool)> = clips.iter().map(|c| (has_v(s, c), has_a(s, c))).collect();
|
||||
let names: Vec<String> = clips.iter().enumerate().map(|(i, c)| camera_name(&s.project, c.item, &names_mode, i + 1)).collect();
|
||||
let nv = flags.iter().filter(|f| f.0).count();
|
||||
let na = flags.iter().filter(|f| f.1).count();
|
||||
let id = s.edit("Create Multi-Camera Source Sequence", |pr, st| {
|
||||
let sid = pr.new_sequence(&name, settings.clone(), nv, na.max(1), None);
|
||||
let mut cameras = Vec::new();
|
||||
let (mut vi, mut ai) = (0usize, 0usize);
|
||||
let mut placed: Vec<(TrackKind, usize, TrackItem)> = Vec::new();
|
||||
let (vtracks, atracks): (Vec<TrackId>, Vec<TrackId>) = {
|
||||
let q = pr.sequence(sid).expect("new sequence");
|
||||
(q.video_tracks.iter().map(|t| t.id).collect(), q.audio_tracks.iter().map(|t| t.id).collect())
|
||||
};
|
||||
for (k, c) in clips.iter().enumerate() {
|
||||
let (hv, ha) = flags[k];
|
||||
let link = (hv && ha).then(|| pr.alloc_id());
|
||||
let mut cam = Camera { name: names[k].clone(), video_track: None, audio_tracks: Vec::new(), enabled: true, source: Some(c.item) };
|
||||
if hv {
|
||||
let mut v = placed_item(pr, c.item, TrackKind::Video, starts[k], c.range, &settings).ok_or_else(|| bad(cmd, "bad clip"))?;
|
||||
v.link = link;
|
||||
placed.push((TrackKind::Video, vi, v));
|
||||
cam.video_track = Some(vtracks[vi]);
|
||||
vi += 1;
|
||||
}
|
||||
if ha {
|
||||
let mut a = placed_item(pr, c.item, TrackKind::Audio, starts[k], c.range, &settings).ok_or_else(|| bad(cmd, "bad clip"))?;
|
||||
a.link = link;
|
||||
placed.push((TrackKind::Audio, ai, a));
|
||||
cam.audio_tracks.push(atracks[ai]);
|
||||
ai += 1;
|
||||
}
|
||||
cameras.push(cam);
|
||||
}
|
||||
{
|
||||
let q = pr.sequence_mut(sid).expect("new sequence");
|
||||
for (kind, idx, it) in placed {
|
||||
q.tracks_mut(kind)[idx].items.push(it);
|
||||
}
|
||||
for cam in &cameras {
|
||||
if let Some(vt) = cam.video_track.and_then(|id| q.video_tracks.iter_mut().find(|x| x.id == id)) {
|
||||
vt.name = cam.name.clone();
|
||||
}
|
||||
for at in &cam.audio_tracks {
|
||||
if let Some(x) = q.audio_tracks.iter_mut().find(|x| x.id == *at) {
|
||||
x.name = cam.name.clone();
|
||||
}
|
||||
}
|
||||
}
|
||||
q.multicam = Some(MulticamSource { cameras, audio, sync: method.name().into() });
|
||||
q.check().map_err(EngineError::Other)?;
|
||||
}
|
||||
if let Some(it) = pr.item_mut(sid) {
|
||||
it.label = Label::Mango;
|
||||
}
|
||||
if move_processed {
|
||||
let bin = processed_bin(pr);
|
||||
for c in &clips {
|
||||
pr.root.remove_item(c.item);
|
||||
if let Some(b) = pr.root.find_bin_mut(bin) {
|
||||
b.children.push(BinEntry::Item(c.item));
|
||||
}
|
||||
}
|
||||
}
|
||||
st.project_selection = vec![sid];
|
||||
Ok(sid)
|
||||
})?;
|
||||
Ok(json!({"sequence": id.0, "cameras": names, "clips": rep}))
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- multi-camera clips
|
||||
|
||||
/// The topmost enabled multi-camera video clip at `t` (targeted tracks first) in the active
|
||||
/// sequence: (track, clip).
|
||||
pub fn multicam_clip_at(s: &Session, t: Tick) -> Option<(TrackId, TrackItem)> {
|
||||
let q = s.active_sequence()?;
|
||||
let tg = s.targeting().targeted;
|
||||
let at = |tr: &Track| tr.item_at(t).filter(|i| i.multicam.is_some_and(|m| m.enabled) && s.project.sequence(i.item).is_some()).cloned();
|
||||
let targeted = q.video_tracks.iter().rev().filter(|tr| tg.contains(&tr.id)).find_map(|tr| at(tr).map(|i| (tr.id, i)));
|
||||
targeted.or_else(|| q.video_tracks.iter().rev().find_map(|tr| at(tr).map(|i| (tr.id, i))))
|
||||
}
|
||||
|
||||
/// Resolve `camera` (1-based, shown order) or `angle` (0-based) for the source of `clip`.
|
||||
fn angle_p(s: &Session, p: &Value, source: ItemId, cmd: &str) -> Result<u32> {
|
||||
let q = s.project.sequence(source).ok_or_else(|| bad(cmd, "not a multi-camera clip"))?;
|
||||
let cams = q.cameras();
|
||||
if let Some(a) = u64_p(p, "angle") {
|
||||
if (a as usize) < cams.cameras.len() {
|
||||
return Ok(a as u32);
|
||||
}
|
||||
return Err(bad(cmd, format!("angle {a} out of range (0–{})", cams.cameras.len().saturating_sub(1))));
|
||||
}
|
||||
let n = u64_p(p, "camera").ok_or_else(|| bad(cmd, "need `camera` (1-based) or `angle`"))? as usize;
|
||||
let shown = cams.shown_angles();
|
||||
shown.get(n.wrapping_sub(1)).map(|a| *a as u32).ok_or_else(|| bad(cmd, format!("there is no camera {n} ({} shown)", shown.len())))
|
||||
}
|
||||
|
||||
/// Linked audio clips of `video` that are multi-camera clips.
|
||||
fn audio_partners(q: &Sequence, video: &TrackItem) -> Vec<ClipId> {
|
||||
let Some(l) = video.link else { return Vec::new() };
|
||||
q.audio_tracks.iter().flat_map(|t| t.items.iter()).filter(|i| i.link == Some(l) && i.multicam.is_some()).map(|i| i.id).collect()
|
||||
}
|
||||
|
||||
fn switch_angle(s: &mut Session, p: &Value) -> Result<Value> {
|
||||
let cmd = "multicam.switchAngle";
|
||||
let q = s.active_sequence().ok_or(EngineError::NoSequence)?.clone();
|
||||
let t = time_p(s, p, "").unwrap_or(s.playhead());
|
||||
let video_only = bool_p(p, "videoOnly").unwrap_or(false);
|
||||
let audio_only = bool_p(p, "audioOnly").unwrap_or(false);
|
||||
let follows = s.state.multicam_audio_follows_video;
|
||||
let videos: Vec<TrackItem> = if p.get("clips").is_some() || p.get("clip").is_some() {
|
||||
clips_p(s, p).iter().filter_map(|c| q.find_item(*c).map(|x| x.1.clone())).filter(|i| i.multicam.is_some()).collect()
|
||||
} else {
|
||||
multicam_clip_at(s, t).map(|x| vec![x.1]).unwrap_or_default()
|
||||
};
|
||||
if videos.is_empty() {
|
||||
return Err(EngineError::Other("no multi-camera clip at the playhead".into()));
|
||||
}
|
||||
let angle = angle_p(s, p, videos[0].item, cmd)?;
|
||||
let mut targets: Vec<ClipId> = Vec::new();
|
||||
for v in &videos {
|
||||
let is_audio = q.audio_tracks.iter().any(|t| t.item(v.id).is_some());
|
||||
if is_audio {
|
||||
targets.push(v.id);
|
||||
continue;
|
||||
}
|
||||
if !audio_only {
|
||||
targets.push(v.id);
|
||||
}
|
||||
if audio_only || (follows && !video_only) {
|
||||
targets.extend(audio_partners(&q, v));
|
||||
}
|
||||
}
|
||||
let n = s.edit_sequence("Switch Multi-Camera Angle", |seq, _, _| Ok(edit::multicam::set_angle(seq, &targets, angle)))?;
|
||||
Ok(json!({"angle": angle, "changed": n}))
|
||||
}
|
||||
|
||||
fn enable(s: &mut Session, p: &Value) -> Result<Value> {
|
||||
let q = s.active_sequence().ok_or(EngineError::NoSequence)?.clone();
|
||||
let clips = with_links(s, &clips_p(s, p));
|
||||
let nested: Vec<TrackItem> = clips.iter().filter_map(|c| q.find_item(*c).map(|x| x.1.clone())).filter(|i| s.project.sequence(i.item).is_some()).collect();
|
||||
if nested.is_empty() {
|
||||
return Err(EngineError::Other("select nested sequence or multi-camera clips".into()));
|
||||
}
|
||||
let on = bool_p(p, "enabled").unwrap_or_else(|| nested.iter().any(|i| !i.multicam.is_some_and(|m| m.enabled)));
|
||||
let project = s.project.clone();
|
||||
let ids: Vec<ClipId> = nested.iter().map(|i| i.id).collect();
|
||||
let n = s.edit_sequence(if on { "Enable Multi-Camera" } else { "Disable Multi-Camera" }, |seq, _, _| {
|
||||
let mut n = 0;
|
||||
for t in seq.all_tracks_mut() {
|
||||
for it in t.items.iter_mut().filter(|i| ids.contains(&i.id)) {
|
||||
let first = project.sequence(it.item).and_then(|q| q.cameras().first_video_angle()).unwrap_or(0) as u32;
|
||||
let sel = it.multicam.get_or_insert(MulticamSel { enabled: false, angle: first });
|
||||
if sel.enabled != on {
|
||||
sel.enabled = on;
|
||||
n += 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(n)
|
||||
})?;
|
||||
Ok(json!({"enabled": on, "changed": n}))
|
||||
}
|
||||
|
||||
fn flatten(s: &mut Session, p: &Value) -> Result<Value> {
|
||||
let seq_id = s.state.active_sequence.ok_or(EngineError::NoSequence)?;
|
||||
let q = s.active_sequence().ok_or(EngineError::NoSequence)?.clone();
|
||||
let clips = with_links(s, &clips_p(s, p));
|
||||
let mut targets: Vec<(TrackKind, usize, TrackItem)> = Vec::new();
|
||||
for (k, ts) in [(TrackKind::Video, &q.video_tracks), (TrackKind::Audio, &q.audio_tracks)] {
|
||||
for (ti, t) in ts.iter().enumerate() {
|
||||
for it in t.items.iter().filter(|i| clips.contains(&i.id) && i.multicam.is_some_and(|m| m.enabled)) {
|
||||
targets.push((k, ti, it.clone()));
|
||||
}
|
||||
}
|
||||
}
|
||||
if targets.is_empty() {
|
||||
return Err(EngineError::Other("select multi-camera clips to flatten".into()));
|
||||
}
|
||||
let project = s.project.clone();
|
||||
let media = s.media.clone();
|
||||
let n = targets.len();
|
||||
let ids = s.edit("Flatten Multi-Camera", |pr, st| {
|
||||
// extra audio tracks needed for sources whose audio spans several tracks
|
||||
let need_a = targets
|
||||
.iter()
|
||||
.filter(|t| t.0 == TrackKind::Audio)
|
||||
.map(|(_, ti, it)| {
|
||||
let nq = project.sequence(it.item).map(|nq| audible(nq, it).len()).unwrap_or(1);
|
||||
ti + nq.max(1)
|
||||
})
|
||||
.max()
|
||||
.unwrap_or(0);
|
||||
let have = pr.sequence(seq_id).map(|q| q.audio_tracks.len()).unwrap_or(0);
|
||||
for k in have..need_a {
|
||||
let id = TrackId(pr.alloc_id());
|
||||
if let Some(q) = pr.sequence_mut(seq_id) {
|
||||
q.audio_tracks.push(Track::new(id, TrackKind::Audio, format!("Audio {}", k + 1)));
|
||||
}
|
||||
}
|
||||
let new_ids = seq_edit(pr, seq_id, &media, |seq, ctx| {
|
||||
let mut links = HashMap::new();
|
||||
let mut out = Vec::new();
|
||||
for (kind, ti, it) in &targets {
|
||||
let Some(nested) = project.sequence(it.item) else { continue };
|
||||
let mut pieces: Vec<(TrackId, TrackItem)> = Vec::new();
|
||||
match kind {
|
||||
TrackKind::Video => {
|
||||
let angle = it.multicam.map(|m| m.angle as usize).unwrap_or(0);
|
||||
if let Some(nt) = nested.angle_video_track_index(angle).and_then(|i| nested.video_tracks.get(i)) {
|
||||
let dest = seq.video_tracks[*ti].id;
|
||||
pieces.extend(edit::multicam::flatten_items(it, nt, ctx, &mut links)?.into_iter().map(|x| (dest, x)));
|
||||
}
|
||||
}
|
||||
TrackKind::Audio => {
|
||||
for (k, at) in audible(nested, it).iter().enumerate() {
|
||||
let Some(nt) = nested.audio_tracks.iter().find(|x| x.id == *at) else { continue };
|
||||
let Some(dest) = seq.audio_tracks.get(ti + k).map(|t| t.id) else { continue };
|
||||
pieces.extend(edit::multicam::flatten_items(it, nt, ctx, &mut links)?.into_iter().map(|x| (dest, x)));
|
||||
}
|
||||
}
|
||||
}
|
||||
out.extend(edit::multicam::replace_with(seq, it.id, pieces, ctx)?);
|
||||
}
|
||||
Ok(out)
|
||||
})?;
|
||||
st.selection = new_ids.clone();
|
||||
Ok(new_ids)
|
||||
})?;
|
||||
Ok(json!({"flattened": n, "clips": ids.iter().map(|c| c.0).collect::<Vec<_>>()}))
|
||||
}
|
||||
|
||||
/// Audio tracks of `nested` a multi-camera audio clip plays.
|
||||
fn audible(nested: &Sequence, it: &TrackItem) -> Vec<TrackId> {
|
||||
let angle = it.multicam_angle(nested);
|
||||
match &nested.multicam {
|
||||
Some(mc) => mc.audible_tracks(if mc.audio == MulticamAudio::SwitchAudio { angle } else { None }),
|
||||
None => nested.audio_tracks.iter().filter(|t| !t.muted).map(|t| t.id).collect(),
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- live switching
|
||||
|
||||
/// A live switching pass (Multi-Camera view, playing): every cut is applied at once, and the whole
|
||||
/// pass is one undo step.
|
||||
#[derive(Clone, Debug, Default)]
|
||||
pub struct Recorder {
|
||||
/// Sequence being recorded (None = no pass running).
|
||||
pub seq: Option<ItemId>,
|
||||
pub start: Tick,
|
||||
/// Tracks the pass switches (fixed by the first cut).
|
||||
pub tracks: Vec<TrackId>,
|
||||
pub cuts: Vec<Cut>,
|
||||
/// The multi-camera source being switched (fixed by the first cut).
|
||||
pub source: Option<ItemId>,
|
||||
/// The project before the pass's first cut.
|
||||
base: Option<Arc<Project>>,
|
||||
key: String,
|
||||
pass: u64,
|
||||
}
|
||||
|
||||
impl Recorder {
|
||||
pub fn active(&self) -> bool {
|
||||
self.seq.is_some()
|
||||
}
|
||||
}
|
||||
|
||||
const RECORD_LABEL: &str = "Record Multi-Camera";
|
||||
|
||||
fn record_start(s: &mut Session, p: &Value) -> Result<Value> {
|
||||
let t = time_p(s, p, "").unwrap_or(s.playhead());
|
||||
let pass = s.mcrec.pass + 1;
|
||||
s.mcrec = Recorder { seq: s.state.active_sequence, start: t, pass, key: format!("multicam-record-{pass}"), ..Default::default() };
|
||||
Ok(json!({"recording": true, "start": t.0}))
|
||||
}
|
||||
|
||||
/// Re-apply the pass from its base project up to `end`.
|
||||
fn apply_pass(s: &mut Session, end: Tick) -> Result<usize> {
|
||||
let Some(seq_id) = s.mcrec.seq else { return Ok(0) };
|
||||
let merging = s.history.merge_key.as_deref() == Some(s.mcrec.key.as_str()) && s.history.undo.last().is_some_and(|u| u.0 == RECORD_LABEL);
|
||||
if !merging || s.mcrec.base.is_none() {
|
||||
s.mcrec.base = Some(s.project.clone());
|
||||
}
|
||||
let base = s.mcrec.base.clone().expect("set above");
|
||||
let (tracks, cuts, key) = (s.mcrec.tracks.clone(), s.mcrec.cuts.clone(), s.mcrec.key.clone());
|
||||
let Some(source) = s.mcrec.source else { return Ok(0) };
|
||||
let media = s.media.clone();
|
||||
s.edit_merged(RECORD_LABEL, &key, |pr, _| {
|
||||
*pr = (*base).clone();
|
||||
seq_edit(pr, seq_id, &media, |seq, ctx| Ok(edit::multicam::record(seq, &tracks, source, &cuts, end, ctx)))
|
||||
})
|
||||
}
|
||||
|
||||
fn cut(s: &mut Session, p: &Value) -> Result<Value> {
|
||||
let cmd = "multicam.cut";
|
||||
let t = time_p(s, p, "").unwrap_or(s.playhead());
|
||||
if !s.mcrec.active() || s.mcrec.seq != s.state.active_sequence {
|
||||
// stopped: switch the clip at the playhead
|
||||
return switch_angle(s, p);
|
||||
}
|
||||
let (_, clip) = multicam_clip_at(s, t).ok_or_else(|| EngineError::Other("no multi-camera clip at the playhead".into()))?;
|
||||
let angle = angle_p(s, p, clip.item, cmd)?;
|
||||
if s.mcrec.tracks.is_empty() {
|
||||
let q = s.active_sequence().ok_or(EngineError::NoSequence)?;
|
||||
let mut tracks: Vec<TrackId> =
|
||||
q.video_tracks.iter().filter(|tr| tr.item_at(t).is_some_and(|i| i.item == clip.item && i.multicam.is_some())).map(|tr| tr.id).collect();
|
||||
tracks.truncate(1);
|
||||
if s.state.multicam_audio_follows_video && !bool_p(p, "videoOnly").unwrap_or(false) {
|
||||
let partners = audio_partners(q, &clip);
|
||||
tracks.extend(q.audio_tracks.iter().filter(|tr| tr.items.iter().any(|i| partners.contains(&i.id))).map(|tr| tr.id));
|
||||
}
|
||||
s.mcrec.tracks = tracks;
|
||||
}
|
||||
s.mcrec.source = Some(clip.item);
|
||||
s.mcrec.cuts.push(Cut { time: t, angle });
|
||||
// provisional: the new angle runs to the end of this multi-camera clip (its contiguous pieces)
|
||||
// until the pass stops
|
||||
let end = s
|
||||
.active_sequence()
|
||||
.and_then(|q| q.track(s.mcrec.tracks[0]))
|
||||
.map(|tr| {
|
||||
let mut end = clip.end();
|
||||
for it in tr.items.iter().filter(|i| i.start >= clip.end()) {
|
||||
if it.start != end || it.item != clip.item {
|
||||
break;
|
||||
}
|
||||
end = it.end();
|
||||
}
|
||||
end
|
||||
})
|
||||
.unwrap_or(clip.end());
|
||||
apply_pass(s, end.max(t))?;
|
||||
Ok(json!({"angle": angle, "time": t.0, "cuts": s.mcrec.cuts.len()}))
|
||||
}
|
||||
|
||||
/// Ctrl+N: an edit at the playhead on the multi-camera clip (and, when audio follows video, its
|
||||
/// linked audio), and camera N from there to the clip's end.
|
||||
fn cut_to_camera(s: &mut Session, p: &Value) -> Result<Value> {
|
||||
let cmd = "multicam.cutToCamera";
|
||||
if s.mcrec.active() && s.mcrec.seq == s.state.active_sequence {
|
||||
return cut(s, p);
|
||||
}
|
||||
let t = time_p(s, p, "").unwrap_or(s.playhead());
|
||||
let (_, clip) = multicam_clip_at(s, t).ok_or_else(|| EngineError::Other("no multi-camera clip at the playhead".into()))?;
|
||||
let angle = angle_p(s, p, clip.item, cmd)?;
|
||||
let q = s.active_sequence().ok_or(EngineError::NoSequence)?;
|
||||
let mut clips = vec![clip.id];
|
||||
if s.state.multicam_audio_follows_video && !bool_p(p, "videoOnly").unwrap_or(false) {
|
||||
clips.extend(audio_partners(q, &clip));
|
||||
}
|
||||
let n = s.edit_sequence("Cut to Camera", |seq, ctx, _| {
|
||||
let new = edit::razor_items(seq, &clips, t, ctx);
|
||||
// the pieces starting at t (new right halves, or the clips themselves at an edit point)
|
||||
let right: Vec<ClipId> =
|
||||
seq.all_tracks().flat_map(|tr| tr.items.iter()).filter(|i| i.start == t && (new.contains(&i.id) || clips.contains(&i.id))).map(|i| i.id).collect();
|
||||
Ok(edit::multicam::set_angle(seq, &right, angle))
|
||||
})?;
|
||||
Ok(json!({"angle": angle, "changed": n}))
|
||||
}
|
||||
|
||||
fn record_stop(s: &mut Session, p: &Value) -> Result<Value> {
|
||||
if !s.mcrec.active() {
|
||||
return Ok(json!({"recording": false, "cuts": 0}));
|
||||
}
|
||||
let t = time_p(s, p, "").unwrap_or(s.playhead());
|
||||
let n = s.mcrec.cuts.len();
|
||||
let mut points = 0;
|
||||
if n > 0 {
|
||||
points = apply_pass(s, t)?;
|
||||
}
|
||||
let pass = s.mcrec.pass;
|
||||
s.mcrec = Recorder { pass, ..Default::default() };
|
||||
s.history.merge_key = None;
|
||||
Ok(json!({"recording": false, "cuts": n, "editPoints": points}))
|
||||
}
|
||||
|
||||
fn audio_follows(s: &mut Session, p: &Value) -> Result<Value> {
|
||||
let on = bool_p(p, "enabled").unwrap_or(!s.state.multicam_audio_follows_video);
|
||||
s.state.multicam_audio_follows_video = on;
|
||||
Ok(json!({"enabled": on}))
|
||||
}
|
||||
|
||||
fn edit_cameras(s: &mut Session, p: &Value) -> Result<Value> {
|
||||
let cmd = "multicam.editCameras";
|
||||
let id = match item_p(p, "sequence") {
|
||||
Some(i) => i,
|
||||
None => multicam_clip_at(s, s.playhead()).map(|c| c.1.item).ok_or_else(|| bad(cmd, "need `sequence` (or a multi-camera clip at the playhead)"))?,
|
||||
};
|
||||
let q = s.project.sequence(id).ok_or_else(|| bad(cmd, "no such sequence"))?;
|
||||
if q.multicam.is_none() {
|
||||
return Err(bad(cmd, "not a multi-camera source sequence"));
|
||||
}
|
||||
let changes: Vec<Value> = p.get("cameras").and_then(Value::as_array).cloned().unwrap_or_default();
|
||||
let audio = match str_p(p, "audio") {
|
||||
Some(a) => Some(MulticamAudio::from_name(a).ok_or_else(|| bad(cmd, format!("unknown audio mode `{a}`")))?),
|
||||
None => None,
|
||||
};
|
||||
s.edit("Edit Cameras", |pr, _| {
|
||||
let q = pr.sequence_mut(id).ok_or_else(|| bad(cmd, "no such sequence"))?;
|
||||
let mc = q.multicam.as_mut().ok_or_else(|| bad(cmd, "not a multi-camera source"))?;
|
||||
for c in &changes {
|
||||
let a = c.get("angle").and_then(Value::as_u64).ok_or_else(|| bad(cmd, "each camera needs `angle`"))? as usize;
|
||||
let cam = mc.cameras.get_mut(a).ok_or_else(|| bad(cmd, format!("no angle {a}")))?;
|
||||
if let Some(n) = c.get("name").and_then(Value::as_str) {
|
||||
cam.name = n.to_string();
|
||||
}
|
||||
if let Some(e) = c.get("enabled").and_then(Value::as_bool) {
|
||||
cam.enabled = e;
|
||||
}
|
||||
}
|
||||
if let Some(a) = audio {
|
||||
mc.audio = a;
|
||||
}
|
||||
Ok(())
|
||||
})?;
|
||||
Ok(json!({"sequence": id.0}))
|
||||
}
|
||||
|
||||
/// The multi-camera clip at the playhead (or `time`), for the Multi-Camera view and agents.
|
||||
pub fn inspect_at(s: &Session, t: Tick) -> Value {
|
||||
let rec = json!({"recording": s.mcrec.active(), "cuts": s.mcrec.cuts.iter().map(|c| json!({"time": c.time.0, "angle": c.angle})).collect::<Vec<_>>()});
|
||||
let Some((track, clip)) = multicam_clip_at(s, t) else {
|
||||
return json!({"clip": null, "recorder": rec, "audioFollowsVideo": s.state.multicam_audio_follows_video});
|
||||
};
|
||||
let q = s.project.sequence(clip.item);
|
||||
let cams = q.map(|q| q.cameras()).unwrap_or_default();
|
||||
let audio_angle = s
|
||||
.active_sequence()
|
||||
.map(|seq| audio_partners(seq, &clip))
|
||||
.and_then(|ps| ps.first().and_then(|c| s.active_sequence().and_then(|seq| seq.find_item(*c)).and_then(|x| x.1.multicam.map(|m| m.angle))));
|
||||
json!({
|
||||
"clip": clip.id.0,
|
||||
"track": track.0,
|
||||
"source": clip.item.0,
|
||||
"sourceName": s.project.item(clip.item).map(|i| i.name.clone()),
|
||||
"sourceTime": clip.source_time_at(t).0,
|
||||
"angle": clip.multicam.map(|m| m.angle),
|
||||
"audioAngle": audio_angle,
|
||||
"audio": cams.audio.name(),
|
||||
"shown": cams.shown_angles(),
|
||||
"cameras": cams.cameras.iter().enumerate().map(|(i, c)| json!({"angle": i, "name": c.name, "enabled": c.enabled, "video": c.video_track.is_some()})).collect::<Vec<_>>(),
|
||||
"recorder": rec,
|
||||
"audioFollowsVideo": s.state.multicam_audio_follows_video,
|
||||
})
|
||||
}
|
||||
|
||||
fn inspect(s: &mut Session, p: &Value) -> Result<Value> {
|
||||
let t = time_p(s, p, "").unwrap_or(s.playhead());
|
||||
Ok(inspect_at(s, t))
|
||||
}
|
||||
|
|
@ -0,0 +1,677 @@
|
|||
//! Synchronisation and multi-camera editing end to end.
|
||||
//!
|
||||
//! Fixtures (ffmpeg, `target/fixtures/multicam/`): one "event" (pink noise gated into bursts, so
|
||||
//! it is not periodic) recorded by three devices that started at different times — two cameras
|
||||
//! (MJPEG + PCM `.mov` with QuickTime timecode) and an external recorder (`.wav`) — each with its
|
||||
//! own gain and independent noise. The true offsets are known to the sample.
|
||||
//!
|
||||
//! The editing tests use movies made with our own encoders (no ffmpeg).
|
||||
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::process::Command;
|
||||
|
||||
use filmcraft_media::DemoScene;
|
||||
use filmcraft_project::{ItemId, Marker, MarkerId, MarkerKind, MulticamAudio, Sequence, TrackItem};
|
||||
use filmcraft_time::{FrameRate, TICKS_PER_SECOND, Tick};
|
||||
use serde_json::{Value, json};
|
||||
|
||||
use crate::Session;
|
||||
use crate::media_test_util::{make_movie, psnr, tmp_dir};
|
||||
|
||||
const SR: i64 = 48_000;
|
||||
/// Event samples at which each recording starts (camera A, camera B, recorder).
|
||||
const START_A: i64 = 96_000;
|
||||
const START_B: i64 = 96_000 + 121_234;
|
||||
const START_REC: i64 = 48_000;
|
||||
const SAMPLE: Tick = Tick(TICKS_PER_SECOND / SR);
|
||||
|
||||
fn ff(ffmpeg: &Path, args: &[&str]) -> bool {
|
||||
Command::new(ffmpeg).args(["-v", "error", "-y"]).args(args).status().map(|s| s.success()).unwrap_or(false)
|
||||
}
|
||||
|
||||
/// The event as heard by one device: starts at event sample `start`, `secs` long, `gain`, white
|
||||
/// noise of amplitude `noise` (seed `seed`).
|
||||
fn mic_filter(start: i64, secs: f64, gain: f64, noise: f64, seed: u32) -> String {
|
||||
let len = (secs * SR as f64) as i64;
|
||||
format!(
|
||||
"anoisesrc=d=60:c=pink:r=48000:a=0.35:s=7,volume='if(lt(mod(t*2.7\\,1)\\,0.55)\\,1\\,0.08)':eval=frame,atrim=start_sample={start},asetpts=N/SR/TB,volume={gain}[s];\
|
||||
anoisesrc=d=60:c=white:r=48000:a={noise}:s={seed}[n];[s][n]amix=inputs=2:normalize=0:duration=first,atrim=end_sample={len}[a]"
|
||||
)
|
||||
}
|
||||
|
||||
/// (camera A .mov, camera B .mov, recorder .wav), or None without ffmpeg.
|
||||
fn fixtures() -> Option<(PathBuf, PathBuf, PathBuf)> {
|
||||
let ffmpeg = filmcraft_testkit::ffmpeg_or_skip("multicam fixtures")?;
|
||||
let dir = filmcraft_testkit::fixtures_dir("multicam");
|
||||
let cam = |name: &str, src: &str, start: i64, secs: f64, gain: f64, noise: f64, seed: u32, tc: &str| {
|
||||
filmcraft_testkit::fixtures::generate(&dir.join(name), |tmp| {
|
||||
let t = tmp.to_string_lossy();
|
||||
let video = format!("{src}=s=320x180:r=24:d={secs}");
|
||||
ff(
|
||||
&ffmpeg,
|
||||
&[
|
||||
"-f",
|
||||
"lavfi",
|
||||
"-i",
|
||||
&video,
|
||||
"-filter_complex",
|
||||
&mic_filter(start, secs, gain, noise, seed),
|
||||
"-map",
|
||||
"0:v",
|
||||
"-map",
|
||||
"[a]",
|
||||
"-c:v",
|
||||
"mjpeg",
|
||||
"-q:v",
|
||||
"8",
|
||||
"-c:a",
|
||||
"pcm_s16le",
|
||||
"-ar",
|
||||
"48000",
|
||||
"-ac",
|
||||
"1",
|
||||
"-timecode",
|
||||
tc,
|
||||
"-shortest",
|
||||
&t,
|
||||
],
|
||||
)
|
||||
})
|
||||
};
|
||||
let a = cam("camA.mov", "testsrc2", START_A, 12.0, 1.0, 0.02, 11, "01:00:00:00")?;
|
||||
let b = cam("camB.mov", "smptebars", START_B, 10.0, 0.25, 0.05, 12, "01:00:02:12")?;
|
||||
let rec = filmcraft_testkit::fixtures::generate(&dir.join("recorder.wav"), |tmp| {
|
||||
let t = tmp.to_string_lossy();
|
||||
ff(
|
||||
&ffmpeg,
|
||||
&[
|
||||
"-f",
|
||||
"lavfi",
|
||||
"-i",
|
||||
"anullsrc=r=48000",
|
||||
"-filter_complex",
|
||||
&mic_filter(START_REC, 16.0, 1.5, 0.003, 13),
|
||||
"-map",
|
||||
"[a]",
|
||||
"-c:a",
|
||||
"pcm_s16le",
|
||||
"-ac",
|
||||
"1",
|
||||
&t,
|
||||
],
|
||||
)
|
||||
})?;
|
||||
Some((a, b, rec))
|
||||
}
|
||||
|
||||
fn import(s: &mut Session, files: &[&Path]) -> Vec<ItemId> {
|
||||
let paths: Vec<String> = files.iter().map(|p| p.to_string_lossy().into_owned()).collect();
|
||||
let r = s.execute("file.import", json!({"paths": paths})).unwrap();
|
||||
r["items"].as_array().unwrap().iter().map(|v| ItemId(v.as_u64().unwrap())).collect()
|
||||
}
|
||||
|
||||
fn ids(v: &[ItemId]) -> Value {
|
||||
json!(v.iter().map(|i| i.0).collect::<Vec<_>>())
|
||||
}
|
||||
|
||||
/// Run a Project-panel command on `items` (selected first, as in the UI).
|
||||
fn on_items(s: &mut Session, cmd: &str, items: &[ItemId], mut p: Value) -> crate::Result<Value> {
|
||||
s.execute("project.select", json!({"items": ids(items)})).unwrap();
|
||||
p["items"] = ids(items);
|
||||
s.execute(cmd, p)
|
||||
}
|
||||
|
||||
/// Timeline time at which event sample `m` plays in clip `ti` (whose media starts at event sample
|
||||
/// `start`).
|
||||
fn event_time(ti: &TrackItem, start: i64, m: i64) -> Tick {
|
||||
let media = Tick::from_units(m - start, SR);
|
||||
ti.start + (media - ti.source_in)
|
||||
}
|
||||
|
||||
fn find_clip(q: &Sequence, item: ItemId, video: bool) -> &TrackItem {
|
||||
let tracks = if video { &q.video_tracks } else { &q.audio_tracks };
|
||||
tracks.iter().flat_map(|t| t.items.iter()).find(|i| i.item == item).unwrap_or_else(|| panic!("no clip of {item:?}"))
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn audio_sync_multicam_sample_accurate() {
|
||||
let Some((a, b, rec)) = fixtures() else { return };
|
||||
let mut s = Session::default();
|
||||
let items = import(&mut s, &[&a, &b, &rec]);
|
||||
let t0 = std::time::Instant::now();
|
||||
let r =
|
||||
on_items(&mut s, "clip.createMulticam", &items, json!({"method": "audio", "audio": "switch", "name": "Interview MC", "processedBin": true})).unwrap();
|
||||
let ms = t0.elapsed().as_secs_f64() * 1000.0;
|
||||
let seq_id = ItemId(r["sequence"].as_u64().unwrap());
|
||||
let q = s.project.sequence(seq_id).unwrap().clone();
|
||||
// structure: two cameras with video, an audio-only recorder, switching audio
|
||||
let mc = q.multicam.as_ref().unwrap();
|
||||
assert_eq!(mc.cameras.len(), 3);
|
||||
assert_eq!(q.video_tracks.len(), 2);
|
||||
assert_eq!(q.audio_tracks.len(), 3);
|
||||
assert_eq!(mc.audio, MulticamAudio::SwitchAudio);
|
||||
assert_eq!(mc.cameras[0].name, "camA.mov");
|
||||
assert!(mc.cameras[2].video_track.is_none() && mc.cameras[2].audio_tracks.len() == 1, "the recorder is an audio-only source");
|
||||
assert_eq!(q.video_tracks[0].name, "camA.mov");
|
||||
assert_eq!(s.project.item(seq_id).unwrap().type_label(), "Multi-Camera Source Sequence");
|
||||
// processed clips moved into their bin
|
||||
let bin = s.project.root.children.iter().find_map(|c| match c {
|
||||
filmcraft_project::BinEntry::Bin(b) if b.name == "Processed Clips" => Some(b.clone()),
|
||||
_ => None,
|
||||
});
|
||||
assert_eq!(bin.map(|b| b.children.len()), Some(3));
|
||||
// every recording plays event sample m at the same timeline time (±1 sample)
|
||||
let clips = [(find_clip(&q, items[0], false), START_A), (find_clip(&q, items[1], false), START_B), (find_clip(&q, items[2], false), START_REC)];
|
||||
let m = 300_000;
|
||||
let t_ref = event_time(clips[0].0, clips[0].1, m);
|
||||
let mut worst = Tick::ZERO;
|
||||
for (c, start) in &clips[1..] {
|
||||
let d = (event_time(c, *start, m) - t_ref).abs();
|
||||
worst = worst.max(d);
|
||||
}
|
||||
// video clips sit on frame boundaries and keep the same alignment as their audio
|
||||
for (k, item) in items[..2].iter().enumerate() {
|
||||
let v = find_clip(&q, *item, true);
|
||||
assert_eq!(v.start, q.settings.frame_rate.snap(v.start), "frame-aligned");
|
||||
assert_eq!((v.start, v.source_in), (clips[k].0.start, clips[k].0.source_in));
|
||||
}
|
||||
let report = &r["clips"];
|
||||
eprintln!(
|
||||
"audio sync: worst error {:.2} samples over 3 recordings; B lag {} (truth {}), recorder lag {} (truth {}); {ms:.0} ms",
|
||||
worst.0 as f64 / SAMPLE.0 as f64,
|
||||
report[1]["audio"]["lagSamples"],
|
||||
START_B - START_A,
|
||||
report[2]["audio"]["lagSamples"],
|
||||
START_REC - START_A
|
||||
);
|
||||
assert!(worst <= SAMPLE, "worst {} samples", worst.0 as f64 / SAMPLE.0 as f64);
|
||||
assert!(report[1]["audio"]["reliable"].as_bool().unwrap() && report[2]["audio"]["reliable"].as_bool().unwrap(), "{report}");
|
||||
// undo removes the sequence and restores the bins
|
||||
s.execute("edit.undo", json!({})).unwrap();
|
||||
assert!(s.project.sequence(seq_id).is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn timecode_marker_and_in_point_sync_within_one_frame() {
|
||||
let Some((a, b, _)) = fixtures() else { return };
|
||||
let mut s = Session::default();
|
||||
let items = import(&mut s, &[&a, &b]);
|
||||
let truth = Tick::from_units(START_B - START_A, SR);
|
||||
let rate = FrameRate::FPS_24;
|
||||
let offset_of = |s: &Session, r: &Value| -> Tick {
|
||||
let q = s.project.sequence(ItemId(r["sequence"].as_u64().unwrap())).unwrap();
|
||||
let (va, vb) = (find_clip(q, items[0], true), find_clip(q, items[1], true));
|
||||
// timeline time of media 0 of each camera; B starts `truth` after A in the event
|
||||
(vb.start - vb.source_in) - (va.start - va.source_in)
|
||||
};
|
||||
// timecode: A 01:00:00:00, B 01:00:02:12 = 2.5 s later (the true offset is 2.5257 s)
|
||||
let r = on_items(&mut s, "clip.createMulticam", &items, json!({"method": "timecode"})).unwrap();
|
||||
let off = offset_of(&s, &r);
|
||||
assert_eq!(off, rate.tick_of(60), "timecode offset is exactly 60 frames");
|
||||
assert!((off - truth).abs() <= rate.frame_duration(), "within a frame of the truth");
|
||||
// ignoring hours gives the same result here
|
||||
let r = on_items(&mut s, "clip.createMulticam", &items, json!({"method": "timecode", "ignoreHours": true})).unwrap();
|
||||
assert_eq!(offset_of(&s, &r), rate.tick_of(60));
|
||||
// clip markers: the same event moment marked in both clips (frame accuracy)
|
||||
let event_frame_a = 72; // 3 s into camera A
|
||||
let at_b = rate.tick_of(event_frame_a) - truth;
|
||||
let mut p = (*s.project).clone();
|
||||
for (it, t) in [(items[0], rate.tick_of(event_frame_a)), (items[1], rate.snap(at_b))] {
|
||||
let m = p.item_mut(it).unwrap().as_media_mut().unwrap();
|
||||
m.markers.push(Marker {
|
||||
id: MarkerId(900 + it.0),
|
||||
start: t,
|
||||
duration: Tick::ZERO,
|
||||
name: "clap".into(),
|
||||
comment: String::new(),
|
||||
kind: MarkerKind::Comment,
|
||||
color: filmcraft_project::Label::Rose,
|
||||
});
|
||||
}
|
||||
s.project = std::sync::Arc::new(p);
|
||||
let r = on_items(&mut s, "clip.createMulticam", &items, json!({"method": "marker", "marker": "clap"})).unwrap();
|
||||
let off = offset_of(&s, &r);
|
||||
assert!((off - truth).abs() <= rate.frame_duration(), "marker sync {} vs {}", off.0, truth.0);
|
||||
// In points: A's In at 2.5 s, B's at 0 → B starts 2.5 s after A
|
||||
let mut p = (*s.project).clone();
|
||||
p.item_mut(items[0]).unwrap().as_media_mut().unwrap().mark_in = Some(rate.tick_of(60));
|
||||
s.project = std::sync::Arc::new(p);
|
||||
let r = on_items(&mut s, "clip.createMulticam", &items, json!({"method": "in"})).unwrap();
|
||||
assert_eq!(offset_of(&s, &r), rate.tick_of(60));
|
||||
// Out points: A ends at 12 s, B at 10 s, both Outs unset → ends aligned: B media 0 at 2 s
|
||||
let r = on_items(&mut s, "clip.createMulticam", &items, json!({"method": "out"})).unwrap();
|
||||
assert_eq!(offset_of(&s, &r), rate.tick_of(48));
|
||||
// a missing marker is an error, not a guess
|
||||
assert!(on_items(&mut s, "clip.createMulticam", &items, json!({"method": "marker", "marker": "nope"})).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn merge_clips_by_audio() {
|
||||
let Some((a, _, rec)) = fixtures() else { return };
|
||||
let mut s = Session::default();
|
||||
let items = import(&mut s, &[&a, &rec]);
|
||||
let r = on_items(&mut s, "clip.mergeClips", &items, json!({"method": "audio", "removeVideoAudio": true})).unwrap();
|
||||
let id = ItemId(r["item"].as_u64().unwrap());
|
||||
let q = s.project.sequence(id).unwrap();
|
||||
assert_eq!(s.project.item(id).unwrap().type_label(), "Merged Clip");
|
||||
assert_eq!(s.project.item(id).unwrap().name, "camA.mov - Merged");
|
||||
assert_eq!((q.video_tracks.len(), q.audio_tracks.len()), (1, 1), "camera audio removed");
|
||||
let v = find_clip(q, items[0], true);
|
||||
let ar = find_clip(q, items[1], false);
|
||||
assert_eq!(v.link, ar.link, "merged parts are linked");
|
||||
// the recorder's event sample m plays with the camera's
|
||||
let m = 400_000;
|
||||
let d = (event_time(ar, START_REC, m) - (v.start + (Tick::from_units(m - START_A, SR) - v.source_in))).abs();
|
||||
assert!(d <= SAMPLE, "merge offset error {} samples", d.0 as f64 / SAMPLE.0 as f64);
|
||||
assert!(on_items(&mut s, "clip.mergeClips", &items[..1], json!({})).is_err(), "needs audio clips");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn timeline_synchronize_by_audio_and_timecode() {
|
||||
let Some((a, b, rec)) = fixtures() else { return };
|
||||
let mut s = Session::default();
|
||||
let items = import(&mut s, &[&a, &b, &rec]);
|
||||
let mut p = (*s.project).clone();
|
||||
let rate = FrameRate::FPS_24;
|
||||
let seq = p.new_sequence("Edit", filmcraft_project::SequenceSettings { width: 320, height: 180, frame_rate: rate, ..Default::default() }, 2, 3, None);
|
||||
// camera B on V2/A2 at 20 s, recorder on A3 at 1 s, camera A on V1/A1 at 5 s
|
||||
let place = |p: &mut filmcraft_project::Project, it: ItemId, at: i64, v: Option<usize>, a: usize| {
|
||||
let d = p.item(it).unwrap().duration();
|
||||
let range = filmcraft_time::TimeRange::new(Tick::ZERO, d);
|
||||
let link = p.alloc_id();
|
||||
if let Some(vt) = v {
|
||||
let mut vi = p.make_track_item(it, filmcraft_project::TrackKind::Video, rate.tick_of(at), range, rate).unwrap();
|
||||
vi.link = Some(link);
|
||||
p.sequence_mut(seq).unwrap().video_tracks[vt].items.push(vi);
|
||||
}
|
||||
let mut ai = p.make_track_item(it, filmcraft_project::TrackKind::Audio, rate.tick_of(at), range, rate).unwrap();
|
||||
ai.link = Some(link);
|
||||
p.sequence_mut(seq).unwrap().audio_tracks[a].items.push(ai);
|
||||
};
|
||||
place(&mut p, items[0], 120, Some(0), 0);
|
||||
place(&mut p, items[1], 480, Some(1), 1);
|
||||
place(&mut p, items[2], 24, None, 2);
|
||||
s.project = std::sync::Arc::new(p);
|
||||
s.state.active_sequence = Some(seq);
|
||||
let all: Vec<u64> = s.project.sequence(seq).unwrap().all_tracks().flat_map(|t| t.items.iter().map(|i| i.id.0)).collect();
|
||||
s.execute("timeline.select", json!({"clips": all})).unwrap();
|
||||
let r = s.execute("clip.synchronize", json!({"method": "audio"})).unwrap();
|
||||
assert_eq!(r["moved"], 2, "{r}");
|
||||
let q = s.active_sequence().unwrap().clone();
|
||||
let ca = find_clip(&q, items[0], false);
|
||||
assert_eq!(ca.start, rate.tick_of(120), "the reference (V1) stays put");
|
||||
let m = 350_000;
|
||||
for (it, start) in [(items[1], START_B), (items[2], START_REC)] {
|
||||
let c = find_clip(&q, it, false);
|
||||
let d = (event_time(c, start, m) - event_time(ca, START_A, m)).abs();
|
||||
assert!(d <= SAMPLE, "timeline sync error {} samples", d.0 as f64 / SAMPLE.0 as f64);
|
||||
assert_eq!(c.start, rate.snap(c.start), "moved clips stay on frames");
|
||||
}
|
||||
let vb = find_clip(&q, items[1], true);
|
||||
let ab = find_clip(&q, items[1], false);
|
||||
assert_eq!((vb.start, vb.source_in), (ab.start, ab.source_in), "linked partners moved together");
|
||||
s.execute("edit.undo", json!({})).unwrap();
|
||||
assert_eq!(find_clip(s.active_sequence().unwrap(), items[1], false).start, rate.tick_of(480));
|
||||
// timecode: B lands exactly 60 frames after A
|
||||
s.execute("timeline.select", json!({"clips": all})).unwrap();
|
||||
let r = s.execute("clip.synchronize", json!({"method": "timecode", "track": "V1"})).unwrap();
|
||||
assert!(r["moved"].as_u64().unwrap() >= 1);
|
||||
let q = s.active_sequence().unwrap();
|
||||
assert_eq!(find_clip(q, items[1], true).start - find_clip(q, items[0], true).start, rate.tick_of(60));
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------------ editing (our own media)
|
||||
|
||||
const W: u32 = 160;
|
||||
const H: u32 = 90;
|
||||
|
||||
/// Three 4-second cameras (different scenes, own audio), a multi-camera source sequence by In
|
||||
/// points, and a sequence holding one multi-camera clip on V1/A1. Returns (session, camera items,
|
||||
/// source sequence, edit sequence).
|
||||
fn mc_session(audio: &str) -> (Session, Vec<ItemId>, ItemId, ItemId) {
|
||||
let dir = tmp_dir("mc-edit");
|
||||
let files: Vec<PathBuf> = ["a.mov", "b.mov", "c.mov"].iter().map(|f| dir.join(f)).collect();
|
||||
for (f, sc) in files.iter().zip([DemoScene::OceanSunset, DemoScene::CityNight, DemoScene::Aurora]) {
|
||||
make_movie(f, sc, W, H, 96);
|
||||
}
|
||||
let mut s = Session::default();
|
||||
let refs: Vec<&Path> = files.iter().map(|p| p.as_path()).collect();
|
||||
let items = import(&mut s, &refs);
|
||||
let r = on_items(&mut s, "clip.createMulticam", &items, json!({"method": "in", "audio": audio, "cameraNames": "track"})).unwrap();
|
||||
let src = ItemId(r["sequence"].as_u64().unwrap());
|
||||
let mut p = (*s.project).clone();
|
||||
let edit =
|
||||
p.new_sequence("Cut", filmcraft_project::SequenceSettings { width: W, height: H, frame_rate: FrameRate::FPS_24, ..Default::default() }, 2, 2, None);
|
||||
s.project = std::sync::Arc::new(p);
|
||||
s.state.active_sequence = Some(edit);
|
||||
s.execute("source.open", json!({"item": src.0})).unwrap();
|
||||
s.execute("source.overwrite", json!({})).unwrap();
|
||||
s.set_playhead(Tick::ZERO);
|
||||
(s, items, src, edit)
|
||||
}
|
||||
|
||||
fn v1(s: &Session) -> Vec<(i64, u32)> {
|
||||
let q = s.active_sequence().unwrap();
|
||||
let rate = q.settings.frame_rate;
|
||||
q.video_tracks[0].items.iter().map(|i| (rate.frame_at(i.start), i.multicam.unwrap().angle)).collect()
|
||||
}
|
||||
|
||||
fn a1(s: &Session) -> Vec<(i64, u32)> {
|
||||
let q = s.active_sequence().unwrap();
|
||||
let rate = q.settings.frame_rate;
|
||||
q.audio_tracks[0].items.iter().map(|i| (rate.frame_at(i.start), i.multicam.unwrap().angle)).collect()
|
||||
}
|
||||
|
||||
fn frame(s: &mut Session, f: i64) -> Vec<u8> {
|
||||
let t = s.sequence_rate().tick_of(f);
|
||||
s.set_playhead(t);
|
||||
s.render_program(1.0).unwrap().over_black_rgba8()
|
||||
}
|
||||
|
||||
/// The frame of camera `item` at its media frame `f`, rendered alone.
|
||||
fn camera_frame(s: &Session, item: ItemId, f: i64) -> Vec<u8> {
|
||||
let provider = s.media.provider(s.project.clone(), s.services.clone());
|
||||
filmcraft_render::render_item(&s.project, item, FrameRate::FPS_24.tick_of(f), 1.0, &provider).unwrap().over_black_rgba8()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn multicam_clip_structure_and_render() {
|
||||
let (mut s, items, src, _) = mc_session("camera1");
|
||||
let q = s.project.sequence(src).unwrap();
|
||||
let mc = q.multicam.as_ref().unwrap();
|
||||
assert_eq!(mc.cameras.iter().map(|c| c.name.as_str()).collect::<Vec<_>>(), ["Camera 1", "Camera 2", "Camera 3"]);
|
||||
assert_eq!(mc.audio, MulticamAudio::Camera1);
|
||||
assert_eq!((q.video_tracks.len(), q.audio_tracks.len()), (3, 3));
|
||||
// edited in as an enabled multi-camera clip on angle 0 (video and audio)
|
||||
assert_eq!(v1(&s), [(0, 0)]);
|
||||
assert_eq!(a1(&s), [(0, 0)]);
|
||||
let inspect = s.execute("multicam.inspect", json!({"frame": 10})).unwrap();
|
||||
assert_eq!(inspect["source"], src.0);
|
||||
assert_eq!(inspect["shown"], json!([0, 1, 2]));
|
||||
// the clip shows angle 0 only (not the composite, whose top track is camera 3)
|
||||
assert!(psnr(&frame(&mut s, 10), &camera_frame(&s, items[0], 10)) > 40.0);
|
||||
s.execute("multicam.switchAngle", json!({"camera": 3})).unwrap();
|
||||
assert_eq!(v1(&s), [(0, 2)]);
|
||||
assert!(psnr(&frame(&mut s, 10), &camera_frame(&s, items[2], 10)) > 40.0);
|
||||
// disabling multi-camera shows the plain nest (top track = camera 3 here, so pick camera 2)
|
||||
s.execute("multicam.switchAngle", json!({"camera": 2})).unwrap();
|
||||
let clip = s.active_sequence().unwrap().video_tracks[0].items[0].id;
|
||||
s.execute("timeline.select", json!({"clips": [clip.0]})).unwrap();
|
||||
s.execute("clip.multicamEnable", json!({"enabled": false})).unwrap();
|
||||
assert!(psnr(&frame(&mut s, 10), &camera_frame(&s, items[2], 10)) > 40.0, "plain nest: the top track");
|
||||
s.execute("clip.multicamEnable", json!({})).unwrap();
|
||||
assert!(psnr(&frame(&mut s, 10), &camera_frame(&s, items[1], 10)) > 40.0, "re-enabled, angle remembered");
|
||||
// the GPU plan draws the angle's media frame directly (one layer, no CPU pass) and matches
|
||||
let provider = s.media.provider(s.project.clone(), s.services.clone());
|
||||
let edit_id = s.state.active_sequence.unwrap();
|
||||
let t10 = FrameRate::FPS_24.tick_of(10);
|
||||
let plan = filmcraft_render::plan::plan_frame(&s.project, edit_id, t10, filmcraft_render::RenderOptions::default(), &provider);
|
||||
match &plan {
|
||||
filmcraft_render::plan::FramePlan::Layers { layers, .. } => {
|
||||
assert_eq!(layers.len(), 1);
|
||||
assert!(!matches!(layers[0].frame.data, filmcraft_frame::PixelData::RgbaF32(_)), "a decoded frame, not a CPU render");
|
||||
}
|
||||
_ => panic!("expected layers"),
|
||||
}
|
||||
let via_plan = filmcraft_render::plan::execute_cpu(&plan);
|
||||
s.set_playhead(t10);
|
||||
let reference = s.render_program(1.0).unwrap();
|
||||
assert!(psnr(&via_plan.over_black_rgba8(), &reference.over_black_rgba8()) > 40.0);
|
||||
// grid of the angles
|
||||
let provider = s.media.provider(s.project.clone(), s.services.clone());
|
||||
let (grid, angles) = filmcraft_render::multicam::render_grid(&s.project, src, Tick::ZERO, 0.25, &provider).unwrap();
|
||||
assert_eq!(angles, [0, 1, 2]);
|
||||
assert_eq!((grid.w, grid.h), (2 * 40, 2 * 23));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn switch_audio_follows_video() {
|
||||
let (mut s, _, src, _) = mc_session("switch");
|
||||
// audio follows video off (Premiere's default): only the picture switches
|
||||
s.execute("multicam.switchAngle", json!({"camera": 2, "frame": 5})).unwrap();
|
||||
assert_eq!((v1(&s), a1(&s)), (vec![(0, 1)], vec![(0, 0)]));
|
||||
s.execute("multicam.audioFollowsVideo", json!({"enabled": true})).unwrap();
|
||||
s.execute("multicam.switchAngle", json!({"camera": 3})).unwrap();
|
||||
assert_eq!((v1(&s), a1(&s)), (vec![(0, 2)], vec![(0, 2)]));
|
||||
// video only (Ctrl-click) leaves the audio alone; audio only switches just the audio
|
||||
s.execute("multicam.switchAngle", json!({"camera": 1, "videoOnly": true})).unwrap();
|
||||
assert_eq!((v1(&s), a1(&s)), (vec![(0, 0)], vec![(0, 2)]));
|
||||
s.execute("multicam.switchAngle", json!({"camera": 2, "audioOnly": true})).unwrap();
|
||||
assert_eq!((v1(&s), a1(&s)), (vec![(0, 0)], vec![(0, 1)]));
|
||||
// switching audio plays the selected angle's track of the source
|
||||
let q = s.project.sequence(src).unwrap();
|
||||
let on = q.multicam.as_ref().unwrap().audible_tracks(Some(1));
|
||||
assert_eq!(on, q.multicam.as_ref().unwrap().cameras[1].audio_tracks);
|
||||
let provider = s.media.provider(s.project.clone(), s.services.clone());
|
||||
let edit_seq = s.active_sequence().unwrap().clone();
|
||||
let mix = filmcraft_render::audio::mix_sequence(&s.project, &edit_seq, 24_000, 4_800, &provider);
|
||||
let only_b = filmcraft_render::audio::mix_sequence(&s.project, &q.with_angle_audio(Some(1)), 24_000, 4_800, &provider);
|
||||
let err = mix.channels[0].iter().zip(&only_b.channels[0]).map(|(a, b)| (a - b).abs()).fold(0.0f32, f32::max);
|
||||
assert!(err < 1e-4, "multi-camera audio = camera 2's track ({err})");
|
||||
assert!(mix.channels[0].iter().any(|v| v.abs() > 1e-3), "audible");
|
||||
// undo steps back one switch at a time
|
||||
s.execute("edit.undo", json!({})).unwrap();
|
||||
assert_eq!(a1(&s), [(0, 2)]);
|
||||
}
|
||||
|
||||
/// Live switching from a simulated playback clock: play 0 → 84 at 24 fps, press 2 at frame 24,
|
||||
/// 3 at frame 48, 3 again at 60 (no new edit), 1 at 72; stop at 84.
|
||||
#[test]
|
||||
fn live_switch_recording_from_playback_clock() {
|
||||
let (mut s, items, _, _) = mc_session("switch");
|
||||
s.execute("multicam.audioFollowsVideo", json!({"enabled": true})).unwrap();
|
||||
let undo_before = s.history.undo.len();
|
||||
let presses: [(i64, u64); 4] = [(24, 2), (48, 3), (60, 3), (72, 1)];
|
||||
let rate = s.sequence_rate();
|
||||
s.execute("multicam.recordStart", json!({"frame": 0})).unwrap();
|
||||
for f in 0..=84i64 {
|
||||
// the clock: one tick per frame
|
||||
s.set_playhead(rate.tick_of(f));
|
||||
if let Some((_, cam)) = presses.iter().find(|(pf, _)| *pf == f) {
|
||||
s.execute("multicam.cut", json!({"camera": cam})).unwrap();
|
||||
// the cut is live: the program shows the new angle from this frame on
|
||||
let want = items[*cam as usize - 1];
|
||||
assert!(psnr(&frame(&mut s, f + 2), &camera_frame(&s, want, f + 2)) > 40.0, "live cut at {f}");
|
||||
s.set_playhead(rate.tick_of(f));
|
||||
}
|
||||
}
|
||||
let r = s.execute("multicam.recordStop", json!({"frame": 84})).unwrap();
|
||||
assert_eq!(r["cuts"], 4);
|
||||
// 0–24 cam 1, 24–48 cam 2, 48–72 cam 3 (the second press of 3 adds no edit), 72–96 cam 1
|
||||
// (healed with the untouched rest of the clip after the stop point)
|
||||
let expect = vec![(0, 0), (24, 1), (48, 2), (72, 0)];
|
||||
assert_eq!(v1(&s), expect);
|
||||
assert_eq!(a1(&s), expect, "audio followed the video");
|
||||
assert_eq!(s.history.undo.len(), undo_before + 1, "one undo step for the whole pass");
|
||||
assert_eq!(s.history.undo.last().unwrap().0, "Record Multi-Camera");
|
||||
// pieces are continuous in source time
|
||||
let q = s.active_sequence().unwrap();
|
||||
for w in q.video_tracks[0].items.windows(2) {
|
||||
assert_eq!(w[0].source_out(), w[1].source_in);
|
||||
assert!(w[0].link.is_some());
|
||||
}
|
||||
// a second pass changes only its own range and is its own undo step
|
||||
s.execute("multicam.recordStart", json!({"frame": 30})).unwrap();
|
||||
s.execute("multicam.cut", json!({"camera": 3, "frame": 30})).unwrap();
|
||||
s.execute("multicam.recordStop", json!({"frame": 40})).unwrap();
|
||||
assert_eq!(v1(&s), vec![(0, 0), (24, 1), (30, 2), (40, 1), (48, 2), (72, 0)]);
|
||||
s.execute("edit.undo", json!({})).unwrap();
|
||||
assert_eq!(v1(&s), expect);
|
||||
s.execute("edit.undo", json!({})).unwrap();
|
||||
assert_eq!(v1(&s), vec![(0, 0)]);
|
||||
// stopped: a key press switches the clip at the playhead instead of cutting
|
||||
s.execute("multicam.cut", json!({"camera": 2, "frame": 10})).unwrap();
|
||||
assert_eq!(v1(&s), vec![(0, 1)]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn flatten_replaces_with_camera_clips() {
|
||||
let (mut s, items, _, _) = mc_session("switch");
|
||||
s.execute("multicam.audioFollowsVideo", json!({"enabled": true})).unwrap();
|
||||
s.execute("multicam.recordStart", json!({"frame": 0})).unwrap();
|
||||
s.execute("multicam.cut", json!({"camera": 2, "frame": 30})).unwrap();
|
||||
s.execute("multicam.cut", json!({"camera": 3, "frame": 60})).unwrap();
|
||||
s.execute("multicam.recordStop", json!({"frame": 96})).unwrap();
|
||||
let before: Vec<Vec<u8>> = [5, 35, 70].iter().map(|f| frame(&mut s, *f)).collect();
|
||||
let all: Vec<u64> = s.active_sequence().unwrap().all_tracks().flat_map(|t| t.items.iter().map(|i| i.id.0)).collect();
|
||||
s.execute("timeline.select", json!({"clips": all})).unwrap();
|
||||
let r = s.execute("clip.multicamFlatten", json!({})).unwrap();
|
||||
assert_eq!(r["flattened"], 6);
|
||||
let q = s.active_sequence().unwrap().clone();
|
||||
let v: Vec<(ItemId, i64, i64)> = q.video_tracks[0].items.iter().map(|i| (i.item, i.start.0, i.source_in.0)).collect();
|
||||
let rate = q.settings.frame_rate;
|
||||
assert_eq!(v, vec![(items[0], 0, 0), (items[1], rate.tick_of(30).0, rate.tick_of(30).0), (items[2], rate.tick_of(60).0, rate.tick_of(60).0)]);
|
||||
assert!(q.all_tracks().flat_map(|t| t.items.iter()).all(|i| i.multicam.is_none()));
|
||||
let a: Vec<ItemId> = q.audio_tracks[0].items.iter().map(|i| i.item).collect();
|
||||
assert_eq!(a, vec![items[0], items[1], items[2]], "audio flattened to the switched cameras");
|
||||
for (i, f) in [5, 35, 70].iter().enumerate() {
|
||||
assert!(psnr(&before[i], &frame(&mut s, *f)) > 45.0, "flattened frame {f} looks the same");
|
||||
}
|
||||
// linked camera video/audio stay linked
|
||||
assert_eq!(q.video_tracks[0].items[1].link, q.audio_tracks[0].items[1].link);
|
||||
s.execute("edit.undo", json!({})).unwrap();
|
||||
assert_eq!(v1(&s).len(), 3);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn edit_cameras_and_save_load() {
|
||||
let (mut s, _, src, _) = mc_session("all");
|
||||
s.execute("multicam.editCameras", json!({"sequence": src.0, "cameras": [{"angle": 1, "name": "Wide", "enabled": false}], "audio": "switch"})).unwrap();
|
||||
let mc = s.project.sequence(src).unwrap().multicam.clone().unwrap();
|
||||
assert_eq!((mc.cameras[1].name.as_str(), mc.cameras[1].enabled, mc.audio), ("Wide", false, MulticamAudio::SwitchAudio));
|
||||
assert_eq!(mc.shown_angles(), [0, 2]);
|
||||
// keys pick from the shown cameras: camera 2 is angle 2 now
|
||||
s.execute("multicam.switchAngle", json!({"camera": 2})).unwrap();
|
||||
assert_eq!(v1(&s), [(0, 2)]);
|
||||
assert!(s.execute("multicam.switchAngle", json!({"camera": 3})).is_err());
|
||||
let dir = tmp_dir("mc-save");
|
||||
let path = dir.join("mc.fcproj").to_string_lossy().into_owned();
|
||||
s.execute("file.save", json!({"path": path})).unwrap();
|
||||
let bytes = std::fs::read(&path).unwrap();
|
||||
let doc: Value = serde_json::from_slice(&bytes).unwrap();
|
||||
assert_eq!(doc["schema_version"], filmcraft_format::SCHEMA_VERSION);
|
||||
let mut t = Session::default();
|
||||
t.execute("file.open", json!({"path": path})).unwrap();
|
||||
assert_eq!(*t.project, *s.project, "multi-camera data survives save/load");
|
||||
assert_eq!(t.project.sequence(src).unwrap().multicam.as_ref().unwrap().cameras[1].name, "Wide");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn any_nest_can_be_multicam() {
|
||||
// a plain nested sequence with two video tracks: angles are its tracks
|
||||
let (mut s, items, _, _) = mc_session("camera1");
|
||||
let mut p = (*s.project).clone();
|
||||
let rate = FrameRate::FPS_24;
|
||||
let nest = p.new_sequence("Nest", filmcraft_project::SequenceSettings { width: W, height: H, frame_rate: rate, ..Default::default() }, 2, 0, None);
|
||||
for (k, it) in items[..2].iter().enumerate() {
|
||||
let mut v = p
|
||||
.make_track_item(*it, filmcraft_project::TrackKind::Video, Tick::ZERO, filmcraft_time::TimeRange::new(Tick::ZERO, rate.tick_of(48)), rate)
|
||||
.unwrap();
|
||||
for e in &mut v.effects {
|
||||
filmcraft_project::resolve_auto_points(e, (W, H), (W, H));
|
||||
}
|
||||
p.sequence_mut(nest).unwrap().video_tracks[k].items.push(v);
|
||||
}
|
||||
let edit = s.state.active_sequence.unwrap();
|
||||
let mut v =
|
||||
p.make_track_item(nest, filmcraft_project::TrackKind::Video, Tick::ZERO, filmcraft_time::TimeRange::new(Tick::ZERO, rate.tick_of(48)), rate).unwrap();
|
||||
assert!(v.multicam.is_none(), "plain nests start as plain nests");
|
||||
for e in &mut v.effects {
|
||||
filmcraft_project::resolve_auto_points(e, (W, H), (W, H));
|
||||
}
|
||||
let clip = v.id;
|
||||
let q = p.sequence_mut(edit).unwrap();
|
||||
q.video_tracks[1].items.push(v);
|
||||
s.project = std::sync::Arc::new(p);
|
||||
assert!(psnr(&frame(&mut s, 3), &camera_frame(&s, items[1], 3)) > 40.0, "nest: top track");
|
||||
s.execute("timeline.select", json!({"clips": [clip.0]})).unwrap();
|
||||
s.execute("clip.multicamEnable", json!({})).unwrap();
|
||||
assert!(psnr(&frame(&mut s, 3), &camera_frame(&s, items[0], 3)) > 40.0, "angle 0 = track V1 of the nest");
|
||||
s.execute("multicam.switchAngle", json!({"clips": [clip.0], "angle": 1})).unwrap();
|
||||
assert!(psnr(&frame(&mut s, 3), &camera_frame(&s, items[1], 3)) > 40.0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn camera_keys_cut_and_select() {
|
||||
let (mut s, _, _, _) = mc_session("camera1");
|
||||
// Ctrl+2 at frame 30: an edit there, camera 2 after it
|
||||
s.set_playhead(FrameRate::FPS_24.tick_of(30));
|
||||
s.execute("multicam.cutToCamera2", json!({})).unwrap();
|
||||
assert_eq!(v1(&s), [(0, 0), (30, 1)]);
|
||||
// Ctrl+3 at the existing edit: no new edit, the right piece switches
|
||||
s.execute("multicam.cutToCamera3", json!({})).unwrap();
|
||||
assert_eq!(v1(&s), [(0, 0), (30, 2)]);
|
||||
// 2 while stopped switches the clip under the playhead
|
||||
s.set_playhead(FrameRate::FPS_24.tick_of(10));
|
||||
s.execute("multicam.selectCamera2", json!({})).unwrap();
|
||||
assert_eq!(v1(&s), [(0, 1), (30, 2)]);
|
||||
// the keys are bound by default
|
||||
let keys: Vec<(String, String)> =
|
||||
s.shortcuts.bindings.iter().filter(|b| b.command.starts_with("multicam.")).map(|b| (b.command.clone(), b.keys.clone())).collect();
|
||||
assert!(keys.contains(&("multicam.selectCamera1".into(), "1".into())), "{keys:?}");
|
||||
assert!(keys.contains(&("multicam.cutToCamera9".into(), "Ctrl+9".into())), "{keys:?}");
|
||||
}
|
||||
|
||||
/// Process CPU time in seconds (user + system), for the perf test.
|
||||
fn process_cpu() -> f64 {
|
||||
let out = Command::new("ps").args(["-o", "cputime=", "-p", &std::process::id().to_string()]).output().ok();
|
||||
let txt = out.map(|o| String::from_utf8_lossy(&o.stdout).trim().to_string()).unwrap_or_default();
|
||||
// [[dd-]hh:]mm:ss(.ss)
|
||||
txt.split(':').fold(0.0, |acc, p| acc * 60.0 + p.trim().parse::<f64>().unwrap_or(0.0))
|
||||
}
|
||||
|
||||
/// Multi-Camera view cost: a 2×2 grid of four 1080p H.264 angles at ¼ scale per cell, played
|
||||
/// sequentially, against the program (one angle at ½ scale). Run with
|
||||
/// `cargo test --release -p filmcraft-engine --lib multicam_tests::perf -- --ignored --nocapture`.
|
||||
#[test]
|
||||
#[ignore]
|
||||
fn perf_grid_of_four_1080p_angles() {
|
||||
let Some(ffmpeg) = filmcraft_testkit::ffmpeg_or_skip("multicam perf") else { return };
|
||||
let dir = filmcraft_testkit::fixtures_dir("multicam");
|
||||
let srcs = ["testsrc2", "smptehdbars", "mandelbrot", "rgbtestsrc"];
|
||||
let files: Vec<PathBuf> = srcs
|
||||
.iter()
|
||||
.map(|src| {
|
||||
filmcraft_testkit::fixtures::generate(&dir.join(format!("perf-{src}.mp4")), |tmp| {
|
||||
let v = format!("{src}=s=1920x1080:r=24");
|
||||
ff(
|
||||
&ffmpeg,
|
||||
&["-f", "lavfi", "-i", &v, "-t", "4", "-c:v", "libx264", "-preset", "veryfast", "-g", "48", "-pix_fmt", "yuv420p", &tmp.to_string_lossy()],
|
||||
)
|
||||
})
|
||||
.expect("fixture")
|
||||
})
|
||||
.collect();
|
||||
// a fresh session (empty decoder caches) per measurement
|
||||
let setup = || {
|
||||
let mut s = Session::default();
|
||||
let refs: Vec<&Path> = files.iter().map(|p| p.as_path()).collect();
|
||||
let items = import(&mut s, &refs);
|
||||
let r = on_items(&mut s, "clip.createMulticam", &items, json!({"method": "in"})).unwrap();
|
||||
(s, ItemId(r["sequence"].as_u64().unwrap()))
|
||||
};
|
||||
let rate = FrameRate::FPS_24;
|
||||
let frames = 72;
|
||||
let (s, src) = setup();
|
||||
let provider = s.media.provider(s.project.clone(), s.services.clone());
|
||||
let q = s.project.sequence(src).unwrap().clone();
|
||||
let t0 = std::time::Instant::now();
|
||||
for f in 0..frames {
|
||||
let opts = filmcraft_render::RenderOptions { scale: 0.5, ..Default::default() };
|
||||
let _ = filmcraft_render::multicam::render_angle(&s.project, &q, 0, rate.tick_of(f), opts, &provider);
|
||||
}
|
||||
let one_ms = t0.elapsed().as_secs_f64() * 1000.0 / frames as f64;
|
||||
let (s, src) = setup();
|
||||
let provider = s.media.provider(s.project.clone(), s.services.clone());
|
||||
let cpu0 = crate::multicam_tests::process_cpu();
|
||||
let t0 = std::time::Instant::now();
|
||||
for f in 0..frames {
|
||||
let (g, _) = filmcraft_render::multicam::render_grid(&s.project, src, rate.tick_of(f), 0.25, &provider).unwrap();
|
||||
assert_eq!((g.w, g.h), (960, 540));
|
||||
}
|
||||
let grid_ms = t0.elapsed().as_secs_f64() * 1000.0 / frames as f64;
|
||||
let grid_cpu = (crate::multicam_tests::process_cpu() - cpu0) * 1000.0 / frames as f64;
|
||||
eprintln!(
|
||||
"Multi-Camera grid (4 × 1080p H.264, ¼-scale cells): {grid_ms:.1} ms/frame wall, {grid_cpu:.1} ms/frame CPU; one angle at ½ scale: {one_ms:.1} ms/frame wall (24 fps budget 41.7 ms)"
|
||||
);
|
||||
}
|
||||
|
|
@ -35,6 +35,27 @@ pub const PREMIERE: &[Entry] = &[
|
|||
("tool.hand", "H", ""),
|
||||
("tool.zoom", "Z", ""),
|
||||
("tool.type", "T", ""),
|
||||
// Multi-camera
|
||||
("multicam.selectCamera1", "1", ""),
|
||||
("multicam.selectCamera2", "2", ""),
|
||||
("multicam.selectCamera3", "3", ""),
|
||||
("multicam.selectCamera4", "4", ""),
|
||||
("multicam.selectCamera5", "5", ""),
|
||||
("multicam.selectCamera6", "6", ""),
|
||||
("multicam.selectCamera7", "7", ""),
|
||||
("multicam.selectCamera8", "8", ""),
|
||||
("multicam.selectCamera9", "9", ""),
|
||||
("multicam.cutToCamera1", "Ctrl+1", ""),
|
||||
("multicam.cutToCamera2", "Ctrl+2", ""),
|
||||
("multicam.cutToCamera3", "Ctrl+3", ""),
|
||||
("multicam.cutToCamera4", "Ctrl+4", ""),
|
||||
("multicam.cutToCamera5", "Ctrl+5", ""),
|
||||
("multicam.cutToCamera6", "Ctrl+6", ""),
|
||||
("multicam.cutToCamera7", "Ctrl+7", ""),
|
||||
("multicam.cutToCamera8", "Ctrl+8", ""),
|
||||
("multicam.cutToCamera9", "Ctrl+9", ""),
|
||||
("multicam.recordToggle", "0", ""),
|
||||
("multicam.toggleView", "Shift+0", ""),
|
||||
// Application
|
||||
("app.preferences.autoSave", "Cmd+,", ""),
|
||||
("app.keyboardShortcuts", "Cmd+Alt+K", ""),
|
||||
|
|
|
|||
|
|
@ -0,0 +1,234 @@
|
|||
//! Synchronising clips: Clip ▸ Synchronize (timeline clips), Merge Clips and Create
|
||||
//! Multi-Camera Source Sequence (project items).
|
||||
//!
|
||||
//! Every method reduces a clip to an **anchor**: the media time of the clip that lines up with
|
||||
//! one common instant (common time = media time − anchor).
|
||||
//!
|
||||
//! | method | anchor |
|
||||
//! |---|---|
|
||||
//! | `in` | the In point (project items: the source In mark, else the start; timeline clips: their source In) |
|
||||
//! | `out` | the Out point (exclusive end) |
|
||||
//! | `timecode` | −(start timecode), optionally ignoring hours |
|
||||
//! | `marker` | the first clip marker (or the first one named `marker`) |
|
||||
//! | `audio` | from the waveform cross-correlation against the reference clip ([`filmcraft_audio_dsp::sync`]) |
|
||||
//!
|
||||
//! `offset` (frames) shifts every clip but the reference by that much (a known delay to correct).
|
||||
|
||||
use filmcraft_audio_dsp::sync::{SyncOptions, SyncResult, find_offset, mixdown};
|
||||
use filmcraft_project::{ItemId, ItemKind, Project, TrackItem};
|
||||
use filmcraft_time::{FrameRate, TICKS_PER_SECOND, Tick, TimeRange};
|
||||
use serde_json::{Value, json};
|
||||
|
||||
use crate::{EngineError, Result, Session};
|
||||
|
||||
/// Sample rate audio is analysed at.
|
||||
pub const ANALYSIS_RATE: u32 = 48_000;
|
||||
/// Longest stretch of a clip analysed for audio sync (from its start).
|
||||
pub const MAX_ANALYSIS: Tick = Tick(30 * 60 * TICKS_PER_SECOND);
|
||||
|
||||
#[derive(Clone, Debug, PartialEq)]
|
||||
pub enum Method {
|
||||
In,
|
||||
Out,
|
||||
Timecode { ignore_hours: bool },
|
||||
Marker { name: Option<String> },
|
||||
Audio,
|
||||
}
|
||||
|
||||
impl Method {
|
||||
pub fn from_params(p: &Value) -> std::result::Result<Method, String> {
|
||||
let m = p.get("method").and_then(Value::as_str).unwrap_or("in").to_ascii_lowercase();
|
||||
Ok(match m.as_str() {
|
||||
"in" | "inpoint" | "inpoints" | "start" | "clipstart" => Method::In,
|
||||
"out" | "outpoint" | "outpoints" | "end" | "clipend" => Method::Out,
|
||||
"timecode" | "tc" => Method::Timecode { ignore_hours: p.get("ignoreHours").and_then(Value::as_bool).unwrap_or(false) },
|
||||
"marker" | "clipmarker" | "markers" => Method::Marker { name: p.get("marker").and_then(Value::as_str).map(str::to_string) },
|
||||
"audio" | "sound" | "waveform" => Method::Audio,
|
||||
o => return Err(format!("unknown sync method `{o}` (in, out, timecode, marker, audio)")),
|
||||
})
|
||||
}
|
||||
pub fn name(&self) -> &'static str {
|
||||
match self {
|
||||
Method::In => "in",
|
||||
Method::Out => "out",
|
||||
Method::Timecode { .. } => "timecode",
|
||||
Method::Marker { .. } => "marker",
|
||||
Method::Audio => "audio",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// What syncing needs to know about one clip.
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct SyncClip {
|
||||
pub item: ItemId,
|
||||
pub name: String,
|
||||
/// Media range the clip uses.
|
||||
pub range: TimeRange,
|
||||
/// In / Out points (media time, exclusive Out).
|
||||
pub in_point: Tick,
|
||||
pub out_point: Tick,
|
||||
/// Start timecode in ticks (0 when the media has none).
|
||||
pub timecode: Tick,
|
||||
/// Clip markers: (media time, name).
|
||||
pub markers: Vec<(Tick, String)>,
|
||||
}
|
||||
|
||||
/// The media item behind an item (subclips resolve to their parent) and its subclip range.
|
||||
fn media_of(p: &Project, item: ItemId) -> Option<(ItemId, &filmcraft_project::MediaClip, Option<TimeRange>)> {
|
||||
let it = p.item(item)?;
|
||||
match &it.kind {
|
||||
ItemKind::Media(m) => Some((item, m, None)),
|
||||
ItemKind::Subclip { parent, range } => media_of(p, *parent).map(|(i, m, _)| (i, m, Some(*range))),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
fn start_timecode(m: &filmcraft_project::MediaClip) -> Tick {
|
||||
let rate = m.frame_rate();
|
||||
m.info.start_timecode.map(|f| rate.tick_of(f)).unwrap_or(Tick::ZERO)
|
||||
}
|
||||
|
||||
/// A project item as a sync clip (the whole media, or a subclip's range; In/Out marks).
|
||||
pub fn project_clip(p: &Project, item: ItemId) -> Option<SyncClip> {
|
||||
let it = p.item(item)?;
|
||||
let (_, m, sub) = media_of(p, item)?;
|
||||
let range = sub.unwrap_or(TimeRange::new(Tick::ZERO, m.duration()));
|
||||
let rate = m.frame_rate();
|
||||
let in_point = m.mark_in.filter(|_| sub.is_none()).unwrap_or(range.start);
|
||||
let out_point = m.mark_out.filter(|_| sub.is_none()).map(|o| o + rate.frame_duration()).unwrap_or(range.end());
|
||||
Some(SyncClip {
|
||||
item,
|
||||
name: it.name.clone(),
|
||||
range,
|
||||
in_point,
|
||||
out_point,
|
||||
timecode: start_timecode(m),
|
||||
markers: m.markers.iter().map(|k| (k.start, k.name.clone())).collect(),
|
||||
})
|
||||
}
|
||||
|
||||
/// A timeline clip as a sync clip (its source range; markers of the clip and of its media).
|
||||
pub fn timeline_clip(p: &Project, ti: &TrackItem) -> SyncClip {
|
||||
let media = media_of(p, ti.item);
|
||||
let range = TimeRange::from_bounds(ti.source_in, ti.source_out());
|
||||
let mut markers: Vec<(Tick, String)> = ti.markers.iter().map(|k| (k.start, k.name.clone())).collect();
|
||||
if let Some((_, m, _)) = media {
|
||||
markers.extend(m.markers.iter().map(|k| (k.start, k.name.clone())));
|
||||
}
|
||||
SyncClip {
|
||||
item: ti.item,
|
||||
name: ti.name.clone(),
|
||||
range,
|
||||
in_point: range.start,
|
||||
out_point: range.end(),
|
||||
timecode: media.map(|(_, m, _)| start_timecode(m)).unwrap_or(Tick::ZERO),
|
||||
markers,
|
||||
}
|
||||
}
|
||||
|
||||
/// Mono audio of a clip's media range at [`ANALYSIS_RATE`] (None: no audio).
|
||||
pub fn clip_audio(s: &Session, c: &SyncClip) -> Option<Vec<f32>> {
|
||||
let src = s.source(c.item)?;
|
||||
if !src.info().has_audio() {
|
||||
return None;
|
||||
}
|
||||
let sr = ANALYSIS_RATE as i64;
|
||||
let start = c.range.start.to_units_floor(sr);
|
||||
let len = c.range.duration.min(MAX_ANALYSIS).to_units_floor(sr).max(0) as usize;
|
||||
let buf = src.audio(start, len, ANALYSIS_RATE).ok()?;
|
||||
let chans: Vec<&[f32]> = buf.channels.iter().map(|c| c.as_slice()).collect();
|
||||
Some(mixdown(&chans))
|
||||
}
|
||||
|
||||
/// The outcome for one clip: its anchor, and for audio the correlation figures.
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct Anchor {
|
||||
pub anchor: Tick,
|
||||
pub audio: Option<SyncResult>,
|
||||
}
|
||||
|
||||
/// Compute the anchors of `clips` (index `reference` is the reference for audio and `offset`).
|
||||
/// `offset` (ticks) moves every other clip later by that much.
|
||||
pub fn anchors(s: &Session, clips: &[SyncClip], reference: usize, method: &Method, offset: Tick) -> Result<Vec<Anchor>> {
|
||||
if clips.is_empty() {
|
||||
return Err(EngineError::Other("nothing to synchronize".into()));
|
||||
}
|
||||
let hour = Tick(3600 * TICKS_PER_SECOND);
|
||||
let mut out: Vec<Anchor> = Vec::with_capacity(clips.len());
|
||||
let ref_audio = if *method == Method::Audio {
|
||||
Some(clip_audio(s, &clips[reference]).ok_or_else(|| EngineError::Other(format!("{} has no audio to synchronize by", clips[reference].name)))?)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
for (i, c) in clips.iter().enumerate() {
|
||||
let a = match method {
|
||||
Method::In => Anchor { anchor: c.in_point, audio: None },
|
||||
Method::Out => Anchor { anchor: c.out_point, audio: None },
|
||||
Method::Timecode { ignore_hours } => {
|
||||
let tc = if *ignore_hours { Tick(c.timecode.0.rem_euclid(hour.0)) } else { c.timecode };
|
||||
Anchor { anchor: Tick::ZERO - tc, audio: None }
|
||||
}
|
||||
Method::Marker { name } => {
|
||||
let m =
|
||||
c.markers.iter().filter(|(_, n)| name.as_deref().is_none_or(|w| n.eq_ignore_ascii_case(w))).map(|(t, _)| *t).min().ok_or_else(|| {
|
||||
EngineError::Other(format!("{} has no clip marker{}", c.name, name.as_ref().map(|n| format!(" named “{n}”")).unwrap_or_default()))
|
||||
})?;
|
||||
Anchor { anchor: m, audio: None }
|
||||
}
|
||||
Method::Audio => {
|
||||
if i == reference {
|
||||
Anchor { anchor: c.range.start, audio: None }
|
||||
} else {
|
||||
let a = ref_audio.as_deref().unwrap_or_default();
|
||||
let b = clip_audio(s, c).ok_or_else(|| EngineError::Other(format!("{} has no audio to synchronize by", c.name)))?;
|
||||
let r = find_offset(a, &b, ANALYSIS_RATE, &SyncOptions::default())
|
||||
.ok_or_else(|| EngineError::Other(format!("{} and {} have no common audio", clips[reference].name, c.name)))?;
|
||||
// b[n] (media r0_i + n) lines up with a[n + L] (media r0_ref + n + L); with the
|
||||
// reference anchored at r0_ref, equal common times give anchor_i = r0_i − L
|
||||
let lag = Tick::from_units(r.lag, ANALYSIS_RATE as i64);
|
||||
Anchor { anchor: c.range.start - lag, audio: Some(r) }
|
||||
}
|
||||
}
|
||||
};
|
||||
out.push(a);
|
||||
}
|
||||
if offset != Tick::ZERO {
|
||||
for (i, a) in out.iter_mut().enumerate() {
|
||||
if i != reference {
|
||||
// later on the common timeline = smaller anchor
|
||||
a.anchor -= offset;
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
/// Where each clip's range starts on a common timeline that begins at 0 (the earliest clip), at
|
||||
/// the given frame rate (snapped to frames when `snap`).
|
||||
pub fn placements(clips: &[SyncClip], anchors: &[Anchor], rate: FrameRate, snap: bool) -> Vec<Tick> {
|
||||
let raw: Vec<Tick> = clips.iter().zip(anchors).map(|(c, a)| c.range.start - a.anchor).collect();
|
||||
let min = raw.iter().copied().min().unwrap_or(Tick::ZERO);
|
||||
raw.into_iter().map(|t| if snap { rate.snap_nearest(t - min) } else { t - min }).collect()
|
||||
}
|
||||
|
||||
/// JSON report of one sync for agents.
|
||||
pub fn report(clips: &[SyncClip], anchors: &[Anchor], starts: &[Tick], rate: FrameRate) -> Value {
|
||||
Value::Array(
|
||||
clips
|
||||
.iter()
|
||||
.zip(anchors)
|
||||
.zip(starts)
|
||||
.map(|((c, a), st)| {
|
||||
json!({
|
||||
"item": c.item.0,
|
||||
"name": c.name,
|
||||
"start": st.0,
|
||||
"startFrame": rate.frame_at(*st),
|
||||
"anchor": a.anchor.0,
|
||||
"audio": a.audio.map(|r| json!({"lagSamples": r.lag, "lagFine": r.lag_fine, "confidence": r.confidence, "distinct": r.distinct, "reliable": r.reliable(), "inverted": r.inverted})),
|
||||
})
|
||||
})
|
||||
.collect(),
|
||||
)
|
||||
}
|
||||
|
|
@ -27,6 +27,9 @@
|
|||
//! - **v5** (M8.8): colour management (sequence working space, Interpret Footage colour space,
|
||||
//! project LUT library). No-op step, for the same reason.
|
||||
//! - **v6** (M11.7/M11.9): media identity fingerprints and ingest settings. No-op step.
|
||||
//! - **v7** (M5.5): effect masks (keyframable Bézier mask paths on effect instances). No-op step.
|
||||
//! - **v8** (M12): multi-camera source sequences, multi-camera clips and merged clips. No-op step;
|
||||
//! older builds would drop the camera data when saving.
|
||||
|
||||
pub mod atomic;
|
||||
pub mod autosave;
|
||||
|
|
@ -45,7 +48,7 @@ pub type Migration = fn(Value) -> Result<Value, String>;
|
|||
|
||||
/// `MIGRATIONS[i]` upgrades schema `i + 1` to `i + 2`. Append one function per schema bump; never
|
||||
/// edit a shipped one.
|
||||
pub const MIGRATIONS: &[Migration] = &[v1_to_v2, v2_to_v3, v3_to_v4, v4_to_v5, v5_to_v6, v6_to_v7];
|
||||
pub const MIGRATIONS: &[Migration] = &[v1_to_v2, v2_to_v3, v3_to_v4, v4_to_v5, v5_to_v6, v6_to_v7, v7_to_v8];
|
||||
|
||||
/// The schema version this build writes (and the newest it reads).
|
||||
pub const SCHEMA_VERSION: u32 = MIGRATIONS.len() as u32 + 1;
|
||||
|
|
@ -244,13 +247,19 @@ fn v6_to_v7(doc: Value) -> Result<Value, String> {
|
|||
Ok(doc)
|
||||
}
|
||||
|
||||
/// v7 → v8: multi-camera source sequences / clips and merged clips added; existing data needs no
|
||||
/// change.
|
||||
fn v7_to_v8(doc: Value) -> Result<Value, String> {
|
||||
Ok(doc)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn schema_version_matches_table() {
|
||||
assert_eq!(SCHEMA_VERSION, 7);
|
||||
assert_eq!(SCHEMA_VERSION, 8);
|
||||
}
|
||||
|
||||
#[test]
|
||||
|
|
|
|||
|
|
@ -425,6 +425,7 @@ impl Builder {
|
|||
frame_hold: None,
|
||||
scale_to_frame: false,
|
||||
essential: None,
|
||||
multicam: None,
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -483,6 +484,8 @@ pub(crate) fn empty_sequence(settings: SequenceSettings) -> Sequence {
|
|||
master_mixer: Default::default(),
|
||||
submix_tracks: Vec::new(),
|
||||
caption_tracks: Vec::new(),
|
||||
multicam: None,
|
||||
merged: None,
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -778,6 +778,7 @@ mod tests {
|
|||
frame_hold: None,
|
||||
scale_to_frame: false,
|
||||
essential: None,
|
||||
multicam: None,
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -14,6 +14,7 @@ pub mod graphic;
|
|||
pub mod keyframe;
|
||||
pub mod mask;
|
||||
pub mod mixer;
|
||||
pub mod multicam;
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
|
||||
|
|
@ -28,6 +29,7 @@ pub use essential::{AudioType, EssentialSound};
|
|||
pub use keyframe::{Interpolation, Keyframe, Param, ParamValue};
|
||||
pub use mask::{Mask, MaskMode, MaskPath, MaskVertex, TrackMethod};
|
||||
pub use mixer::{AutomationMode, InputMap, MixerStrip, TrackSend};
|
||||
pub use multicam::{Camera, MergedClip, MulticamAudio, MulticamSel, MulticamSource};
|
||||
|
||||
macro_rules! id_type {
|
||||
($name:ident) => {
|
||||
|
|
@ -320,6 +322,8 @@ impl ProjectItem {
|
|||
filmcraft_media::MediaKind::ImageSequence => "Image Sequence",
|
||||
filmcraft_media::MediaKind::Synthetic => "Synthetic",
|
||||
},
|
||||
ItemKind::Sequence(s) if s.multicam.is_some() => "Multi-Camera Source Sequence",
|
||||
ItemKind::Sequence(s) if s.merged.is_some() => "Merged Clip",
|
||||
ItemKind::Sequence(_) => "Sequence",
|
||||
ItemKind::Subclip { .. } => "Subclip",
|
||||
ItemKind::AdjustmentLayer { .. } => "Adjustment Layer",
|
||||
|
|
@ -470,6 +474,9 @@ pub struct TrackItem {
|
|||
/// Essential Sound audio type and settings (audio clips).
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub essential: Option<EssentialSound>,
|
||||
/// Multi-camera clip (a nested multi-camera source sequence): enabled flag and angle.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub multicam: Option<MulticamSel>,
|
||||
}
|
||||
|
||||
impl TrackItem {
|
||||
|
|
@ -727,6 +734,12 @@ pub struct Sequence {
|
|||
/// Caption tracks (drawn above the video tracks; first = top). Older files have none (serde default).
|
||||
#[serde(default)]
|
||||
pub caption_tracks: Vec<CaptionTrack>,
|
||||
/// Set on multi-camera source sequences (cameras, audio mode).
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub multicam: Option<MulticamSource>,
|
||||
/// Set on merged clips (the merged video and audio items).
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub merged: Option<MergedClip>,
|
||||
}
|
||||
|
||||
impl Sequence {
|
||||
|
|
@ -961,6 +974,8 @@ impl Project {
|
|||
master_mixer: MixerStrip::default(),
|
||||
submix_tracks: Vec::new(),
|
||||
caption_tracks: Vec::new(),
|
||||
multicam: None,
|
||||
merged: None,
|
||||
};
|
||||
for i in 0..v {
|
||||
let id = TrackId(self.alloc_id());
|
||||
|
|
@ -1025,6 +1040,9 @@ impl Project {
|
|||
TrackKind::Video => effect::intrinsic_video(),
|
||||
TrackKind::Audio => effect::intrinsic_audio(),
|
||||
};
|
||||
// a multi-camera source sequence edits in as a multi-camera clip showing its first angle
|
||||
let multicam =
|
||||
it.as_sequence().and_then(|q| q.multicam.as_ref()).map(|m| MulticamSel { enabled: true, angle: m.first_video_angle().unwrap_or(0) as u32 });
|
||||
let id = ClipId(self.alloc_id());
|
||||
let dur = seq_rate.snap_nearest(source.duration).max(seq_rate.frame_duration());
|
||||
Some(TrackItem {
|
||||
|
|
@ -1046,6 +1064,7 @@ impl Project {
|
|||
frame_hold: None,
|
||||
scale_to_frame: false,
|
||||
essential: None,
|
||||
multicam,
|
||||
})
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,201 @@
|
|||
//! Multi-camera source sequences and merged clips.
|
||||
//!
|
||||
//! A **multi-camera source sequence** is an ordinary [`Sequence`](crate::Sequence) whose
|
||||
//! [`Sequence::multicam`](crate::Sequence::multicam) describes its cameras: each camera (angle) is
|
||||
//! one video track plus its audio tracks, with the clips already synchronised. Edited into another
|
||||
//! sequence it is a nested clip; with [`TrackItem::multicam`](crate::TrackItem::multicam) enabled
|
||||
//! the clip shows only the selected angle's video track (video clips) or plays the selected
|
||||
//! angle's audio (audio clips, when the source switches audio), instead of the composite.
|
||||
//!
|
||||
//! A **merged clip** is a sequence too ([`Sequence::merged`](crate::Sequence::merged)): one video
|
||||
//! clip and up to 16 separately recorded audio clips, synchronised, used like a single clip.
|
||||
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
use crate::{ItemId, Sequence, TrackId};
|
||||
|
||||
/// How a multi-camera source sequence plays audio (Create Multi-Camera Source Sequence ▸ Audio).
|
||||
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub enum MulticamAudio {
|
||||
/// The audio of camera 1 (plus audio-only clips); the other cameras' audio tracks are muted.
|
||||
#[default]
|
||||
Camera1,
|
||||
/// Every camera's audio, mixed.
|
||||
AllCameras,
|
||||
/// The audio of the selected angle (plus audio-only clips): multi-camera audio clips switch
|
||||
/// with their angle.
|
||||
SwitchAudio,
|
||||
}
|
||||
|
||||
impl MulticamAudio {
|
||||
pub fn from_name(s: &str) -> Option<Self> {
|
||||
match s.to_ascii_lowercase().replace([' ', '_', '-'], "").as_str() {
|
||||
"camera1" | "cam1" => Some(Self::Camera1),
|
||||
"all" | "allcameras" => Some(Self::AllCameras),
|
||||
"switch" | "switchaudio" => Some(Self::SwitchAudio),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
pub fn name(self) -> &'static str {
|
||||
match self {
|
||||
Self::Camera1 => "camera1",
|
||||
Self::AllCameras => "all",
|
||||
Self::SwitchAudio => "switch",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// One camera angle of a multi-camera source sequence.
|
||||
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
|
||||
pub struct Camera {
|
||||
pub name: String,
|
||||
/// The angle's video track (None for an audio-only source, which is never an angle of the
|
||||
/// grid but is mixed in).
|
||||
pub video_track: Option<TrackId>,
|
||||
/// The angle's audio tracks.
|
||||
#[serde(default)]
|
||||
pub audio_tracks: Vec<TrackId>,
|
||||
/// Shown in the Multi-Camera view (Edit Cameras).
|
||||
#[serde(default = "yes")]
|
||||
pub enabled: bool,
|
||||
/// The project item the angle was made from.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub source: Option<ItemId>,
|
||||
}
|
||||
|
||||
fn yes() -> bool {
|
||||
true
|
||||
}
|
||||
|
||||
/// [`Sequence::multicam`](crate::Sequence::multicam): what makes a sequence a multi-camera source.
|
||||
#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)]
|
||||
pub struct MulticamSource {
|
||||
/// Angles in camera order (angle `n` = `cameras[n]`). Audio-only sources come last and have no
|
||||
/// video track.
|
||||
pub cameras: Vec<Camera>,
|
||||
pub audio: MulticamAudio,
|
||||
/// How the clips were synchronised (`in`, `out`, `timecode`, `marker`, `audio`), informational.
|
||||
#[serde(default)]
|
||||
pub sync: String,
|
||||
}
|
||||
|
||||
impl MulticamSource {
|
||||
/// Angles that have a video track (the grid), in camera order: `(angle, camera)`.
|
||||
pub fn video_angles(&self) -> impl Iterator<Item = (usize, &Camera)> {
|
||||
self.cameras.iter().enumerate().filter(|(_, c)| c.video_track.is_some())
|
||||
}
|
||||
/// Enabled video angles: the cameras shown in the Multi-Camera view (keys 1–9 pick from these).
|
||||
pub fn shown_angles(&self) -> Vec<usize> {
|
||||
self.video_angles().filter(|(_, c)| c.enabled).map(|(i, _)| i).collect()
|
||||
}
|
||||
/// Angle whose video is on `track`.
|
||||
pub fn angle_of_video_track(&self, track: TrackId) -> Option<usize> {
|
||||
self.cameras.iter().position(|c| c.video_track == Some(track))
|
||||
}
|
||||
/// Whether an audio track belongs to a camera with video (switchable) rather than to an
|
||||
/// audio-only source.
|
||||
pub fn camera_of_audio_track(&self, track: TrackId) -> Option<usize> {
|
||||
self.cameras.iter().position(|c| c.audio_tracks.contains(&track))
|
||||
}
|
||||
/// The audio tracks that play for `angle` (None = all of them, as in a plain nest).
|
||||
/// Audio-only sources always play; camera audio follows [`MulticamAudio`].
|
||||
pub fn audible_tracks(&self, angle: Option<usize>) -> Vec<TrackId> {
|
||||
let mut v = Vec::new();
|
||||
for (i, c) in self.cameras.iter().enumerate() {
|
||||
let audio_only = c.video_track.is_none();
|
||||
let on = audio_only
|
||||
|| match self.audio {
|
||||
MulticamAudio::Camera1 => i == self.first_video_angle().unwrap_or(0),
|
||||
MulticamAudio::AllCameras => true,
|
||||
MulticamAudio::SwitchAudio => Some(i) == angle.or(self.first_video_angle()),
|
||||
};
|
||||
if on {
|
||||
v.extend(c.audio_tracks.iter().copied());
|
||||
}
|
||||
}
|
||||
v
|
||||
}
|
||||
pub fn first_video_angle(&self) -> Option<usize> {
|
||||
self.video_angles().map(|(i, _)| i).next()
|
||||
}
|
||||
}
|
||||
|
||||
/// [`TrackItem::multicam`](crate::TrackItem::multicam): the angle a multi-camera clip shows.
|
||||
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct MulticamSel {
|
||||
/// Multi-Camera ▸ Enable. Off: the clip plays like any nested sequence.
|
||||
pub enabled: bool,
|
||||
/// Camera index into [`MulticamSource::cameras`] (video clips: the picture; audio clips: the
|
||||
/// sound when the source switches audio).
|
||||
pub angle: u32,
|
||||
}
|
||||
|
||||
/// [`Sequence::merged`](crate::Sequence::merged): what makes a sequence a merged clip.
|
||||
#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)]
|
||||
pub struct MergedClip {
|
||||
/// The video clip (None: audio-only merge).
|
||||
pub video: Option<ItemId>,
|
||||
pub audio: Vec<ItemId>,
|
||||
/// How the clips were synchronised.
|
||||
#[serde(default)]
|
||||
pub sync: String,
|
||||
}
|
||||
|
||||
impl crate::TrackItem {
|
||||
/// The angle this clip shows when it is an enabled multi-camera clip, else None (it plays as an
|
||||
/// ordinary nest). Any nested sequence can be a multi-camera clip: without camera data its
|
||||
/// video tracks are the angles.
|
||||
pub fn multicam_angle(&self, _nested: &Sequence) -> Option<usize> {
|
||||
self.multicam.filter(|m| m.enabled).map(|m| m.angle as usize)
|
||||
}
|
||||
}
|
||||
|
||||
impl Sequence {
|
||||
/// The cameras of this sequence used as a multi-camera source: its camera data, or for a plain
|
||||
/// sequence one camera per video track (named after the track) with all audio mixed.
|
||||
pub fn cameras(&self) -> MulticamSource {
|
||||
if let Some(m) = &self.multicam {
|
||||
return m.clone();
|
||||
}
|
||||
MulticamSource {
|
||||
cameras: self
|
||||
.video_tracks
|
||||
.iter()
|
||||
.enumerate()
|
||||
.map(|(i, t)| Camera {
|
||||
name: t.name.clone(),
|
||||
video_track: Some(t.id),
|
||||
audio_tracks: self.audio_tracks.get(i).map(|a| vec![a.id]).unwrap_or_default(),
|
||||
enabled: true,
|
||||
source: None,
|
||||
})
|
||||
.collect(),
|
||||
audio: MulticamAudio::AllCameras,
|
||||
sync: String::new(),
|
||||
}
|
||||
}
|
||||
/// Video track index of `angle` when this sequence is used as a multi-camera source.
|
||||
pub fn angle_video_track_index(&self, angle: usize) -> Option<usize> {
|
||||
match &self.multicam {
|
||||
Some(mc) => {
|
||||
let id = mc.cameras.get(angle)?.video_track?;
|
||||
self.video_tracks.iter().position(|t| t.id == id)
|
||||
}
|
||||
None => (angle < self.video_tracks.len()).then_some(angle),
|
||||
}
|
||||
}
|
||||
/// A copy whose audio tracks are muted except those audible for `angle` (None = the source's
|
||||
/// default: camera 1, all, or the first angle when switching). Plain sequences are unchanged.
|
||||
pub fn with_angle_audio(&self, angle: Option<usize>) -> Sequence {
|
||||
let mut q = self.clone();
|
||||
if let Some(mc) = &self.multicam {
|
||||
let on = mc.audible_tracks(angle);
|
||||
for t in &mut q.audio_tracks {
|
||||
if !on.contains(&t.id) {
|
||||
t.muted = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
q
|
||||
}
|
||||
}
|
||||
|
|
@ -76,16 +76,6 @@ pub fn pan_gains(pan: f32) -> (f32, f32) {
|
|||
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
fn mix_item(project: &Project, item: &TrackItem, start: i64, frames: usize, sr: u32, sources: &dyn SourceProvider, out: &mut AudioBuffer, extra: f32) {
|
||||
let Some(src) = sources.source(item.item) else { return };
|
||||
if !src.info().has_audio() {
|
||||
// nested sequences
|
||||
if let Some(nested) = project.sequence(item.item) {
|
||||
let rel0 = start - item.start.to_units_floor(sr as i64) + item.source_in.to_units_floor(sr as i64);
|
||||
let b = mix_sequence(project, nested, rel0, frames, sources);
|
||||
out.mix_from(&b, &[extra, extra]);
|
||||
}
|
||||
return;
|
||||
}
|
||||
let item_s0 = item.start.to_units_floor(sr as i64);
|
||||
let item_s1 = item.end().to_units_floor(sr as i64);
|
||||
let a0 = start.max(item_s0);
|
||||
|
|
@ -97,7 +87,15 @@ fn mix_item(project: &Project, item: &TrackItem, start: i64, frames: usize, sr:
|
|||
return;
|
||||
}
|
||||
let n = (a1 - a0) as usize;
|
||||
let buf = effected(item, src.as_ref(), a0, n, sr);
|
||||
let buf = if let Some(nested) = project.sequence(item.item) {
|
||||
nested_audio(project, item, nested, a0, n, sr, sources)
|
||||
} else {
|
||||
let Some(src) = sources.source(item.item) else { return };
|
||||
if !src.info().has_audio() {
|
||||
return;
|
||||
}
|
||||
effected(item, src.as_ref(), a0, n, sr)
|
||||
};
|
||||
// gains: clip gain × Volume (keyframed, per 64-sample block) × channel volume × panner
|
||||
let clip_gain = db_to_gain(item.gain_db);
|
||||
let vol = item.effect("volume").filter(|e| e.enabled && !e.param("bypass").and_then(|p| p.value.as_bool()).unwrap_or(false));
|
||||
|
|
@ -124,6 +122,23 @@ fn mix_item(project: &Project, item: &TrackItem, start: i64, frames: usize, sr:
|
|||
}
|
||||
}
|
||||
|
||||
/// A nested sequence's mix for timeline samples `[a0, a0 + n)` of the clip that shows it. A
|
||||
/// multi-camera clip plays the audio of its source's audio setting: camera 1, all cameras, or (when
|
||||
/// switching audio) the angle the clip selects.
|
||||
fn nested_audio(project: &Project, item: &TrackItem, nested: &Sequence, a0: i64, n: usize, sr: u32, sources: &dyn SourceProvider) -> [Vec<f32>; 2] {
|
||||
let rel0 = a0 - item.start.to_units_floor(sr as i64) + item.source_in.to_units_floor(sr as i64);
|
||||
let b = if nested.multicam.is_some() {
|
||||
let q = nested.with_angle_audio(item.multicam_angle(nested));
|
||||
mix_sequence(project, &q, rel0, n, sources)
|
||||
} else {
|
||||
mix_sequence(project, nested, rel0, n, sources)
|
||||
};
|
||||
let mut it = b.channels.into_iter();
|
||||
let l = it.next().unwrap_or_else(|| vec![0.0; n]);
|
||||
let r = it.next().unwrap_or_else(|| l.clone());
|
||||
[l, r]
|
||||
}
|
||||
|
||||
/// The clip's audio after speed/reverse and clip effects (before clip gain, Volume and Panner) for
|
||||
/// timeline samples `[a0, a0 + n)`.
|
||||
fn effected(item: &TrackItem, src: &dyn filmcraft_media::MediaSource, a0: i64, n: usize, sr: u32) -> [Vec<f32>; 2] {
|
||||
|
|
|
|||
|
|
@ -19,6 +19,7 @@ pub mod image;
|
|||
pub mod luts;
|
||||
pub mod mask;
|
||||
pub mod mixer;
|
||||
pub mod multicam;
|
||||
pub mod offline;
|
||||
pub mod plan;
|
||||
pub mod preview;
|
||||
|
|
@ -80,18 +81,24 @@ pub fn render_sequence(project: &Project, seq_id: ItemId, t: Tick, opts: RenderO
|
|||
}
|
||||
|
||||
fn render_seq(project: &Project, seq: &Sequence, t: Tick, opts: RenderOptions, sources: &dyn SourceProvider) -> Image {
|
||||
render_seq_tracks(project, seq, t, opts, sources, None)
|
||||
}
|
||||
|
||||
/// Render a sequence, or only its video track `only` (a multi-camera angle; drawn even when the
|
||||
/// track's output is off).
|
||||
pub(crate) fn render_seq_tracks(project: &Project, seq: &Sequence, t: Tick, opts: RenderOptions, sources: &dyn SourceProvider, only: Option<usize>) -> Image {
|
||||
let (w, h) = output_size(seq, opts.scale);
|
||||
let mut canvas = Image::new(w, h);
|
||||
if opts.depth > 8 {
|
||||
return canvas;
|
||||
}
|
||||
let tc = format_time(t, seq.settings.frame_rate, seq.settings.drop_frame, TimeDisplay::Timecode, seq.settings.sample_rate as i64);
|
||||
for track in &seq.video_tracks {
|
||||
for (ti, track) in seq.video_tracks.iter().enumerate() {
|
||||
// A cancelled frame job (playback moved on) stops here; its result is discarded.
|
||||
if filmcraft_media::cancel::cancelled() {
|
||||
return canvas;
|
||||
}
|
||||
if !track.enabled {
|
||||
if only.is_some_and(|o| o != ti) || (!track.enabled && only.is_none()) {
|
||||
continue;
|
||||
}
|
||||
// transition covering t?
|
||||
|
|
@ -285,7 +292,17 @@ pub(crate) fn item_layer(
|
|||
}
|
||||
ItemKind::Sequence(nested) => {
|
||||
let sub = RenderOptions { scale: want, effects: opts.effects, depth: opts.depth + 1, captions: false, working_output: true };
|
||||
render_seq(project, nested, mt, sub, sources)
|
||||
match item.multicam_angle(nested) {
|
||||
// a multi-camera clip shows its angle's track only (nothing for an audio-only angle)
|
||||
Some(angle) => match nested.angle_video_track_index(angle) {
|
||||
Some(ti) => render_seq_tracks(project, nested, mt, sub, sources, Some(ti)),
|
||||
None => {
|
||||
let (nw, nh) = output_size(nested, want);
|
||||
Image::new(nw, nh)
|
||||
}
|
||||
},
|
||||
None => render_seq(project, nested, mt, sub, sources),
|
||||
}
|
||||
}
|
||||
ItemKind::AdjustmentLayer { .. } => return None,
|
||||
ItemKind::Graphic { .. } => {
|
||||
|
|
|
|||
|
|
@ -0,0 +1,61 @@
|
|||
//! Multi-camera rendering: one angle of a multi-camera source sequence, and the Multi-Camera
|
||||
//! view's grid of angles.
|
||||
//!
|
||||
//! The grid renders every shown angle at a reduced scale (sources are asked for small frames, so
|
||||
//! YUV→RGB conversion and compositing run at the cell size) in parallel, and tiles them into one
|
||||
//! image. A 2×2 grid of 1080p angles at ¼ scale costs about as much as one ½-resolution frame of
|
||||
//! compositing plus the four decodes.
|
||||
|
||||
use filmcraft_project::{ItemId, Project, Sequence};
|
||||
use filmcraft_time::Tick;
|
||||
use rayon::prelude::*;
|
||||
|
||||
use crate::{Image, RenderOptions, SourceProvider, output_size};
|
||||
|
||||
/// Grid layout for `n` angles: (columns, rows). 1 → 1×1, 2–4 → 2×2, 5–9 → 3×3, 10–16 → 4×4.
|
||||
pub fn grid_dims(n: usize) -> (usize, usize) {
|
||||
let c = (1..=4).find(|c| c * c >= n).unwrap_or(4);
|
||||
let r = n.div_ceil(c).max(1);
|
||||
(c, r.min(c))
|
||||
}
|
||||
|
||||
/// Render angle `angle` of the multi-camera source sequence `seq` at its time `t` (transparent
|
||||
/// for an audio-only angle). The image is in the sequence's working space unless `opts.depth == 0`
|
||||
/// and `!opts.working_output` (then display-ready like [`crate::render_sequence`]).
|
||||
pub fn render_angle(project: &Project, seq: &Sequence, angle: usize, t: Tick, opts: RenderOptions, sources: &dyn SourceProvider) -> Image {
|
||||
match seq.angle_video_track_index(angle) {
|
||||
Some(ti) => crate::render_seq_tracks(project, seq, t, opts, sources, Some(ti)),
|
||||
None => {
|
||||
let (w, h) = output_size(seq, opts.scale);
|
||||
Image::new(w, h)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The Multi-Camera view's angle grid of the sequence `item` used as a multi-camera source at its time `t`: the shown
|
||||
/// angles (Edit Cameras), each at `cell_scale` of the sequence frame size, tiled left to right,
|
||||
/// top to bottom over black. Returns the image and the angles in grid order.
|
||||
pub fn render_grid(project: &Project, item: ItemId, t: Tick, cell_scale: f32, sources: &dyn SourceProvider) -> Option<(Image, Vec<usize>)> {
|
||||
let seq = project.sequence(item)?;
|
||||
let angles = seq.cameras().shown_angles();
|
||||
let (cols, rows) = grid_dims(angles.len().max(1));
|
||||
let (cw, ch) = output_size(seq, cell_scale);
|
||||
let opts = RenderOptions { scale: cell_scale, ..Default::default() };
|
||||
let cells: Vec<Image> = angles.par_iter().map(|&a| render_angle(project, seq, a, t, opts, sources)).collect();
|
||||
let mut out = Image::filled(cw * cols, ch * rows, [0.0, 0.0, 0.0, 1.0]);
|
||||
for (k, cell) in cells.iter().enumerate().take(cols * rows) {
|
||||
let (x0, y0) = ((k % cols) * cw, (k / cols) * ch);
|
||||
for y in 0..ch.min(cell.h) {
|
||||
for x in 0..cw.min(cell.w) {
|
||||
let s = (y * cell.w + x) * 4;
|
||||
let d = ((y0 + y) * out.w + x0 + x) * 4;
|
||||
// cells over black: premultiplied over an opaque background
|
||||
let a = cell.px[s + 3];
|
||||
for c in 0..3 {
|
||||
out.px[d + c] = cell.px[s + c] + out.px[d + c] * (1.0 - a);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Some((out, angles))
|
||||
}
|
||||
|
|
@ -145,6 +145,22 @@ fn push_item(
|
|||
) {
|
||||
let mt = item.source_time_at(t);
|
||||
let (op, bl) = crate::opacity_blend(item, mt);
|
||||
// A multi-camera clip that only shows its angle (no effects, untransformed, same frame size)
|
||||
// draws the angle's clip directly: no CPU pass over the nested sequence.
|
||||
if let Some(ItemKind::Sequence(nested)) = project.item(item.item).map(|p| &p.kind)
|
||||
&& let Some(angle) = item.multicam_angle(nested)
|
||||
&& bl == Blend::Normal
|
||||
&& !(opts.effects && item.has_standard_effects())
|
||||
&& (nested.settings.width, nested.settings.height) == (seq.settings.width, seq.settings.height)
|
||||
&& near_identity(&motion_matrix(seq, item, (nested.settings.width, nested.settings.height), mt))
|
||||
&& let Some(tr) = nested.angle_video_track_index(angle).and_then(|i| nested.video_tracks.get(i))
|
||||
&& !tr.transitions.iter().any(|x| x.range().contains(mt))
|
||||
{
|
||||
if let Some(inner) = tr.item_at(mt).filter(|i| i.enabled) {
|
||||
push_item(project, nested, inner, mt, opts, sources, extra_opacity * op, out);
|
||||
}
|
||||
return;
|
||||
}
|
||||
// Graphic clips without standard effects: the layers are rasterised (cached) into one tight
|
||||
// image the GPU places as a layer.
|
||||
if bl == Blend::Normal
|
||||
|
|
@ -183,6 +199,10 @@ fn push_item(
|
|||
}
|
||||
}
|
||||
|
||||
fn near_identity(m: &Affine) -> bool {
|
||||
(m.a - 1.0).abs() < 1e-9 && (m.d - 1.0).abs() < 1e-9 && m.b.abs() < 1e-9 && m.c.abs() < 1e-9 && m.e.abs() < 1e-6 && m.f.abs() < 1e-6
|
||||
}
|
||||
|
||||
/// Execute a plan on the CPU (reference for the GPU compositor).
|
||||
pub fn execute_cpu(plan: &FramePlan) -> crate::Image {
|
||||
match plan {
|
||||
|
|
|
|||
|
|
@ -26,6 +26,9 @@ pub enum Target {
|
|||
Item(ItemId),
|
||||
/// A GPU frame plan of a sequence (decoded layers + transforms), composited on the GPU.
|
||||
SequencePlan(ItemId),
|
||||
/// The Multi-Camera view's angle grid of a multi-camera source at its time (all shown angles,
|
||||
/// each at the job's scale).
|
||||
MulticamGrid(ItemId),
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
|
||||
|
|
@ -707,6 +710,7 @@ fn render_job(job: &Job, pool: &Arc<MediaPool>, services: &Arc<dyn Services>, pr
|
|||
let img = match job.key.target {
|
||||
Target::Sequence(seq) | Target::SequencePlan(seq) => Some(filmcraft_render::render_sequence(&job.project, seq, job.time, opts, &provider)),
|
||||
Target::Item(item) => filmcraft_render::render_item(&job.project, item, job.time, job.scale, &provider),
|
||||
Target::MulticamGrid(item) => filmcraft_render::multicam::render_grid(&job.project, item, job.time, job.scale, &provider).map(|(img, _)| img),
|
||||
};
|
||||
let rgba = match img {
|
||||
Some(img) => Rgba { w: img.w, h: img.h, px: img.over_black_rgba8() },
|
||||
|
|
|
|||
|
|
@ -338,6 +338,8 @@ impl FilmcraftApp {
|
|||
let t = self.session.playhead();
|
||||
let _ = self.session.execute("mixer.recordStart", json!({"time": t.0}));
|
||||
}
|
||||
// Multi-Camera view: playing records live cuts (keys 1–9 / clicking angles)
|
||||
panels::multicam::on_play(self);
|
||||
}
|
||||
|
||||
pub fn stop(&mut self) {
|
||||
|
|
@ -355,6 +357,7 @@ impl FilmcraftApp {
|
|||
self.ui.status = e.to_string();
|
||||
}
|
||||
}
|
||||
panels::multicam::on_stop(self);
|
||||
}
|
||||
|
||||
fn start_audio(&mut self) {
|
||||
|
|
|
|||
|
|
@ -39,6 +39,8 @@ pub const UI_COMMANDS: &[UiCommand] = &[
|
|||
uic!("view.playbackRes.quarter", "1/4", ["View", "Playback Resolution"], None),
|
||||
uic!("view.playbackRes.eighth", "1/8", ["View", "Playback Resolution"], None),
|
||||
uic!("view.safeMargins", "Safe Margins", ["View"], None),
|
||||
uic!("multicam.toggleView", "Multi-Camera View", ["View"], Some("Shift+0")),
|
||||
uic!("multicam.recordToggle", "Multi-Camera Record On/Off Toggle", [], Some("0")),
|
||||
uic!("view.theme.dark", "Darkest", ["View", "Appearance"], None),
|
||||
uic!("view.theme.medium", "Medium", ["View", "Appearance"], None),
|
||||
uic!("view.theme.light", "Light", ["View", "Appearance"], None),
|
||||
|
|
@ -100,6 +102,12 @@ pub fn invoke(app: &mut FilmcraftApp, ctx: &egui::Context, id: &str, params: Val
|
|||
if let Some(r) = crate::panels::trim_monitor::route_transport(app, ctx, id) {
|
||||
return r;
|
||||
}
|
||||
if let Some(r) = crate::panels::multicam::route(app, id, ¶ms) {
|
||||
if let Err(e) = &r {
|
||||
app.ui.status = e.clone();
|
||||
}
|
||||
return r;
|
||||
}
|
||||
if let Some(r) = crate::panels::media_dialogs::route(app, id, ¶ms) {
|
||||
if let Err(e) = &r {
|
||||
app.ui.status = e.clone();
|
||||
|
|
|
|||
|
|
@ -7,6 +7,7 @@ pub fn show(app: &mut FilmcraftApp, ctx: &egui::Context) {
|
|||
crate::panels::media_dialogs::show(app, ctx);
|
||||
crate::panels::color_dialogs::show(app, ctx);
|
||||
crate::panels::presets::save_dialog(app, ctx);
|
||||
crate::panels::multicam::show_dialog(app, ctx);
|
||||
let Some(d) = app.dialog else { return };
|
||||
if let Some(still_open) = crate::panels::file_dialogs::show(app, ctx, d) {
|
||||
if !still_open && app.dialog == Some(d) {
|
||||
|
|
|
|||
|
|
@ -18,6 +18,7 @@ pub mod misc;
|
|||
pub mod mixer;
|
||||
pub mod monitor;
|
||||
pub mod presets;
|
||||
pub mod multicam;
|
||||
pub mod project;
|
||||
pub mod shortcuts_dialog;
|
||||
pub mod text;
|
||||
|
|
|
|||
|
|
@ -66,6 +66,17 @@ pub fn show(app: &mut FilmcraftApp, ui: &mut egui::Ui, rect: Rect, which: Which)
|
|||
}
|
||||
};
|
||||
let prefix = if which == Which::Program { "program" } else { "source" };
|
||||
// Multi-Camera view: the angle grid on the left, the program on the right
|
||||
let multicam = which == Which::Program && app.ui.program.multicam;
|
||||
let (grid_area, video_area) = if multicam {
|
||||
let mid = video_area.center().x;
|
||||
(Some(Rect::from_min_max(video_area.min, pos2(mid - 2.0, video_area.max.y))), Rect::from_min_max(pos2(mid + 2.0, video_area.min.y), video_area.max))
|
||||
} else {
|
||||
(None, video_area)
|
||||
};
|
||||
if let Some(g) = grid_area {
|
||||
crate::panels::multicam::grid(app, ui, g);
|
||||
}
|
||||
ui.painter().rect_filled(video_area, 0.0, t.panel_bg);
|
||||
// ---- picture
|
||||
let has_video = frame_size.0 > 0;
|
||||
|
|
@ -203,7 +214,14 @@ pub fn show(app: &mut FilmcraftApp, ui: &mut egui::Ui, rect: Rect, which: Which)
|
|||
if which == Which::Program {
|
||||
ui.separator();
|
||||
ui.checkbox(&mut app.ui.show_scopes, "Lumetri Scopes");
|
||||
ui.checkbox(&mut app.ui.program.multicam, "Multi-Camera");
|
||||
ui.checkbox(&mut app.playback.looping, "Loop");
|
||||
ui.separator();
|
||||
let mut follows = app.session.state.multicam_audio_follows_video;
|
||||
if ui.checkbox(&mut follows, "Multi-Camera Audio Follows Video").changed() {
|
||||
let _ = app.session.execute("multicam.audioFollowsVideo", json!({"enabled": follows}));
|
||||
}
|
||||
ui.checkbox(&mut app.ui.multicam_record, "Multi-Camera Record");
|
||||
}
|
||||
});
|
||||
|
||||
|
|
@ -216,7 +234,7 @@ pub fn show(app: &mut FilmcraftApp, ui: &mut egui::Ui, rect: Rect, which: Which)
|
|||
transport(app, ui, row3, which);
|
||||
}
|
||||
|
||||
fn quantize_scale(s: f32) -> f32 {
|
||||
pub fn quantize_scale(s: f32) -> f32 {
|
||||
// Buckets keep the cache effective while resizing the panel.
|
||||
let buckets = [1.0 / 32.0, 1.0 / 16.0, 1.0 / 8.0, 0.1875, 0.25, 0.375, 0.5, 0.75, 1.0];
|
||||
*buckets.iter().find(|b| **b >= s - 1e-4).unwrap_or(&1.0)
|
||||
|
|
|
|||
|
|
@ -0,0 +1,381 @@
|
|||
//! Multi-camera UI: the Program monitor's Multi-Camera view (angle grid next to the program) and
|
||||
//! the Synchronize, Merge Clips and Create Multi-Camera Source Sequence dialogs.
|
||||
//!
|
||||
//! The grid is one frame job ([`Target::MulticamGrid`]): every shown angle rendered at the grid
|
||||
//! cell's size (reduced resolution) and tiled, prefetched like the program while playing.
|
||||
//! Clicking an angle (or keys 1–9) cuts to it while recording (playing with Multi-Camera Record
|
||||
//! on) and switches the clip at the playhead when stopped; Ctrl/⌘-click switches the video only.
|
||||
//!
|
||||
//! Automation ids:
|
||||
//! - Multi-Camera view: `program.multicam.grid`, `program.multicam.angle.<n>` (n = 1-based camera
|
||||
//! in grid order), `program.multicam.record`.
|
||||
//! - Synchronize: `sync.method.<in|out|timecode|marker|audio>`, `sync.ignoreHours`, `sync.marker`,
|
||||
//! `sync.offset`, `sync.track`, `sync.ok`, `sync.cancel`.
|
||||
//! - Merge Clips: `merge.name`, `merge.method.<…>`, `merge.ignoreHours`, `merge.marker`,
|
||||
//! `merge.offset`, `merge.removeVideoAudio`, `merge.ok`, `merge.cancel`.
|
||||
//! - Create Multi-Camera Source Sequence: `mcam.name`, `mcam.method.<…>`, `mcam.ignoreHours`,
|
||||
//! `mcam.marker`, `mcam.offset`, `mcam.audio.<camera1|all|switch>`,
|
||||
//! `mcam.cameraNames.<clip|track|metadata>`, `mcam.processedBin`, `mcam.ok`, `mcam.cancel`.
|
||||
|
||||
use egui::{Align2, Color32, Rect, RichText, Sense, Stroke, StrokeKind, pos2, vec2};
|
||||
use filmcraft_project::ItemId;
|
||||
use filmcraft_time::Tick;
|
||||
use serde_json::{Value, json};
|
||||
|
||||
use crate::FilmcraftApp;
|
||||
use crate::frames::{FrameKey, Target};
|
||||
use crate::state::SyncDraft;
|
||||
use crate::theme::Tokens;
|
||||
|
||||
type Elems = Vec<(String, Rect, String)>;
|
||||
|
||||
/// Menu / shortcut entry points: the three dialogs (without params) and the view toggles.
|
||||
pub fn route(app: &mut FilmcraftApp, id: &str, params: &Value) -> Option<Result<Value, String>> {
|
||||
match id {
|
||||
"multicam.toggleView" => {
|
||||
app.ui.program.multicam = params.get("enabled").and_then(Value::as_bool).unwrap_or(!app.ui.program.multicam);
|
||||
return Some(Ok(json!({"multicam": app.ui.program.multicam})));
|
||||
}
|
||||
"multicam.recordToggle" => {
|
||||
app.ui.multicam_record = params.get("enabled").and_then(Value::as_bool).unwrap_or(!app.ui.multicam_record);
|
||||
app.ui.status = format!("Multi-Camera Record {}", if app.ui.multicam_record { "on" } else { "off" });
|
||||
return Some(Ok(json!({"record": app.ui.multicam_record})));
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
if !params.as_object().is_none_or(|m| m.is_empty()) {
|
||||
return None;
|
||||
}
|
||||
let kind = match id {
|
||||
"clip.synchronize" => "synchronize",
|
||||
"clip.mergeClips" => "merge",
|
||||
"clip.createMulticam" => "multicam",
|
||||
_ => return None,
|
||||
};
|
||||
if let Err(e) = filmcraft_engine::find_command(id).map_or(Ok(()), |c| (c.enabled)(&app.session)) {
|
||||
return Some(Err(e));
|
||||
}
|
||||
let items: Vec<u64> = app.session.state.project_selection.iter().map(|i| i.0).collect();
|
||||
let first = app.session.state.project_selection.first().and_then(|i| app.session.project.item(*i)).map(|i| i.name.clone()).unwrap_or_default();
|
||||
let name = match kind {
|
||||
"merge" => format!("{first} - Merged"),
|
||||
"multicam" => format!("{first} Multicam"),
|
||||
_ => String::new(),
|
||||
};
|
||||
app.ui.sync_dialog = Some(SyncDraft { kind: kind.into(), items, name, ..Default::default() });
|
||||
Some(Ok(json!({"dialog": kind})))
|
||||
}
|
||||
|
||||
fn push(elems: &mut Elems, id: impl Into<String>, r: &egui::Response, label: impl Into<String>) {
|
||||
elems.push((id.into(), r.rect, label.into()));
|
||||
}
|
||||
|
||||
const METHODS: [(&str, &str); 5] = [("in", "In Points"), ("out", "Out Points"), ("timecode", "Timecode"), ("marker", "Clip Marker"), ("audio", "Audio")];
|
||||
|
||||
/// Draw the open Synchronize / Merge Clips / Create Multi-Camera dialog.
|
||||
pub fn show_dialog(app: &mut FilmcraftApp, ctx: &egui::Context) {
|
||||
let Some(mut d) = app.ui.sync_dialog.clone() else { return };
|
||||
let (title, pre) = match d.kind.as_str() {
|
||||
"merge" => ("Merge Clips", "merge"),
|
||||
"multicam" => ("Create Multi-Camera Source Sequence", "mcam"),
|
||||
_ => ("Synchronize Clips", "sync"),
|
||||
};
|
||||
let mut elems: Elems = Vec::new();
|
||||
let mut action: Option<&str> = None;
|
||||
let accent = app.tokens.accent;
|
||||
let names: Vec<String> = d.items.iter().filter_map(|i| app.session.project.item(ItemId(*i)).map(|it| it.name.clone())).collect();
|
||||
egui::Window::new(title).collapsible(false).resizable(false).default_width(440.0).anchor(Align2::CENTER_CENTER, [0.0, 0.0]).show(ctx, |ui| {
|
||||
if pre != "sync" {
|
||||
ui.horizontal(|ui| {
|
||||
ui.label(if pre == "merge" { "Clip Name:" } else { "Sequence Name:" });
|
||||
let r = ui.add(egui::TextEdit::singleline(&mut d.name).desired_width(300.0));
|
||||
push(&mut elems, format!("{pre}.name"), &r, "name");
|
||||
});
|
||||
ui.label(RichText::new(format!("{} clip(s): {}", names.len(), names.join(", "))).weak().small());
|
||||
ui.add_space(4.0);
|
||||
} else {
|
||||
let n = app.session.state.selection.len();
|
||||
ui.label(RichText::new(format!("{n} clip(s) selected; the clip on the reference track stays put.")).weak().small());
|
||||
}
|
||||
ui.group(|ui| {
|
||||
ui.set_min_width(400.0);
|
||||
ui.label(RichText::new("Synchronize Point").strong());
|
||||
for (m, label) in METHODS {
|
||||
ui.horizontal(|ui| {
|
||||
let r = ui.radio(d.method == m, label);
|
||||
push(&mut elems, format!("{pre}.method.{m}"), &r, label);
|
||||
if r.clicked() {
|
||||
d.method = m.into();
|
||||
}
|
||||
match m {
|
||||
"timecode" => {
|
||||
let r = ui.add_enabled(d.method == "timecode", egui::Checkbox::new(&mut d.ignore_hours, "Ignore Hours"));
|
||||
push(&mut elems, format!("{pre}.ignoreHours"), &r, "Ignore Hours");
|
||||
}
|
||||
"marker" => {
|
||||
let r =
|
||||
ui.add_enabled(d.method == "marker", egui::TextEdit::singleline(&mut d.marker).hint_text("first marker").desired_width(140.0));
|
||||
push(&mut elems, format!("{pre}.marker"), &r, "marker name");
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
});
|
||||
}
|
||||
ui.horizontal(|ui| {
|
||||
ui.label(if pre == "merge" { "Offset Audio by:" } else { "Offset:" });
|
||||
let r = ui.add(egui::DragValue::new(&mut d.offset).range(-100_000..=100_000).suffix(" frames"));
|
||||
push(&mut elems, format!("{pre}.offset"), &r, "offset");
|
||||
});
|
||||
});
|
||||
match pre {
|
||||
"sync" => {
|
||||
ui.horizontal(|ui| {
|
||||
ui.label("Reference track:");
|
||||
let r = ui.add(egui::TextEdit::singleline(&mut d.track).hint_text("lowest (V1…)").desired_width(90.0));
|
||||
push(&mut elems, "sync.track", &r, "reference track");
|
||||
});
|
||||
}
|
||||
"merge" => {
|
||||
let r = ui.checkbox(&mut d.remove_video_audio, "Remove Audio from AV Clip");
|
||||
push(&mut elems, "merge.removeVideoAudio", &r, "Remove Audio from AV Clip");
|
||||
}
|
||||
_ => {
|
||||
ui.group(|ui| {
|
||||
ui.set_min_width(400.0);
|
||||
ui.label(RichText::new("Audio").strong());
|
||||
ui.horizontal(|ui| {
|
||||
for (a, label) in [("camera1", "Camera 1"), ("all", "All Cameras"), ("switch", "Switch Audio")] {
|
||||
let r = ui.radio(d.audio == a, label);
|
||||
push(&mut elems, format!("mcam.audio.{a}"), &r, label);
|
||||
if r.clicked() {
|
||||
d.audio = a.into();
|
||||
}
|
||||
}
|
||||
});
|
||||
ui.label(RichText::new("Camera Names").strong());
|
||||
ui.horizontal(|ui| {
|
||||
for (a, label) in [("clip", "Clip Names"), ("track", "Enumerate Cameras"), ("metadata", "Camera Angle Metadata")] {
|
||||
let r = ui.radio(d.camera_names == a, label);
|
||||
push(&mut elems, format!("mcam.cameraNames.{a}"), &r, label);
|
||||
if r.clicked() {
|
||||
d.camera_names = a.into();
|
||||
}
|
||||
}
|
||||
});
|
||||
});
|
||||
let r = ui.checkbox(&mut d.processed_bin, "Move source clips to Processed Clips bin");
|
||||
push(&mut elems, "mcam.processedBin", &r, "Move source clips to Processed Clips bin");
|
||||
}
|
||||
}
|
||||
if !d.message.is_empty() {
|
||||
ui.colored_label(Color32::from_rgb(0xe0, 0x8a, 0x6a), &d.message);
|
||||
}
|
||||
ui.add_space(6.0);
|
||||
ui.horizontal(|ui| {
|
||||
let r = ui.button("Cancel");
|
||||
push(&mut elems, format!("{pre}.cancel"), &r, "Cancel");
|
||||
if r.clicked() {
|
||||
action = Some("cancel");
|
||||
}
|
||||
let r = ui.add(egui::Button::new(RichText::new("OK").color(Color32::WHITE)).fill(accent));
|
||||
push(&mut elems, format!("{pre}.ok"), &r, "OK");
|
||||
if r.clicked() {
|
||||
action = Some("ok");
|
||||
}
|
||||
});
|
||||
});
|
||||
for (id, r, l) in elems {
|
||||
app.auto.add(&id, r, &l);
|
||||
}
|
||||
if ctx.input(|i| i.key_pressed(egui::Key::Escape)) {
|
||||
action = Some("cancel");
|
||||
}
|
||||
match action {
|
||||
Some("cancel") => {
|
||||
app.ui.sync_dialog = None;
|
||||
return;
|
||||
}
|
||||
Some("ok") => {
|
||||
let mut p = json!({"method": d.method, "offset": d.offset, "ignoreHours": d.ignore_hours});
|
||||
if !d.marker.is_empty() {
|
||||
p["marker"] = json!(d.marker);
|
||||
}
|
||||
let cmd = match pre {
|
||||
"merge" => {
|
||||
p["items"] = json!(d.items);
|
||||
p["removeVideoAudio"] = json!(d.remove_video_audio);
|
||||
if !d.name.is_empty() {
|
||||
p["name"] = json!(d.name);
|
||||
}
|
||||
"clip.mergeClips"
|
||||
}
|
||||
"mcam" => {
|
||||
p["items"] = json!(d.items);
|
||||
p["audio"] = json!(d.audio);
|
||||
p["cameraNames"] = json!(d.camera_names);
|
||||
p["processedBin"] = json!(d.processed_bin);
|
||||
if !d.name.is_empty() {
|
||||
p["name"] = json!(d.name);
|
||||
}
|
||||
"clip.createMulticam"
|
||||
}
|
||||
_ => {
|
||||
if !d.track.is_empty() {
|
||||
p["track"] = json!(d.track);
|
||||
}
|
||||
"clip.synchronize"
|
||||
}
|
||||
};
|
||||
match app.session.execute(cmd, p) {
|
||||
Ok(v) => {
|
||||
app.ui.status = match pre {
|
||||
"sync" => format!("Synchronized {} clip group(s)", v["moved"]),
|
||||
"merge" => "Merged clip created".into(),
|
||||
_ => format!("Multi-camera source sequence with {} camera(s)", v["cameras"].as_array().map_or(0, Vec::len)),
|
||||
};
|
||||
app.ui.sync_dialog = None;
|
||||
return;
|
||||
}
|
||||
Err(e) => d.message = e.to_string(),
|
||||
}
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
app.ui.sync_dialog = Some(d);
|
||||
}
|
||||
|
||||
/// A revision for grid frames: changes when the multi-camera source (or its cameras' media)
|
||||
/// changes, not with every edit of the cut (cuts recorded live must not throw away the grid).
|
||||
fn grid_revision(app: &FilmcraftApp, src: ItemId) -> u64 {
|
||||
use std::hash::{Hash, Hasher};
|
||||
let mut h = std::collections::hash_map::DefaultHasher::new();
|
||||
if let Some(q) = app.session.project.sequence(src) {
|
||||
serde_json::to_vec(q).unwrap_or_default().hash(&mut h);
|
||||
for t in &q.video_tracks {
|
||||
for it in &t.items {
|
||||
app.item_revision(it.item).hash(&mut h);
|
||||
}
|
||||
}
|
||||
}
|
||||
h.finish()
|
||||
}
|
||||
|
||||
/// The angle grid of the Multi-Camera view in `area`.
|
||||
pub fn grid(app: &mut FilmcraftApp, ui: &mut egui::Ui, area: Rect) {
|
||||
let t = app.tokens;
|
||||
let ctx = ui.ctx().clone();
|
||||
ui.painter().rect_filled(area, 0.0, t.panel_bg);
|
||||
app.auto.add("program.multicam.grid", area, "Multi-Camera");
|
||||
let playhead = app.session.playhead();
|
||||
let info = filmcraft_engine::multicam::inspect_at(&app.session, playhead);
|
||||
let Some(src) = info["source"].as_u64().map(ItemId) else {
|
||||
ui.painter().text(area.center(), Align2::CENTER_CENTER, "No multi-camera clip at the playhead", Tokens::ui(12.0), t.text_dim);
|
||||
return;
|
||||
};
|
||||
let Some(q) = app.session.project.sequence(src) else { return };
|
||||
let cams = q.cameras();
|
||||
let shown = cams.shown_angles();
|
||||
let (cols, rows) = filmcraft_render::multicam::grid_dims(shown.len().max(1));
|
||||
let (sw, sh) = (q.settings.width as f32, q.settings.height as f32);
|
||||
let rate = q.settings.frame_rate;
|
||||
let pic = crate::panels::monitor::fit(area.shrink(2.0), sw * cols as f32, sh * rows as f32);
|
||||
let (cw, ch) = (pic.width() / cols as f32, pic.height() / rows as f32);
|
||||
// decode and composite at the cell's size (never more than the playback resolution)
|
||||
let cell_px = cw * ctx.pixels_per_point();
|
||||
let scale = crate::panels::monitor::quantize_scale(app.ui.program.res.scale().min((cell_px / sw).max(1.0 / 32.0)));
|
||||
let mt = Tick(info["sourceTime"].as_i64().unwrap_or(0));
|
||||
let frame = rate.frame_at(mt);
|
||||
let key = FrameKey { target: Target::MulticamGrid(src), frame, size: (scale * 1000.0) as u32, revision: grid_revision(app, src) };
|
||||
let project = app.session.project.clone();
|
||||
if app.playback.playing {
|
||||
app.frames.schedule_playback(key, rate, scale, &project, app.playback.speed, app.playback.preroll.is_some());
|
||||
} else {
|
||||
app.frames.request(key, rate.tick_of(frame), scale, &project, 0);
|
||||
}
|
||||
let tex = match app.frames.get(&key) {
|
||||
Some(img) => Some(app.texture_for(&ctx, "monitor-multicam", key, &img)),
|
||||
None => match app.frames.nearest(key.target, frame, key.size, key.revision, 6) {
|
||||
Some(img) => Some(app.texture_for(&ctx, "monitor-multicam", FrameKey { frame: frame - 1, ..key }, &img)),
|
||||
None => app.texture_existing("monitor-multicam").map(|(id, _)| id),
|
||||
},
|
||||
};
|
||||
ui.painter().rect_filled(pic, 0.0, t.monitor_bg);
|
||||
if let Some(tex) = tex {
|
||||
ui.painter().image(tex, pic, Rect::from_min_max(pos2(0.0, 0.0), pos2(1.0, 1.0)), Color32::WHITE);
|
||||
}
|
||||
let current = info["angle"].as_u64().map(|a| a as usize);
|
||||
let recording = app.session.mcrec.active();
|
||||
let mut click: Option<(usize, bool)> = None;
|
||||
for (k, &angle) in shown.iter().enumerate() {
|
||||
let cell = Rect::from_min_size(pic.min + vec2((k % cols) as f32 * cw, (k / cols) as f32 * ch), vec2(cw, ch));
|
||||
let name = cams.cameras.get(angle).map(|c| c.name.clone()).unwrap_or_default();
|
||||
let resp = ui.interact(cell, egui::Id::new(("multicam-angle", k)), Sense::click()).on_hover_text(format!("Camera {} — {name}", k + 1));
|
||||
app.auto.add(&format!("program.multicam.angle.{}", k + 1), cell, &name);
|
||||
if resp.clicked() {
|
||||
// modifiers held now, or carried by the click event (synthetic input)
|
||||
let video_only = ui.input(|i| {
|
||||
let ev = i.events.iter().any(|e| matches!(e, egui::Event::PointerButton { modifiers: m, .. } if m.ctrl || m.command));
|
||||
i.modifiers.ctrl || i.modifiers.command || ev
|
||||
});
|
||||
click = Some((k + 1, video_only));
|
||||
}
|
||||
if resp.hovered() {
|
||||
ui.painter().rect_stroke(cell.shrink(1.0), 0.0, Stroke::new(1.0, Color32::from_white_alpha(90)), StrokeKind::Inside);
|
||||
}
|
||||
if current == Some(angle) {
|
||||
// the active angle: yellow, red while recording
|
||||
let col = if recording { Color32::from_rgb(0xe0, 0x3a, 0x3a) } else { Color32::from_rgb(0xf2, 0xc9, 0x4c) };
|
||||
ui.painter().rect_stroke(cell.shrink(1.5), 0.0, Stroke::new(3.0, col), StrokeKind::Inside);
|
||||
}
|
||||
let label = format!("{} {name}", k + 1);
|
||||
let galley = ui.painter().layout_no_wrap(label, Tokens::ui(11.0), Color32::WHITE);
|
||||
let lr = Rect::from_min_size(cell.left_bottom() + vec2(6.0, -galley.size().y - 8.0), galley.size() + vec2(8.0, 4.0));
|
||||
ui.painter().rect_filled(lr, 3.0, Color32::from_black_alpha(160));
|
||||
ui.painter().galley(lr.min + vec2(4.0, 2.0), galley, Color32::WHITE);
|
||||
}
|
||||
// record indicator
|
||||
let rr = Rect::from_min_size(area.min + vec2(8.0, 8.0), vec2(64.0, 18.0));
|
||||
let rec_on = app.ui.multicam_record;
|
||||
let resp = ui.interact(rr, egui::Id::new("multicam-record"), Sense::click()).on_hover_text("Multi-Camera Record On/Off (0)");
|
||||
app.auto.add("program.multicam.record", rr, "Multi-Camera Record");
|
||||
let dot = if recording {
|
||||
Color32::from_rgb(0xe0, 0x3a, 0x3a)
|
||||
} else if rec_on {
|
||||
Color32::from_rgb(0x9a, 0x3a, 0x3a)
|
||||
} else {
|
||||
t.text_faint
|
||||
};
|
||||
ui.painter().rect_filled(rr, 3.0, Color32::from_black_alpha(140));
|
||||
ui.painter().circle_filled(rr.left_center() + vec2(9.0, 0.0), 4.5, dot);
|
||||
ui.painter().text(rr.left_center() + vec2(18.0, 0.0), Align2::LEFT_CENTER, if recording { "REC" } else { "Record" }, Tokens::ui(10.5), Color32::WHITE);
|
||||
if resp.clicked() {
|
||||
app.ui.multicam_record = !app.ui.multicam_record;
|
||||
}
|
||||
if let Some((n, video_only)) = click
|
||||
&& let Err(e) = app.session.execute("multicam.cut", json!({"camera": n, "videoOnly": video_only}))
|
||||
{
|
||||
app.ui.status = e.to_string();
|
||||
}
|
||||
}
|
||||
|
||||
/// Start a recording pass when playback starts in the Multi-Camera view.
|
||||
pub fn on_play(app: &mut FilmcraftApp) {
|
||||
if app.ui.program.multicam && app.ui.multicam_record && !app.session.mcrec.active() && (app.playback.speed - 1.0).abs() < 1e-9 {
|
||||
let t = app.session.playhead();
|
||||
if filmcraft_engine::multicam::multicam_clip_at(&app.session, t).is_some() {
|
||||
let _ = app.session.execute("multicam.recordStart", json!({"time": t.0}));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// End the recording pass (one undo step) when playback stops.
|
||||
pub fn on_stop(app: &mut FilmcraftApp) {
|
||||
if app.session.mcrec.active() {
|
||||
let t = app.session.playhead();
|
||||
match app.session.execute("multicam.recordStop", json!({"time": t.0})) {
|
||||
Ok(v) if v["cuts"].as_u64().unwrap_or(0) > 0 => app.ui.status = format!("Recorded {} multi-camera cut(s)", v["cuts"]),
|
||||
Ok(_) => {}
|
||||
Err(e) => app.ui.status = e.to_string(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -428,7 +428,18 @@ fn draw_clip(
|
|||
if let Some(aspect) = aspect {
|
||||
let tw = (th.height() * aspect).min(th.width());
|
||||
let mt = rate.snap(it.source_in);
|
||||
if let Some((tex, _)) = app.thumbnail(ctx, it.item, mt, 160) {
|
||||
// a multi-camera clip shows its angle's clip
|
||||
let (thumb_item, mt) = app
|
||||
.session
|
||||
.project
|
||||
.sequence(it.item)
|
||||
.and_then(|q| {
|
||||
let ti = q.angle_video_track_index(it.multicam_angle(q)?)?;
|
||||
let inner = q.video_tracks[ti].item_at(mt)?;
|
||||
Some((inner.item, inner.source_time_at(mt)))
|
||||
})
|
||||
.unwrap_or((it.item, mt));
|
||||
if let Some((tex, _)) = app.thumbnail(ctx, thumb_item, mt, 160) {
|
||||
let r = Rect::from_min_size(th.min, vec2(tw, th.height()));
|
||||
let cp = p.with_clip_rect(th.intersect(p.clip_rect()));
|
||||
cp.image(tex, r, Rect::from_min_max(pos2(0.0, 0.0), pos2((tw / (th.height() * aspect)).min(1.0), 1.0)), Color32::WHITE);
|
||||
|
|
@ -443,6 +454,10 @@ fn draw_clip(
|
|||
// name band
|
||||
if w > 24.0 {
|
||||
let mut label = it.name.clone();
|
||||
if let Some(m) = it.multicam.filter(|m| m.enabled) {
|
||||
let cam = app.session.project.sequence(it.item).and_then(|q| q.cameras().cameras.get(m.angle as usize).map(|c| c.name.clone()));
|
||||
label = format!("[MC{}] {}", m.angle + 1, cam.unwrap_or(label));
|
||||
}
|
||||
if (it.speed - 1.0).abs() > 1e-6 || it.reverse {
|
||||
label = format!("{label} [{}%]", ((if it.reverse { -1.0 } else { 1.0 }) * it.speed * 100.0).round());
|
||||
}
|
||||
|
|
|
|||
|
|
@ -220,11 +220,14 @@ pub struct MonitorView {
|
|||
pub zoom: Option<f32>,
|
||||
pub safe_margins: bool,
|
||||
pub show_transport: bool,
|
||||
/// Program monitor display mode Multi-Camera (angle grid + program).
|
||||
#[serde(default)]
|
||||
pub multicam: bool,
|
||||
}
|
||||
|
||||
impl Default for MonitorView {
|
||||
fn default() -> Self {
|
||||
Self { res: PlaybackRes::Half, zoom: None, safe_margins: false, show_transport: true }
|
||||
Self { res: PlaybackRes::Half, zoom: None, safe_margins: false, show_transport: true, multicam: false }
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -316,6 +319,12 @@ pub struct UiState {
|
|||
/// Effect presets: the preset being renamed / the Save Preset dialog (open when Some).
|
||||
#[serde(default)]
|
||||
pub save_preset: Option<SavePresetDraft>,
|
||||
/// Synchronize / Merge Clips / Create Multi-Camera Source Sequence dialog (open when Some).
|
||||
#[serde(default)]
|
||||
pub sync_dialog: Option<SyncDraft>,
|
||||
/// Multi-Camera Record On/Off (key 0): playing in the Multi-Camera view records cuts.
|
||||
#[serde(default = "yes")]
|
||||
pub multicam_record: bool,
|
||||
}
|
||||
|
||||
/// A pen mask in progress: vertices placed so far, in clip pixels (`[x, y, tangent x, tangent y]`).
|
||||
|
|
@ -337,6 +346,54 @@ pub struct SavePresetDraft {
|
|||
pub keyframes: String,
|
||||
}
|
||||
|
||||
/// Clip ▸ Synchronize…, Merge Clips… and Create Multi-Camera Source Sequence… share one draft;
|
||||
/// `kind` says which dialog it is (`synchronize`, `merge`, `multicam`).
|
||||
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
|
||||
#[serde(default)]
|
||||
pub struct SyncDraft {
|
||||
pub kind: String,
|
||||
/// Project items (Merge Clips, Create Multi-Camera).
|
||||
pub items: Vec<u64>,
|
||||
pub name: String,
|
||||
/// `in` | `out` | `timecode` | `marker` | `audio`
|
||||
pub method: String,
|
||||
pub ignore_hours: bool,
|
||||
/// Clip marker name to sync on (empty = the first marker).
|
||||
pub marker: String,
|
||||
/// Frames added to every clip but the reference.
|
||||
pub offset: i64,
|
||||
/// Synchronize: reference track (`V1`, `A2`…; empty = the lowest track).
|
||||
pub track: String,
|
||||
/// Create Multi-Camera: `camera1` | `all` | `switch`.
|
||||
pub audio: String,
|
||||
/// Create Multi-Camera: `clip` | `track` | `metadata`.
|
||||
pub camera_names: String,
|
||||
pub processed_bin: bool,
|
||||
/// Merge Clips: drop the video clip's own audio.
|
||||
pub remove_video_audio: bool,
|
||||
pub message: String,
|
||||
}
|
||||
|
||||
impl Default for SyncDraft {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
kind: "synchronize".into(),
|
||||
items: vec![],
|
||||
name: String::new(),
|
||||
method: "audio".into(),
|
||||
ignore_hours: false,
|
||||
marker: String::new(),
|
||||
offset: 0,
|
||||
track: String::new(),
|
||||
audio: "camera1".into(),
|
||||
camera_names: "clip".into(),
|
||||
processed_bin: true,
|
||||
remove_video_audio: false,
|
||||
message: String::new(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// File ▸ Link Media… (shown automatically when a project opens with missing media).
|
||||
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
|
||||
#[serde(default)]
|
||||
|
|
@ -508,6 +565,8 @@ impl Default for UiState {
|
|||
color_dialog: None,
|
||||
mask_pen: None,
|
||||
save_preset: None,
|
||||
sync_dialog: None,
|
||||
multicam_record: true,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -0,0 +1,301 @@
|
|||
//! Headless UI tests of multi-camera editing: the Create Multi-Camera Source Sequence, Synchronize
|
||||
//! and Merge Clips dialogs, the Program monitor's Multi-Camera view (angle grid, clicking an
|
||||
//! angle, Ctrl-click for video only) and live switching with the 1–9 keys while playing.
|
||||
//!
|
||||
//! Set `FILMCRAFT_UI_SNAPSHOT_DIR=<dir>` to also render the window offscreen with wgpu and write PNGs
|
||||
//! there (`multicam-*.png`).
|
||||
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::Arc;
|
||||
use std::sync::mpsc::{Sender, channel};
|
||||
|
||||
use egui_kittest::Harness;
|
||||
use filmcraft_engine::Session;
|
||||
use filmcraft_engine::project::{ItemKind, Label, MediaClip, MediaRef, Project, SequenceSettings, TrackKind};
|
||||
use filmcraft_media::generators::GeneratorSource;
|
||||
use filmcraft_media::{DemoScene, Generator, MediaSource};
|
||||
use filmcraft_time::{FrameRate, Tick, TimeRange};
|
||||
use filmcraft_ui_egui::FilmcraftApp;
|
||||
use filmcraft_ui_egui::control::ControlRequest;
|
||||
use serde_json::{Value, json};
|
||||
|
||||
const W: u32 = 320;
|
||||
const H: u32 = 180;
|
||||
|
||||
fn tmp_dir(name: &str) -> PathBuf {
|
||||
let nanos = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).unwrap().as_nanos();
|
||||
let d = std::env::temp_dir().join(format!("filmcraft-ui-{name}-{}-{nanos}", std::process::id()));
|
||||
std::fs::create_dir_all(&d).unwrap();
|
||||
d
|
||||
}
|
||||
|
||||
/// A 24 fps ProRes movie of a demo scene (our own encoder).
|
||||
fn make_movie(path: &Path, scene: DemoScene, frames: i64) {
|
||||
let rate = FrameRate::FPS_24;
|
||||
let dur = rate.tick_of(frames);
|
||||
let src = GeneratorSource::new(Generator::Demo(scene), W, H, rate, dur);
|
||||
let clip = MediaClip {
|
||||
media: MediaRef::Generator(Generator::Demo(scene)),
|
||||
info: src.info().clone(),
|
||||
interpret: Default::default(),
|
||||
mark_in: None,
|
||||
mark_out: None,
|
||||
markers: vec![],
|
||||
offline: false,
|
||||
proxy: None,
|
||||
identity: None,
|
||||
};
|
||||
let mut p = Project::new("fixture");
|
||||
let item = p.add_item("scene", Label::Iris, ItemKind::Media(clip), None);
|
||||
let seq = p.new_sequence("s", SequenceSettings { width: W, height: H, frame_rate: rate, ..Default::default() }, 1, 0, None);
|
||||
let mut v = p.make_track_item(item, TrackKind::Video, Tick::ZERO, TimeRange::new(Tick::ZERO, dur), rate).unwrap();
|
||||
for e in &mut v.effects {
|
||||
filmcraft_engine::project::resolve_auto_points(e, (W, H), (W, H));
|
||||
}
|
||||
p.sequence_mut(seq).unwrap().video_tracks[0].items.push(v);
|
||||
let settings = filmcraft_engine::export::ExportSettings {
|
||||
format: filmcraft_engine::export::Format::ProRes,
|
||||
path: path.to_string_lossy().into_owned(),
|
||||
include_audio: false,
|
||||
..Default::default()
|
||||
};
|
||||
let shared: filmcraft_media::SharedSource = Arc::new(src);
|
||||
let provider = move |id| (id == item).then(|| shared.clone());
|
||||
filmcraft_engine::export::export(&Arc::new(p), seq, &settings, &provider, &Default::default()).unwrap();
|
||||
}
|
||||
|
||||
/// Four cameras imported into a session with an empty 320×180 sequence "Cut".
|
||||
fn session() -> (Session, Vec<u64>) {
|
||||
let dir = tmp_dir("multicam");
|
||||
let scenes = [DemoScene::OceanSunset, DemoScene::CityNight, DemoScene::Aurora, DemoScene::Forest];
|
||||
let files: Vec<PathBuf> = (0..4).map(|i| dir.join(format!("Cam {}.mov", (b'A' + i as u8) as char))).collect();
|
||||
for (f, sc) in files.iter().zip(scenes) {
|
||||
make_movie(f, sc, 96);
|
||||
}
|
||||
let mut s = Session::default();
|
||||
let r = s.execute("file.import", json!({"paths": files.iter().map(|f| f.to_string_lossy()).collect::<Vec<_>>()})).unwrap();
|
||||
let items: Vec<u64> = r["items"].as_array().unwrap().iter().map(|v| v.as_u64().unwrap()).collect();
|
||||
let mut p = (*s.project).clone();
|
||||
let seq = p.new_sequence("Cut", SequenceSettings { width: W, height: H, frame_rate: FrameRate::FPS_24, ..Default::default() }, 2, 2, None);
|
||||
s.project = Arc::new(p);
|
||||
s.state.active_sequence = Some(seq);
|
||||
s.state.open_sequences = vec![seq];
|
||||
(s, items)
|
||||
}
|
||||
|
||||
struct Driver {
|
||||
harness: Harness<'static, FilmcraftApp>,
|
||||
tx: Sender<ControlRequest>,
|
||||
snapshots: Option<PathBuf>,
|
||||
}
|
||||
|
||||
impl Driver {
|
||||
fn new(session: Session) -> Self {
|
||||
let (tx, rx) = channel();
|
||||
let app = FilmcraftApp::new(session).with_control(rx);
|
||||
let snapshots = std::env::var_os("FILMCRAFT_UI_SNAPSHOT_DIR").map(PathBuf::from);
|
||||
let mut b = Harness::builder().with_size(egui::vec2(1600.0, 980.0)).with_max_steps(10_000);
|
||||
if snapshots.is_some() {
|
||||
b = b.wgpu();
|
||||
}
|
||||
let harness = b.build_eframe(move |_cc| app);
|
||||
let mut d = Driver { harness, tx, snapshots };
|
||||
d.frames(4);
|
||||
d
|
||||
}
|
||||
|
||||
fn frames(&mut self, n: usize) {
|
||||
for _ in 0..n {
|
||||
let ctx = self.harness.ctx.clone();
|
||||
let mut raw = std::mem::take(self.harness.input_mut());
|
||||
eframe::App::raw_input_hook(self.harness.state_mut(), &ctx, &mut raw);
|
||||
*self.harness.input_mut() = raw;
|
||||
self.harness.step();
|
||||
}
|
||||
}
|
||||
|
||||
/// Run frames (with short sleeps so frame workers finish) until `done`.
|
||||
fn until(&mut self, what: &str, mut done: impl FnMut(&mut Self) -> bool) {
|
||||
for _ in 0..600 {
|
||||
if done(self) {
|
||||
return;
|
||||
}
|
||||
self.frames(1);
|
||||
std::thread::sleep(std::time::Duration::from_millis(10));
|
||||
}
|
||||
panic!("timed out waiting for {what}");
|
||||
}
|
||||
|
||||
fn call(&mut self, method: &str, params: Value) -> Value {
|
||||
let (req, reply) = ControlRequest::new(method, params.clone());
|
||||
self.tx.send(req).unwrap();
|
||||
for _ in 0..600 {
|
||||
self.frames(1);
|
||||
if let Ok(v) = reply.try_recv() {
|
||||
return v;
|
||||
}
|
||||
}
|
||||
panic!("no reply to {method} {params}");
|
||||
}
|
||||
|
||||
fn ok(&mut self, method: &str, params: Value) -> Value {
|
||||
let v = self.call(method, params.clone());
|
||||
assert_eq!(v["ok"], json!(true), "{method} {params} failed: {v}");
|
||||
v["result"].clone()
|
||||
}
|
||||
|
||||
fn exec(&mut self, command: &str, params: Value) -> Value {
|
||||
self.ok("engine.execute", json!({"command": command, "params": params}))
|
||||
}
|
||||
|
||||
fn click(&mut self, id: &str) {
|
||||
self.ok("ui.click", json!({"id": id}));
|
||||
self.frames(3);
|
||||
}
|
||||
|
||||
fn ids(&mut self, prefix: &str) -> Vec<String> {
|
||||
let v = self.ok("ui.elements", json!({"prefix": prefix}));
|
||||
v.as_array().unwrap().iter().filter_map(|e| e["id"].as_str().map(str::to_string)).collect()
|
||||
}
|
||||
|
||||
fn app(&mut self) -> &mut FilmcraftApp {
|
||||
self.harness.state_mut()
|
||||
}
|
||||
|
||||
/// Angles of the clips on V1 / A1 of the active sequence.
|
||||
fn angles(&mut self, audio: bool) -> Vec<u32> {
|
||||
let q = self.app().session.active_sequence().unwrap().clone();
|
||||
let t = if audio { &q.audio_tracks[0] } else { &q.video_tracks[0] };
|
||||
t.items.iter().map(|i| i.multicam.map_or(99, |m| m.angle)).collect()
|
||||
}
|
||||
|
||||
fn snapshot(&mut self, name: &str) {
|
||||
let Some(dir) = self.snapshots.clone() else { return };
|
||||
self.frames(3);
|
||||
let img = match self.harness.render() {
|
||||
Ok(i) => i,
|
||||
Err(e) => {
|
||||
eprintln!("snapshot {name} skipped: {e}");
|
||||
return;
|
||||
}
|
||||
};
|
||||
std::fs::create_dir_all(&dir).unwrap();
|
||||
let path = dir.join(format!("{name}.png"));
|
||||
img.save(&path).unwrap();
|
||||
eprintln!("snapshot: {}", path.display());
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn create_multicam_dialog_view_and_live_switching() {
|
||||
let (s, items) = session();
|
||||
let mut d = Driver::new(s);
|
||||
// Create Multi-Camera Source Sequence from the Project panel selection
|
||||
d.exec("project.select", json!({"items": items}));
|
||||
d.ok("ui.menu.invoke", json!({"id": "clip.createMulticam"}));
|
||||
d.frames(3);
|
||||
assert_eq!(d.app().ui.sync_dialog.as_ref().map(|x| x.kind.clone()).as_deref(), Some("multicam"));
|
||||
for id in ["mcam.name", "mcam.method.audio", "mcam.method.timecode", "mcam.audio.switch", "mcam.cameraNames.track", "mcam.processedBin", "mcam.ok"] {
|
||||
assert!(!d.ids(id).is_empty(), "{id}");
|
||||
}
|
||||
d.click("mcam.method.in");
|
||||
d.click("mcam.audio.switch");
|
||||
d.snapshot("multicam-create-dialog");
|
||||
d.click("mcam.ok");
|
||||
assert!(d.app().ui.sync_dialog.is_none(), "dialog closed: {}", d.app().ui.status);
|
||||
let src = d.app().session.state.project_selection[0];
|
||||
let mc = d.app().session.project.sequence(src).unwrap().multicam.clone().unwrap();
|
||||
assert_eq!(mc.cameras.len(), 4);
|
||||
assert_eq!(mc.cameras[0].name, "Cam A.mov");
|
||||
// edit it into the sequence, open the Multi-Camera view
|
||||
d.exec("source.open", json!({"item": src.0}));
|
||||
d.exec("playhead.set", json!({"frame": 0}));
|
||||
d.exec("source.overwrite", json!({}));
|
||||
d.exec("playhead.set", json!({"frame": 12}));
|
||||
d.ok("ui.menu.invoke", json!({"id": "multicam.toggleView"}));
|
||||
d.frames(2);
|
||||
assert!(d.app().ui.program.multicam);
|
||||
let angles = d.ids("program.multicam.angle.");
|
||||
assert_eq!(angles.len(), 4, "{angles:?}");
|
||||
d.until("grid frame", |d| {
|
||||
let k = d.app().frames.queue_len();
|
||||
k == 0
|
||||
});
|
||||
d.frames(6);
|
||||
d.snapshot("multicam-view");
|
||||
// clicking an angle switches the clip at the playhead; audio follows video when asked
|
||||
d.exec("multicam.audioFollowsVideo", json!({"enabled": true}));
|
||||
d.click("program.multicam.angle.2");
|
||||
assert_eq!(d.angles(false), [1]);
|
||||
assert_eq!(d.angles(true), [1], "audio followed");
|
||||
// Ctrl-click: video only
|
||||
d.ok("ui.click", json!({"id": "program.multicam.angle.3", "modifiers": {"ctrl": true}}));
|
||||
d.frames(3);
|
||||
assert_eq!((d.angles(false), d.angles(true)), (vec![2], vec![1]));
|
||||
d.until("program frame", |d| d.app().frames.queue_len() == 0);
|
||||
d.frames(6);
|
||||
d.snapshot("multicam-view-camera3");
|
||||
d.exec("playhead.set", json!({"frame": 0}));
|
||||
d.click("program.multicam.angle.1");
|
||||
// live switching: play, press 2 then 4, stop → one undo step with the recorded cuts
|
||||
let undo_before = d.app().session.history.undo.len();
|
||||
d.ok("ui.playback", json!({"action": "play"}));
|
||||
d.until("recording", |d| d.app().session.mcrec.active());
|
||||
d.until("playhead past 0.3 s", |d| d.app().session.playhead() > FrameRate::FPS_24.tick_of(7));
|
||||
d.ok("ui.key", json!({"key": "2"}));
|
||||
d.frames(2);
|
||||
d.until("playhead past 0.8 s", |d| d.app().session.playhead() > FrameRate::FPS_24.tick_of(19));
|
||||
d.ok("ui.key", json!({"key": "4"}));
|
||||
d.frames(4);
|
||||
d.snapshot("multicam-recording");
|
||||
d.until("playhead past 1.2 s", |d| d.app().session.playhead() > FrameRate::FPS_24.tick_of(29));
|
||||
d.ok("ui.playback", json!({"action": "stop"}));
|
||||
d.frames(2);
|
||||
assert!(!d.app().session.mcrec.active());
|
||||
let v = d.angles(false);
|
||||
assert_eq!(&v[..3], [0, 1, 3], "{v:?}");
|
||||
assert_eq!(d.angles(true)[..3], [0, 1, 3], "audio followed the recording");
|
||||
assert_eq!(d.app().session.history.undo.len(), undo_before + 1, "one undo step per pass");
|
||||
d.frames(4);
|
||||
d.snapshot("multicam-after-recording");
|
||||
d.exec("edit.undo", json!({}));
|
||||
assert_eq!(d.angles(false), [0]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn synchronize_and_merge_dialogs() {
|
||||
let (s, items) = session();
|
||||
let mut d = Driver::new(s);
|
||||
// two cameras on V1 and V2 at different times
|
||||
d.exec("source.open", json!({"item": items[0]}));
|
||||
d.exec("playhead.set", json!({"frame": 0}));
|
||||
d.exec("source.overwrite", json!({}));
|
||||
d.exec("timeline.setTargeting", json!({"track": "V2", "sourcePatch": true}));
|
||||
d.exec("timeline.setTargeting", json!({"track": "A2", "sourcePatch": true}));
|
||||
d.exec("source.open", json!({"item": items[1]}));
|
||||
d.exec("playhead.set", json!({"frame": 30}));
|
||||
d.exec("source.overwrite", json!({}));
|
||||
assert_eq!(d.app().session.active_sequence().unwrap().video_tracks[1].items.len(), 1);
|
||||
let all: Vec<u64> = d.app().session.active_sequence().unwrap().all_tracks().flat_map(|t| t.items.iter().map(|i| i.id.0)).collect();
|
||||
d.exec("timeline.select", json!({"clips": all}));
|
||||
d.ok("ui.menu.invoke", json!({"id": "clip.synchronize"}));
|
||||
d.frames(3);
|
||||
assert_eq!(d.app().ui.sync_dialog.as_ref().map(|x| x.kind.clone()).as_deref(), Some("synchronize"));
|
||||
d.click("sync.method.in");
|
||||
d.snapshot("multicam-sync-dialog");
|
||||
d.click("sync.ok");
|
||||
assert!(d.app().ui.sync_dialog.is_none(), "{}", d.app().ui.status);
|
||||
let q = d.app().session.active_sequence().unwrap().clone();
|
||||
assert_eq!(q.video_tracks[0].items[0].start, q.video_tracks[1].items[0].start, "In points aligned");
|
||||
// Merge Clips needs audio clips: the dialog explains what is wrong
|
||||
d.exec("project.select", json!({"items": [items[0], items[1]]}));
|
||||
d.ok("ui.menu.invoke", json!({"id": "clip.mergeClips"}));
|
||||
d.frames(3);
|
||||
assert!(!d.ids("merge.removeVideoAudio").is_empty());
|
||||
d.snapshot("multicam-merge-dialog");
|
||||
d.click("merge.ok");
|
||||
let msg = d.app().ui.sync_dialog.as_ref().map(|x| x.message.clone()).unwrap_or_default();
|
||||
assert!(msg.contains("one video clip"), "{msg}");
|
||||
d.click("merge.cancel");
|
||||
assert!(d.app().ui.sync_dialog.is_none());
|
||||
}
|
||||
|
|
@ -384,6 +384,47 @@ frame (Y'CbCr/RGB + metadata) ─► source colour space: Interpret Footage over
|
|||
conversions and looks). `filmcraft-gpu::GpuLut` is the WGSL tetrahedral counterpart, tested for
|
||||
parity.
|
||||
|
||||
### 5.4 Multi-camera and synchronisation
|
||||
|
||||
```text
|
||||
clips ──sync (in | out | timecode[±hours] | marker | audio)──► anchors (media time ↔ common instant)
|
||||
├─ clip.synchronize moves selected timeline clips (link groups together) onto the reference
|
||||
├─ clip.mergeClips video + ≤16 audio clips → a merged-clip sequence
|
||||
└─ clip.createMulticam cameras → a multi-camera source sequence (one video track per angle)
|
||||
multi-camera clip = nested source + TrackItem::multicam {enabled, angle}
|
||||
render: only the angle's video track · audio: camera 1 | all | the angle (Switch Audio)
|
||||
```
|
||||
|
||||
- **Model** (`project::multicam`). `Sequence::multicam` (`MulticamSource`: cameras with their video
|
||||
track, audio tracks, name, shown flag and source item; audio mode) marks a multi-camera source;
|
||||
`Sequence::merged` a merged clip. `TrackItem::multicam` (`MulticamSel`) makes a nested clip a
|
||||
multi-camera clip; any nest can be one (its video tracks are then the angles,
|
||||
`Sequence::cameras()`). Editing a multi-camera source into a sequence gives an enabled clip on the
|
||||
first angle (`Project::make_track_item`). Project schema v7.
|
||||
- **Sync** (`engine::sync`). Each method reduces a clip to an *anchor* (the media time that lines up
|
||||
with the common instant). Audio uses `audio_dsp::sync::find_offset`: DC removal, windowed-sinc
|
||||
decimation to ≤ 8 kHz, GCC-PHAT-β (β = 0.75) via one packed complex FFT for the coarse lag, then
|
||||
the same at the full rate on the loudest common window (≤ 2.7 s) and parabolic interpolation.
|
||||
Recordings with different gains, microphones (filtered), 0 dB SNR noise or a strong echo are
|
||||
aligned to the sample; two 10-minute recordings take ~2.4 s (release). Clips are placed on frame
|
||||
boundaries with the sub-frame remainder taken from their source In, so video stays on frames
|
||||
while audio keeps sample accuracy.
|
||||
- **Render.** `render::item_layer` renders a multi-camera clip's angle track only
|
||||
(`render_seq_tracks`); `render::audio` mixes the nested source with only the audible tracks
|
||||
(`Sequence::with_angle_audio`). Nested audio now plays at all (it was skipped when the nest had
|
||||
no media source) and is limited to the clip's range. `render::multicam::render_grid` renders the
|
||||
shown angles at the cell scale in parallel (rayon) and tiles them: the Multi-Camera view is one
|
||||
frame job (`frames::Target::MulticamGrid`), prefetched while playing like the program.
|
||||
- **Editing** (`edit::multicam`, `engine::multicam`). `multicam.switchAngle` (click an angle,
|
||||
Ctrl/⌘-click for video only), `multicam.selectCamera1…9` (keys 1–9) and `cutToCamera1…9`
|
||||
(Ctrl+1–9), Enable/Flatten, Edit Cameras, Audio Follows Video (`EditorState`). Live switching:
|
||||
playback in the Multi-Camera view runs `multicam.recordStart`; each key/click is a
|
||||
`multicam.cut` that is applied at once (the program shows it) by re-applying the whole pass to
|
||||
the project from before the pass (`edit_merged`), so a pass is one undo step; Stop runs
|
||||
`multicam.recordStop`, which ends the last angle at the stop point. Through edits inside the
|
||||
recorded range are healed, so pressing the angle already showing adds no edit. Flatten replaces
|
||||
a clip by the clip(s) its angle shows (outer effects carried over; linked pairs stay linked).
|
||||
|
||||
## 6. Export jobs (`filmcraft-export`)
|
||||
|
||||
```text
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@ writing them is `crates/format` (`filmcraft-format`); the engine's `file.*` comm
|
|||
```json
|
||||
{
|
||||
"format": "filmcraft.project",
|
||||
"schema_version": 7,
|
||||
"schema_version": 8,
|
||||
"generator": "FilmCraft 0.1.0",
|
||||
"project": { "name": "…", "settings": { … }, "root": { … }, "items": { … }, "next_id": 48 }
|
||||
}
|
||||
|
|
@ -31,6 +31,7 @@ writing them is `crates/format` (`filmcraft-format`); the engine's `file.*` comm
|
|||
| 5 | M8.8 | colour management: working space, Interpret Footage colour space, LUT library; no-op step |
|
||||
| 6 | M11.7/M11.9 | media identity fingerprints, ingest settings; no-op step |
|
||||
| 7 | M5.5 | effect masks: `masks` on effect instances with keyframable Bézier `Path` values; no-op step |
|
||||
| 8 | M12 | multi-camera source sequences and clips, merged clips; no-op step |
|
||||
|
||||
### Migrations
|
||||
|
||||
|
|
@ -58,8 +59,8 @@ on disk is unchanged until you save. The **first save over it** keeps the origin
|
|||
|
||||
A file with a `schema_version` newer than the build supports is **refused**, not half-read:
|
||||
|
||||
> this project was saved by a newer version of FilmCraft (project schema v7); this build reads up to
|
||||
> v6. Update FilmCraft to open it.
|
||||
> this project was saved by a newer version of FilmCraft (project schema v9); this build reads up to
|
||||
> v8. Update FilmCraft to open it.
|
||||
|
||||
Reading it partially and saving it back would silently drop whatever the newer version added.
|
||||
|
||||
|
|
|
|||
|
|
@ -20,6 +20,7 @@ FILMCRAFT_REQUIRE_ORACLES=1 cargo test --workspace # CI: missing ffmpeg fails
|
|||
| Golden images | `crates/golden/tests/golden.rs` | CPU renders vs committed PNGs; GPU vs CPU on the same scenes (§3) |
|
||||
| Engine / command | `crates/engine/src/tests.rs` | run commands on the demo project, assert the sequence, undo/redo, disabled cases |
|
||||
| Render | `crates/render/src/tests.rs` | compositing, opacity, Motion, cross dissolve midpoint, ½-res vs full, GPU plan vs reference, audio mix, audio-effect continuity |
|
||||
| Multi-camera / sync | `crates/audio-dsp/src/sync.rs`, `crates/engine/src/multicam_tests.rs`, `crates/ui-egui/tests/multicam_ui.rs` | audio offsets within one sample: Rust-generated multi-mic speech (gains −20…+10 dB, SNR 40/25/10/3 dB, zero-phase colouring, fractional delays, a 0 dB-SNR room with a −4 dB echo) and ffmpeg-made camera/recorder files (pink-noise bursts, independent noise; exact lags recovered); timecode (± hours) and marker sync within a frame; Merge Clips; timeline Synchronize keeps frame-aligned starts; multicam structure, angle render (PSNR vs the camera), Switch Audio mix = the angle's track, live switching from a simulated playback clock (cuts, healing, one undo step), Ctrl+N cuts, flatten (frames identical), save/load; `perf_grid_of_four_1080p_angles` (ignored) prints the grid cost |
|
||||
| Mixer | `crates/render/src/mixer_tests.rs`, `crates/engine/src/mixer_tests.rs` | sample-exact fader gain and pan laws, automation at block boundaries (bit-identical however requests are cut), solo/mute/solo-safe, sends and submix routing, latency-compensation alignment, render-vs-playback identity, Touch ramp-back, recorder modes (Latch/Touch/Write), thinning, Audio Gain, transition curves; `perf_24_tracks_3_effects_realtime_factor` prints the realtime factor |
|
||||
| Oracle | `crates/*/tests/*oracle*.rs`, `conformance.rs` | compare with ffmpeg/ffprobe (§2) |
|
||||
| Robustness / fuzz | `*/tests/robustness.rs`, `*/tests/fuzz.rs` | seeded mutation and truncation of real and synthetic files; nothing may panic |
|
||||
|
|
|
|||
Loading…
Reference in New Issue