From b66074a77ce3ce8c845bdd21990c5b9c281afb50 Mon Sep 17 00:00:00 2001 From: Brandon Thomas Date: Thu, 1 Oct 2026 09:13:56 -0400 Subject: [PATCH] M11.8: Project Manager (collect files, consolidate and transcode) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - file.projectManager {destination, mode: collect|consolidate, sequences?, excludeUnused, handles, preset, includeProxies, includePreviews, dryRun, wait}. - Collect copies every file the chosen sequences use (nested sequences and subclips followed), optionally proxies and render previews; consolidate writes only the used media range plus handles with a transcode preset and re-bases media time so edits, keyframes and markers stay put. Unused clips can be excluded. - Dry run reports original / resulting disk space and the file plan; the copy runs as a job and writes the project last with the new paths and fingerprints. - UI: Project Manager dialog (sequences, mode, preset, options, destination, Calculate → sizes) with automation ids pm.*. - Tests: collected project opens with its media moved away and renders the same frames; consolidate trims to used range + handles and matches (>38 dB) the original. - ROADMAP: M11 done. Co-Authored-By: Claude Opus 5.5 --- ROADMAP.md | 3 +- crates/engine/src/commands.rs | 1 + crates/engine/src/lib.rs | 3 + crates/engine/src/project_manager.rs | 355 +++++++++++++++++++++ crates/engine/src/project_manager_tests.rs | 102 ++++++ crates/ui-egui/src/panels/media_dialogs.rs | 176 +++++++++- crates/ui-egui/src/state.rs | 39 +++ crates/ui-egui/tests/media_ui.rs | 12 +- docs/control-protocol.md | 4 +- docs/project-files.md | 25 ++ 10 files changed, 712 insertions(+), 8 deletions(-) create mode 100644 crates/engine/src/project_manager.rs create mode 100644 crates/engine/src/project_manager_tests.rs diff --git a/ROADMAP.md b/ROADMAP.md index 07f80214..fdbf2b86 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -39,7 +39,7 @@ Status: ✅ done · 🟡 in progress · ⬜ not started. Estimates are remaining | M8 | Colour | 🟡 | Lumetri: basic, creative + looks, RGB & hue curves, wheels, HSL secondary, vignette; basic scopes | LUT import UI, colour match, colour management + HDR (PQ/HLG, log, tone mapping) | 8–12 | | M9 | More codecs | 🟡 | ProRes decode+encode, AAC decode+encode, HEVC Main/Main 10 decoder (bit-exact on 41 fixtures, ~225 fps 1080p), VP9 decoder (profiles 0–3, 8/10/12-bit, bit-exact on 50+ fixtures; WebM/MKV `V_VP9` and MP4 `vp09` import with key-frame-checked seeking), Matroska/WebM import (H.264/HEVC/VP9/ProRes/MJPEG + AAC/Opus/FLAC/MP3/Vorbis/PCM), Opus decoder (SILK/CELT/hybrid, 5.1/7.1 multistream; all RFC 8251 vectors range-exact; WebM/MKV/MP4) | VP9 frame threading, Ogg Opus files, AV1 (rav1d), DNxHR, MXF, hardware decode | 12–20 | | M10 | Graphics & captions | 🟡 | Caption tracks (Subtitle/CEA-608/708/Teletext formats, track style), SRT/WebVTT/SCC import+export (frame-exact, property-tested), caption editing (add/split/merge/trim/move, sync-locked insert/extract), Text panel Captions tab, burn-in in Program monitor and export; text engine (`crates/text`: bundled + system fonts, harfrust shaping, bidi, line breaking, paragraph layout, glyph cache; 3-line 1080p title ≈ 0.15 ms warm); graphic clips with text + shape layers (fill, 2 strokes, background, shadow, keyframable transform), Type tool with on-monitor editing, shape/pen tools, Properties/Essential Graphics editor, align/distribute | Responsive design pins, rolls/crawls, per-character styles, motion graphics templates, MCC/STL/TTML, 608/708 embedding, speech-to-text | 12–18 | -| M11 | Interchange & project management | 🟡 | `.fcproj` schema versions + migrations, atomic saves, Save a Copy/Revert, auto-save ring + crash-recovery journal (Preferences ▸ Auto Save, recovery prompt), FCP7 XML, FCPXML, EDL, OTIO | Relink/offline, project manager, proxies | 8 | +| M11 | Interchange & project management | ✅ | `.fcproj` schema versions + migrations, atomic saves, Save a Copy/Revert, auto-save ring + crash-recovery journal (Preferences ▸ Auto Save, recovery prompt), FCP7 XML, FCPXML, EDL, OTIO; offline media (own slate) + Link Media (fingerprint-checked relink, folder remap, search, Align Timecode, Make Offline); proxies (ProRes Proxy/LT, H.264 ¼/½ background jobs, attach/detach/reconnect, monitor toggle, export full-res) + ingest (copy/transcode/proxies); Project Manager (collect, consolidate + transcode with handles, size estimate) | Rename media to clip names, image-sequence conversion, Media Browser-driven relink, smart (cross-drive) path tracking | 1–2 | | M12–M16 | Web (WASM), platform, long tail | 🟡 | L0–L4 crates compile to wasm32 | Web app shell (file access, WebCodecs, audio), ~850 remaining commands and dialogs, performance hardening | 40–65 | ## Running now @@ -48,6 +48,7 @@ Status: ✅ done · 🟡 in progress · ⬜ not started. Estimates are remaining ## Log +- **2026-10-01 (evening):** M11 done: offline media with our own slate and the Link Media dialog (fingerprint-checked relink, folder remap, search), proxies (create in background / attach / toggle; export stays full-res) with ingest settings, Project Manager (collect, consolidate + transcode). Fixed: GOP cache deadlock when a rayon decoder inside a parallel export re-entered the same source. - **2026-10-01 (later):** text engine (shaping, bidi, line breaking) + Type/Shape/Pen tools + graphic clips + graphics panel; Trim Monitor + dynamic trimming; Keyboard Shortcuts editor; audio mixer with automation. Fixed: MP4 muxer wrote unreadable all-empty sample tables; system font scan race. - **2026-10-01:** Audio mixing (M7.2/M7.5/M7.6 basics): mixer graph with submixes, sends, inserts and latency compensation; Audio Track / Clip Mixer panels; Latch/Touch/Write automation recorded live; timeline track keyframes; Audio Gain dialog. diff --git a/crates/engine/src/commands.rs b/crates/engine/src/commands.rs index bca4e4a1..46420587 100644 --- a/crates/engine/src/commands.rs +++ b/crates/engine/src/commands.rs @@ -1867,6 +1867,7 @@ fn build() -> Vec { v.extend(crate::shortcuts::commands()); v.extend(crate::relink::commands()); v.extend(crate::proxies::commands()); + v.extend(crate::project_manager::commands()); // Labels as individual commands (Edit ▸ Label ▸ ) for l in Label::ALL { let _ = l; diff --git a/crates/engine/src/lib.rs b/crates/engine/src/lib.rs index ee7b4b8c..b242f901 100644 --- a/crates/engine/src/lib.rs +++ b/crates/engine/src/lib.rs @@ -17,6 +17,7 @@ pub mod interchange; pub mod media_pool; pub mod mixer; pub mod previews; +pub mod project_manager; pub mod proxies; pub mod relink; pub mod shortcut_presets; @@ -611,6 +612,8 @@ mod mixer_tests; #[cfg(test)] mod previews_tests; #[cfg(test)] +mod project_manager_tests; +#[cfg(test)] mod proxies_tests; #[cfg(test)] mod relink_tests; diff --git a/crates/engine/src/project_manager.rs b/crates/engine/src/project_manager.rs new file mode 100644 index 00000000..04241876 --- /dev/null +++ b/crates/engine/src/project_manager.rs @@ -0,0 +1,355 @@ +//! File ▸ Project Manager…: make a self-contained copy of a project. +//! +//! - **Collect Files and Copy to New Location** copies every media file the chosen sequences use +//! (optionally their proxies and render previews) into one folder, and writes the project there +//! with its paths pointing at the copies. +//! - **Consolidate and Transcode** writes only the used part of each clip (plus handles) with a +//! transcode preset, and re-bases the clips' media time so every edit, keyframe and marker stays +//! where it was. +//! +//! Both can exclude unused clips. A dry run reports the disk space before and after. The copy +//! runs as a background job; the project file is written last, with the outputs' fingerprints. + +use std::collections::{BTreeMap, BTreeSet}; +use std::path::{Path, PathBuf}; +use std::sync::atomic::Ordering; + +use filmcraft_media::MediaKind; +use filmcraft_project::{ItemId, ItemKind, MediaRef, Project}; +use filmcraft_time::{Tick, TimeRange}; +use serde_json::{Value, json}; + +use crate::commands::{CommandSpec, bad, bool_p, str_p, u64_p}; +use crate::proxies::{Preset, preset, unique_path}; +use crate::{EngineError, Result, Session}; + +/// Media time used per item (min, max) across the sequences, following nested sequences. +pub fn used_ranges(p: &Project, seqs: &[ItemId]) -> (BTreeMap, BTreeSet) { + let mut ranges: BTreeMap = BTreeMap::new(); + let mut used: BTreeSet = BTreeSet::new(); + let mut stack: Vec = seqs.to_vec(); + while let Some(sid) = stack.pop() { + if !used.insert(sid) { + continue; + } + let Some(seq) = p.sequence(sid) else { continue }; + for t in seq.all_tracks() { + for ti in &t.items { + let mut target = ti.item; + match p.item(ti.item).map(|i| &i.kind) { + Some(ItemKind::Sequence(_)) => { + stack.push(ti.item); + continue; + } + Some(ItemKind::Subclip { parent, .. }) => { + used.insert(ti.item); + target = *parent; + } + Some(_) => {} + None => continue, + } + used.insert(target); + let a = ti.source_time_at(ti.start); + let b = ti.source_time_at(ti.end() - Tick(1)); + let (lo, hi) = (a.min(b), a.max(b) + Tick(1)); + let e = ranges.entry(target).or_insert((lo, hi)); + e.0 = e.0.min(lo); + e.1 = e.1.max(hi); + } + } + } + (ranges, used) +} + +struct Work { + item: ItemId, + src_path: String, + out: String, + /// Consolidate: the media range to write; None = copy the file. + range: Option, + proxy: Option<(String, String)>, +} + +/// Plan, estimate and (unless `dryRun`) run the Project Manager. +pub fn run(s: &mut Session, p: &Value) -> Result { + let dest = str_p(p, "destination").ok_or_else(|| bad("file.projectManager", "need `destination`"))?.to_string(); + let consolidate = match str_p(p, "mode").unwrap_or("collect") { + "collect" => false, + "consolidate" | "transcode" => true, + m => return Err(bad("file.projectManager", format!("unknown mode `{m}` (collect | consolidate)"))), + }; + let exclude_unused = bool_p(p, "excludeUnused").unwrap_or(true); + let handles = u64_p(p, "handles").unwrap_or(30) as i64; + let include_proxies = bool_p(p, "includeProxies").unwrap_or(!consolidate) && !consolidate; + let include_previews = bool_p(p, "includePreviews").unwrap_or(false); + let pr: &'static Preset = + preset(str_p(p, "preset").unwrap_or(crate::proxies::DEFAULT_TRANSCODE_PRESET)).ok_or_else(|| bad("file.projectManager", "unknown preset"))?; + let seqs: Vec = match p.get("sequences").and_then(Value::as_array) { + Some(a) => a.iter().filter_map(Value::as_u64).map(ItemId).collect(), + None => s.project.sequences().map(|i| i.id).collect(), + }; + if seqs.iter().any(|q| s.project.sequence(*q).is_none()) { + return Err(bad("file.projectManager", "`sequences` must list sequence ids")); + } + let (ranges, used) = used_ranges(&s.project, &seqs); + let name = str_p(p, "projectName") + .map(str::to_string) + .or_else(|| s.path.as_deref().and_then(|x| Path::new(x).file_stem()).map(|n| n.to_string_lossy().into_owned())) + .unwrap_or_else(|| s.project.name.clone()); + let dest_dir = PathBuf::from(&dest); + let project_path = dest_dir.join(format!("{name}.fcproj")).to_string_lossy().into_owned(); + + // ---- the new project + let mut proj = (*s.project).clone(); + if exclude_unused { + let keep: BTreeSet = proj + .items + .values() + .filter(|i| { + used.contains(&i.id) + || matches!(i.kind, ItemKind::Graphic { .. }) + || (!matches!(i.kind, ItemKind::Media(_) | ItemKind::Subclip { .. } | ItemKind::Sequence(_))) + }) + .map(|i| i.id) + .collect(); + let drop: Vec = proj.items.keys().filter(|i| !keep.contains(i)).copied().collect(); + for i in drop { + proj.items.remove(&i); + proj.root.remove_item(i); + } + } + // ---- work list and estimate + let mut work = Vec::new(); + let mut taken: Vec = Vec::new(); + let mut original_bytes = 0u64; + let mut result_bytes = 0u64; + let fd_handles = |rate: filmcraft_time::FrameRate| Tick(rate.frame_duration().0 * handles); + let ids: Vec = proj.items.keys().copied().collect(); + for id in ids { + let Some(m) = proj.item(id).and_then(|i| i.as_media()).cloned() else { continue }; + let MediaRef::File { path } = &m.media else { continue }; + let size = s.services.file_size(path).unwrap_or(0); + original_bytes += size; + let file = Path::new(path); + let stem = file.file_stem().map(|n| n.to_string_lossy().into_owned()).unwrap_or_else(|| format!("media-{}", id.0)); + let ext = file.extension().map(|n| n.to_string_lossy().into_owned()).unwrap_or_default(); + let transcode = consolidate && matches!(m.info.kind, MediaKind::Movie | MediaKind::AudioOnly) && ranges.contains_key(&id); + let (out, range) = if transcode { + let rate = m.frame_rate(); + let (lo, hi) = ranges[&id]; + let a = rate.snap((lo - fd_handles(rate)).max(Tick::ZERO)); + let b = (rate.snap(hi + fd_handles(rate) + rate.frame_duration() - Tick(1))).min(m.info.duration).max(a + rate.frame_duration()); + let ext = if m.info.video.is_some() { pr.extension } else { "wav" }; + let out = unique_path(&dest_dir, &stem, "", ext, &taken).to_string_lossy().into_owned(); + result_bytes += estimate_bytes(&m, b - a, pr); + (out, Some(TimeRange::from_bounds(a, b))) + } else { + result_bytes += size; + (unique_path(&dest_dir, &stem, "", &ext, &taken).to_string_lossy().into_owned(), None) + }; + taken.push(out.clone()); + let proxy = match (&m.proxy, include_proxies) { + (Some(MediaRef::File { path: pp }), true) if s.services.file_size(pp).is_ok() => { + let pf = Path::new(pp); + let ps = pf.file_stem().map(|n| n.to_string_lossy().into_owned()).unwrap_or_default(); + let pe = pf.extension().map(|n| n.to_string_lossy().into_owned()).unwrap_or_default(); + let o = unique_path(&dest_dir.join("Proxies"), &ps, "", &pe, &taken).to_string_lossy().into_owned(); + taken.push(o.clone()); + let b = s.services.file_size(pp).unwrap_or(0); + original_bytes += b; + result_bytes += b; + Some((pp.clone(), o)) + } + _ => None, + }; + work.push(Work { item: id, src_path: path.clone(), out, range, proxy }); + } + let previews = s.path.as_deref().map(crate::previews::dir_for_project).filter(|d| include_previews && d.is_dir()); + if let Some(d) = &previews { + let b = dir_size(d); + original_bytes += b; + result_bytes += b; + } + let files: Vec = work + .iter() + .map(|w| json!({"item": w.item.0, "from": w.src_path, "to": w.out, "trim": w.range.map(|r| json!({"start": r.start.0, "end": r.end().0}))})) + .collect(); + let summary = json!({ + "project": project_path, + "mode": if consolidate { "consolidate" } else { "collect" }, + "originalBytes": original_bytes, + "resultBytes": result_bytes, + "files": files, + "excluded": s.project.items.len() - proj.items.len(), + }); + if bool_p(p, "dryRun").unwrap_or(false) { + return Ok(summary); + } + if Path::new(&project_path).exists() && !bool_p(p, "overwrite").unwrap_or(false) { + return Err(EngineError::Other(format!("{project_path} already exists"))); + } + std::fs::create_dir_all(&dest_dir).map_err(|e| EngineError::Other(format!("{dest}: {e}")))?; + + // ---- the job + let sources: BTreeMap = + work.iter().filter(|w| w.range.is_some()).filter_map(|w| s.media.full_res_source(&s.project, w.item, &*s.services).map(|src| (w.item, src))).collect(); + if let Some(w) = work.iter().find(|w| w.range.is_some() && s.media.offline_status(w.item).is_some()) { + return Err(EngineError::Other(format!("{} is offline; link it before consolidating", w.src_path))); + } + let total: u64 = work + .iter() + .map(|w| w.range.map_or(1, |r| proj.item(w.item).and_then(|i| i.as_media()).map_or(1, |m| crate::proxies::frame_count(m, Some(r))))) + .sum::() + + 1; + let id = s.jobs.iter().map(|j| j.id).max().unwrap_or(0) + 1; + let job = crate::Job { id, label: format!("Project Manager: {name}"), progress: Default::default(), result: Default::default() }; + job.progress.total.store(total, Ordering::Relaxed); + let (prog, res) = (job.progress.clone(), job.result.clone()); + let prev_dst = crate::previews::dir_for_project(&project_path); + let services = s.services.clone(); + let run = move || { + let t0 = std::time::Instant::now(); + let r = execute(proj, work, sources, previews.map(|d| (d, prev_dst)), &project_path, pr, &prog, &*services); + let secs = t0.elapsed().as_secs_f64(); + let r = r.map(|bytes| filmcraft_export::Report { + path: project_path.clone(), + frames: prog.done.load(Ordering::Relaxed), + seconds: secs, + bytes, + render_fps: 0.0, + }); + if let Err(e) = &r { + *prog.error.lock().unwrap_or_else(|x| x.into_inner()) = Some(e.clone()); + } + *prog.status.lock().unwrap_or_else(|e| e.into_inner()) = match &r { + Ok(_) => format!("Done in {secs:.1}s"), + Err(e) => e.clone(), + }; + prog.finished.store(true, Ordering::Relaxed); + *res.lock().unwrap_or_else(|x| x.into_inner()) = Some(r); + }; + s.jobs.push(job); + if bool_p(p, "wait").unwrap_or(false) || cfg!(target_arch = "wasm32") { + run(); + } else { + std::thread::Builder::new().name("filmcraft-project-manager".into()).spawn(run).map_err(|e| EngineError::Other(e.to_string()))?; + } + let mut out = summary; + out["job"] = json!(id); + Ok(out) +} + +#[allow(clippy::too_many_arguments)] +fn execute( + mut proj: Project, + work: Vec, + sources: BTreeMap, + previews: Option<(PathBuf, PathBuf)>, + project_path: &str, + pr: &Preset, + prog: &filmcraft_export::Progress, + services: &dyn crate::Services, +) -> std::result::Result { + let mut bytes = 0; + let openers = filmcraft_codecs::openers(); + for w in &work { + if prog.cancel.load(Ordering::Relaxed) { + return Err("cancelled".into()); + } + *prog.status.lock().unwrap_or_else(|e| e.into_inner()) = format!("{}", Path::new(&w.out).file_name().map(|n| n.to_string_lossy()).unwrap_or_default()); + if let Some(d) = Path::new(&w.out).parent() { + std::fs::create_dir_all(d).map_err(|e| format!("{}: {e}", d.display()))?; + } + match w.range { + None => { + bytes += std::fs::copy(&w.src_path, &w.out).map_err(|e| format!("copying {}: {e}", w.src_path))?; + prog.done.fetch_add(1, Ordering::Relaxed); + } + Some(r) => { + let m = proj.item(w.item).and_then(|i| i.as_media()).cloned().ok_or("item vanished")?; + let name = proj.item(w.item).map(|i| i.name.clone()).unwrap_or_default(); + let src = sources.get(&w.item).cloned().ok_or_else(|| format!("{name}: no source"))?; + let rep = crate::proxies::transcode(&m, &name, src, Some(r), &w.out, pr, prog)?; + bytes += rep.bytes; + // the clip now starts at the range start: shift its uses, take the new file's info + proj.shift_media_time(w.item, r.start); + let b = std::fs::read(&w.out).map_err(|e| format!("{}: {e}", w.out))?; + let fname = Path::new(&w.out).file_name().map(|n| n.to_string_lossy().into_owned()).unwrap_or_default(); + let info = filmcraft_media::open_bytes(&fname, b.into(), &openers).map_err(|e| format!("{}: {e}", w.out))?.info().clone(); + if let Some(mm) = proj.item_mut(w.item).and_then(|i| i.as_media_mut()) { + mm.info = info; + mm.proxy = None; + } + } + } + if let Some((from, to)) = &w.proxy { + bytes += std::fs::create_dir_all(Path::new(to).parent().unwrap_or(Path::new("."))) + .and_then(|_| std::fs::copy(from, to)) + .map_err(|e| format!("copying {from}: {e}"))?; + } + let identity = crate::relink::identity_of(services, &w.out).ok(); + if let Some(mm) = proj.item_mut(w.item).and_then(|i| i.as_media_mut()) { + mm.media = MediaRef::File { path: w.out.clone() }; + mm.offline = false; + mm.identity = identity; + mm.proxy = match (&w.proxy, w.range) { + (Some((_, to)), None) => Some(MediaRef::File { path: to.clone() }), + _ => None, + }; + } + } + if let Some((from, to)) = previews { + bytes += copy_dir(&from, &to).map_err(|e| format!("copying previews: {e}"))?; + } + let data = filmcraft_format::encode(&proj, false); + filmcraft_format::atomic_write(Path::new(project_path), &data).map_err(|e| format!("{project_path}: {e}"))?; + prog.done.fetch_add(1, Ordering::Relaxed); + Ok(bytes + data.len() as u64) +} + +fn estimate_bytes(m: &filmcraft_project::MediaClip, dur: Tick, pr: &Preset) -> u64 { + let secs = dur.seconds().max(0.0); + let audio = m.info.audio.as_ref().map_or(0.0, |a| a.sample_rate as f64 * 2.0 * 2.0 * secs); + let Some(v) = &m.info.video else { return audio as u64 }; + let fps = v.frame_rate.num as f64 / v.frame_rate.den.max(1) as f64; + let px = (v.width * v.height) as f64 * (pr.scale * pr.scale) as f64; + let mbps = match pr.format { + filmcraft_export::Format::ProRes => filmcraft_export::prores_profile(pr.prores).nominal_mbps_1080p30() * px / (1920.0 * 1080.0) * fps / 29.97, + _ => (px / 200.0).max(800.0) / 1000.0, + }; + (mbps * 1e6 / 8.0 * secs + audio) as u64 +} + +fn dir_size(d: &Path) -> u64 { + std::fs::read_dir(d) + .map(|rd| rd.filter_map(|e| e.ok()).map(|e| if e.path().is_dir() { dir_size(&e.path()) } else { e.metadata().map_or(0, |m| m.len()) }).sum()) + .unwrap_or(0) +} + +fn copy_dir(from: &Path, to: &Path) -> std::io::Result { + std::fs::create_dir_all(to)?; + let mut n = 0; + for e in std::fs::read_dir(from)? { + let e = e?; + let p = e.path(); + if p.is_dir() { + n += copy_dir(&p, &to.join(e.file_name()))?; + } else { + n += std::fs::copy(&p, to.join(e.file_name()))?; + } + } + Ok(n) +} + +pub fn commands() -> Vec { + vec![CommandSpec { + id: "file.projectManager", + label: "Project Manager…", + menu: &["File"], + shortcut: None, + params: r#"{"destination":str,"mode":"collect|consolidate","sequences":[id]?,"excludeUnused":bool=true,"handles":frames=30,"preset":"prores_lt|prores_hq|h264|…","includeProxies":bool,"includePreviews":bool=false,"projectName":str?,"dryRun":bool=false,"overwrite":bool=false,"wait":bool=false}"#, + enabled: |s| if s.project.sequences().next().is_some() { Ok(()) } else { Err("the project has no sequences".into()) }, + run, + journal: false, + }] +} diff --git a/crates/engine/src/project_manager_tests.rs b/crates/engine/src/project_manager_tests.rs new file mode 100644 index 00000000..82c4a7d4 --- /dev/null +++ b/crates/engine/src/project_manager_tests.rs @@ -0,0 +1,102 @@ +//! Project Manager: collect copies used media (excluding unused clips) and the copy opens and +//! plays; consolidate trims to the used range plus handles and every edit stays in place. + +use filmcraft_media::DemoScene; +use filmcraft_project::{ItemId, MediaRef}; +use serde_json::json; + +use crate::Session; +use crate::media_test_util::{frame_rgba, make_movie, psnr, session_with, tmp_dir}; + +fn project(root: &std::path::Path) -> (Session, Vec, String) { + let media = root.join("Media"); + std::fs::create_dir_all(&media).unwrap(); + let (a, b, unused) = (media.join("a.mov"), media.join("b.mov"), media.join("unused.mov")); + make_movie(&a, DemoScene::Aurora, 160, 90, 48); + make_movie(&b, DemoScene::Dunes, 160, 90, 24); + make_movie(&unused, DemoScene::Plasma, 160, 90, 12); + let (mut s, items, _) = session_with(&[&a, &b]); + s.execute("file.import", json!({"paths": [unused.to_string_lossy()]})).unwrap(); + // use only frames 20..32 of a (V1 + A1) + let seq = s.state.active_sequence.unwrap(); + let mut p = (*s.project).clone(); + let rate = filmcraft_time::FrameRate::FPS_24; + let q = p.sequence_mut(seq).unwrap(); + for t in q.all_tracks_mut() { + let it = t.items.iter_mut().find(|i| i.item == items[0]).unwrap(); + it.source_in = rate.tick_of(20); + it.duration = rate.tick_of(12); + for x in t.items.iter_mut().filter(|i| i.item == items[1]) { + x.start = rate.tick_of(12); + } + } + s.project = std::sync::Arc::new(p); + let path = root.join("p.fcproj").to_string_lossy().into_owned(); + s.execute("file.save", json!({"path": path})).unwrap(); + (s, items, path) +} + +fn opened(path: &str) -> Session { + let mut s = Session::default(); + let r = s.execute("file.open", json!({"path": path})).unwrap(); + assert_eq!(r["missingMedia"], json!(0), "{r}"); + s +} + +#[test] +fn collect_files_copies_used_media_and_opens() { + let root = tmp_dir("pm-collect"); + let (mut s, _, _) = project(&root); + let before: Vec> = (0..36).step_by(5).map(|f| frame_rgba(&mut s, f, 1.0).2).collect(); + let dest = root.join("Collected"); + let dry = s.execute("file.projectManager", json!({"destination": dest.to_string_lossy(), "dryRun": true})).unwrap(); + assert_eq!(dry["files"].as_array().unwrap().len(), 2, "the unused clip is excluded: {dry}"); + assert_eq!(dry["excluded"], json!(1)); + let size = |n: &str| std::fs::metadata(root.join("Media").join(n)).unwrap().len(); + assert_eq!(dry["originalBytes"].as_u64().unwrap(), size("a.mov") + size("b.mov")); + assert!(!dest.exists(), "a dry run writes nothing"); + let r = s.execute("file.projectManager", json!({"destination": dest.to_string_lossy(), "wait": true})).unwrap(); + let jobs = s.execute("jobs.list", json!({})).unwrap(); + assert!(jobs[0]["result"].get("error").is_none(), "{jobs}"); + // the copy opens with its media next to it, and plays the same frames + std::fs::rename(root.join("Media"), root.join("Media-gone")).unwrap(); + let mut t = opened(r["project"].as_str().unwrap()); + for it in t.project.items.values().filter_map(|i| i.as_media()) { + let MediaRef::File { path } = &it.media else { panic!() }; + assert!(path.starts_with(&*dest.to_string_lossy()), "{path}"); + assert!(it.identity.is_some()); + } + let after: Vec> = (0..36).step_by(5).map(|f| frame_rgba(&mut t, f, 1.0).2).collect(); + assert_eq!(before, after); + let _ = std::fs::remove_dir_all(&root); +} + +#[test] +fn consolidate_trims_to_used_ranges_with_handles() { + let root = tmp_dir("pm-consolidate"); + let (mut s, items, _) = project(&root); + let frames = [0i64, 5, 11, 12, 20, 35]; + let before: Vec> = frames.iter().map(|f| frame_rgba(&mut s, *f, 1.0).2).collect(); + let dest = root.join("Consolidated"); + let r = s + .execute( + "file.projectManager", + json!({"destination": dest.to_string_lossy(), "mode": "consolidate", "handles": 4, "preset": "prores_hq", "wait": true}), + ) + .unwrap(); + let trim = &r["files"].as_array().unwrap().iter().find(|f| f["item"] == json!(items[0].0)).unwrap()["trim"]; + let rate = filmcraft_time::FrameRate::FPS_24; + assert_eq!(trim["start"], json!(rate.tick_of(16).0), "used from frame 20, 4 handle frames: {r}"); + assert_eq!(trim["end"], json!(rate.tick_of(36).0), "used through frame 31, 4 handles: {r}"); + assert!(r["resultBytes"].as_u64().unwrap() > 0); + let jobs = s.execute("jobs.list", json!({})).unwrap(); + assert!(jobs[0]["result"].get("error").is_none(), "{jobs}"); + let mut t = opened(r["project"].as_str().unwrap()); + let a = t.project.item(items[0]).unwrap().as_media().unwrap().clone(); + assert_eq!(a.info.duration, rate.tick_of(20), "a.mov now holds only the used range"); + for (k, f) in frames.iter().enumerate() { + let q = psnr(&frame_rgba(&mut t, *f, 1.0).2, &before[k]); + assert!(q > 38.0, "frame {f}: {q:.1} dB"); + } + let _ = std::fs::remove_dir_all(&root); +} diff --git a/crates/ui-egui/src/panels/media_dialogs.rs b/crates/ui-egui/src/panels/media_dialogs.rs index 86647d19..3415a0af 100644 --- a/crates/ui-egui/src/panels/media_dialogs.rs +++ b/crates/ui-egui/src/panels/media_dialogs.rs @@ -1,6 +1,7 @@ -//! Media management dialogs: Link Media (offline media / relink), Make Offline and Create Proxies. -//! Their state lives in `UiState` (serde), so agents can open and fill them as well as click them; -//! the actions run engine commands (`media.*`). +//! Media management dialogs: Link Media (offline media / relink), Make Offline, Create Proxies and +//! Project Manager. Their state lives in `UiState` (serde), so agents can open and fill them with +//! `ui.set` as well as by clicking; the actions run engine commands (`media.*`, +//! `file.projectManager`). //! //! Automation ids: //! - Link Media: `linkMedia.row.`, `linkMedia.match.`, @@ -9,12 +10,15 @@ //! `linkMedia.link`, `linkMedia.locate`, `linkMedia.offline`, `linkMedia.offlineAll`, `linkMedia.cancel`. //! - Make Offline: `makeOffline.keep`, `makeOffline.delete`, `makeOffline.ok`, `makeOffline.cancel`. //! - Create Proxies: `proxies.preset.`, `proxies.destination`, `proxies.browse`, `proxies.ok`, `proxies.cancel`. +//! - Project Manager: `pm.seq.`, `pm.mode.`, `pm.preset.`, `pm.excludeUnused`, +//! `pm.handles`, `pm.includeProxies`, `pm.includePreviews`, `pm.destination`, `pm.browse`, +//! `pm.calculate`, `pm.sizes`, `pm.ok`, `pm.cancel`. use egui::{Color32, RichText}; use serde_json::{Value, json}; use crate::FilmcraftApp; -use crate::state::{LinkMediaDraft, ProxyDraft}; +use crate::state::{LinkMediaDraft, ProjectManagerDraft, ProxyDraft}; type Elems = Vec<(String, egui::Rect, String)>; @@ -22,6 +26,10 @@ fn push(elems: &mut Elems, id: impl Into, r: &egui::Response, label: imp elems.push((id.into(), r.rect, label.into())); } +fn mb(b: u64) -> String { + if b >= 1 << 30 { format!("{:.2} GB", b as f64 / (1u64 << 30) as f64) } else { format!("{:.1} MB", b as f64 / (1u64 << 20) as f64) } +} + /// Draw whichever media dialogs are open; open Link Media when a project needs it. pub fn show(app: &mut FilmcraftApp, ctx: &egui::Context) { if app.session.offline.prompt && app.ui.link_media.is_none() { @@ -36,6 +44,9 @@ pub fn show(app: &mut FilmcraftApp, ctx: &egui::Context) { if app.ui.create_proxies.is_some() { create_proxies(app, ctx); } + if app.ui.project_manager.is_some() { + project_manager(app, ctx); + } } /// Menu / shortcut entry points without params open a dialog. Returns None for other ids. @@ -74,6 +85,18 @@ pub fn route(app: &mut FilmcraftApp, id: &str, params: &Value) -> Option Some(enabled(app, id).map(|_| { + let seqs = app.session.state.active_sequence.map(|s| vec![s.0]).unwrap_or_default(); + let dest = app + .session + .path + .as_deref() + .and_then(|p| std::path::Path::new(p).parent()) + .map(|d| d.join(format!("{} (copy)", app.session.project.name)).to_string_lossy().into_owned()) + .unwrap_or_default(); + app.ui.project_manager = Some(ProjectManagerDraft { sequences: seqs, destination: dest, ..Default::default() }); + json!({"dialog": "projectManager"}) + })), _ => None, } } @@ -418,3 +441,148 @@ fn create_proxies(app: &mut FilmcraftApp, ctx: &egui::Context) { } app.ui.create_proxies = (keep && !ctx.input(|i| i.key_pressed(egui::Key::Escape))).then_some(d); } + +fn pm_params(d: &ProjectManagerDraft, dry: bool) -> Value { + json!({ + "destination": d.destination, + "mode": d.mode, + "sequences": d.sequences, + "excludeUnused": d.exclude_unused, + "handles": d.handles, + "preset": d.preset, + "includeProxies": d.include_proxies, + "includePreviews": d.include_previews, + "dryRun": dry, + }) +} + +fn project_manager(app: &mut FilmcraftApp, ctx: &egui::Context) { + let Some(mut d) = app.ui.project_manager.clone() else { return }; + let before = d.clone(); + let mut elems: Elems = Vec::new(); + let mut keep = true; + let (mut ok, mut calc, mut browse) = (false, false, false); + let seqs: Vec<(u64, String)> = app.session.project.sequences().map(|i| (i.id.0, i.name.clone())).collect(); + egui::Window::new("Project Manager").collapsible(false).resizable(false).anchor(egui::Align2::CENTER_CENTER, [0.0, 0.0]).show(ctx, |ui| { + ui.label(RichText::new("Sequences").strong()); + for (id, name) in &seqs { + let mut on = d.sequences.contains(id); + let r = ui.checkbox(&mut on, name); + push(&mut elems, format!("pm.seq.{id}"), &r, name.clone()); + if r.changed() { + if on { + d.sequences.push(*id); + } else { + d.sequences.retain(|s| s != id); + } + } + } + ui.separator(); + ui.label(RichText::new("Resulting Project").strong()); + for (id, label) in [("collect", "Collect Files and Copy to New Location"), ("consolidate", "Consolidate and Transcode")] { + let r = ui.radio(d.mode == id, label); + push(&mut elems, format!("pm.mode.{id}"), &r, label); + if r.clicked() { + d.mode = id.into(); + } + } + if d.mode == "consolidate" { + ui.indent("pm-presets", |ui| { + for p in filmcraft_engine::proxies::PRESETS.iter().filter(|p| !p.proxy) { + let r = ui.radio(d.preset == p.id, p.label); + push(&mut elems, format!("pm.preset.{}", p.id), &r, p.label); + if r.clicked() { + d.preset = p.id.into(); + } + } + }); + } + ui.separator(); + ui.label(RichText::new("Options").strong()); + let r = ui.checkbox(&mut d.exclude_unused, "Exclude Unused Clips"); + push(&mut elems, "pm.excludeUnused", &r, "Exclude Unused Clips"); + ui.horizontal(|ui| { + ui.add_enabled_ui(d.mode == "consolidate", |ui| { + ui.label("Include Handles:"); + let r = ui.add(egui::DragValue::new(&mut d.handles).range(0..=600).suffix(" frames")); + push(&mut elems, "pm.handles", &r, format!("{} frames", d.handles)); + }); + }); + ui.add_enabled_ui(d.mode == "collect", |ui| { + let r = ui.checkbox(&mut d.include_proxies, "Include Proxies"); + push(&mut elems, "pm.includeProxies", &r, "Include Proxies"); + }); + let r = ui.checkbox(&mut d.include_previews, "Include Preview Files"); + push(&mut elems, "pm.includePreviews", &r, "Include Preview Files"); + ui.separator(); + ui.label(RichText::new("Destination Path").strong()); + ui.horizontal(|ui| { + let r = ui.add(egui::TextEdit::singleline(&mut d.destination).desired_width(360.0)); + push(&mut elems, "pm.destination", &r, "destination"); + let r = ui.button("Browse…"); + push(&mut elems, "pm.browse", &r, "Browse…"); + browse = r.clicked(); + }); + ui.horizontal(|ui| { + let text = match d.estimate { + Some((a, b, n)) => format!("Disk space: original {} · resulting {} ({n} files)", mb(a), mb(b)), + None => "Disk space: —".into(), + }; + let r = ui.label(&text); + push(&mut elems, "pm.sizes", &r, text); + let r = ui.button("Calculate"); + push(&mut elems, "pm.calculate", &r, "Calculate"); + calc = r.clicked(); + }); + if !d.message.is_empty() { + ui.colored_label(Color32::from_rgb(0xe0, 0x8a, 0x6a), &d.message); + } + ui.horizontal(|ui| { + let r = ui.button("Cancel"); + push(&mut elems, "pm.cancel", &r, "Cancel"); + keep &= !r.clicked(); + let r = ui.add_enabled( + !d.destination.is_empty() && !d.sequences.is_empty(), + egui::Button::new(RichText::new("OK").color(Color32::WHITE)).fill(app.tokens.accent), + ); + push(&mut elems, "pm.ok", &r, "OK"); + ok = r.clicked(); + }); + }); + for (id, r, l) in elems { + app.auto.add(&id, r, &l); + } + if browse && let Some(f) = app.hooks.pick_folder.as_mut().and_then(|f| f()) { + d.destination = f; + } + // settings changed: the estimate is stale + if (d.mode != before.mode + || d.sequences != before.sequences + || d.exclude_unused != before.exclude_unused + || d.handles != before.handles + || d.preset != before.preset) + && !calc + { + d.estimate = None; + } + if calc { + match app.session.execute("file.projectManager", pm_params(&d, true)) { + Ok(v) => { + d.estimate = + Some((v["originalBytes"].as_u64().unwrap_or(0), v["resultBytes"].as_u64().unwrap_or(0), v["files"].as_array().map_or(0, Vec::len))); + d.message.clear(); + } + Err(e) => d.message = e.to_string(), + } + } + if ok { + match app.session.execute("file.projectManager", pm_params(&d, false)) { + Ok(v) => { + app.ui.status = format!("Project Manager: writing {}", v["project"].as_str().unwrap_or_default()); + keep = false; + } + Err(e) => d.message = e.to_string(), + } + } + app.ui.project_manager = (keep && !ctx.input(|i| i.key_pressed(egui::Key::Escape))).then_some(d); +} diff --git a/crates/ui-egui/src/state.rs b/crates/ui-egui/src/state.rs index 33bf40b5..459552b4 100644 --- a/crates/ui-egui/src/state.rs +++ b/crates/ui-egui/src/state.rs @@ -298,6 +298,9 @@ pub struct UiState { /// Create Proxies dialog (open when Some). #[serde(default)] pub create_proxies: Option, + /// Project Manager dialog (open when Some). + #[serde(default)] + pub project_manager: Option, /// Make Offline dialog: Some(delete files?) while open. #[serde(default)] pub make_offline: Option, @@ -360,6 +363,41 @@ pub struct ProxyDraft { pub destination: String, } +/// File ▸ Project Manager…. +#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] +#[serde(default)] +pub struct ProjectManagerDraft { + pub sequences: Vec, + /// "collect" | "consolidate" + pub mode: String, + pub preset: String, + pub exclude_unused: bool, + pub handles: u32, + pub include_proxies: bool, + pub include_previews: bool, + pub destination: String, + /// Last dry-run estimate: (original bytes, resulting bytes, files). + pub estimate: Option<(u64, u64, usize)>, + pub message: String, +} + +impl Default for ProjectManagerDraft { + fn default() -> Self { + Self { + sequences: vec![], + mode: "collect".into(), + preset: "prores_lt".into(), + exclude_unused: true, + handles: 30, + include_proxies: true, + include_previews: false, + destination: String::new(), + estimate: None, + message: String::new(), + } + } +} + /// The Audio Gain dialog (Clip ▸ Audio Gain…, G). #[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)] #[serde(default)] @@ -424,6 +462,7 @@ impl Default for UiState { pen_points: vec![], link_media: None, create_proxies: None, + project_manager: None, make_offline: None, } } diff --git a/crates/ui-egui/tests/media_ui.rs b/crates/ui-egui/tests/media_ui.rs index 60e42169..34bdf20d 100644 --- a/crates/ui-egui/tests/media_ui.rs +++ b/crates/ui-egui/tests/media_ui.rs @@ -1,6 +1,6 @@ //! Headless UI tests of media management: the Link Media dialog opening for a project whose media //! moved (search a folder, link, the others follow), Project panel offline / proxy badges, the -//! monitors' Toggle Proxies button and the Create Proxies dialog. +//! monitors' Toggle Proxies button, Create Proxies and Project Manager dialogs. //! //! Set `FILMCRAFT_UI_SNAPSHOT_DIR=` to also render the window offscreen with wgpu and write PNGs //! there (`media-*.png`). @@ -246,5 +246,15 @@ fn offline_badges_proxy_toggle_and_dialogs() { assert_eq!(d.ids("project.item.").into_iter().filter(|i| i.ends_with(".proxy")).count(), 2, "proxy badges"); d.frames(20); d.snapshot("media-proxies-on"); + // Project Manager dialog: calculate the size estimate + d.ok("ui.menu.invoke", json!({"id": "file.projectManager"})); + d.frames(2); + d.app().ui.project_manager.as_mut().unwrap().destination = root.join("Collected").to_string_lossy().into_owned(); + d.click("pm.calculate"); + let est = d.app().ui.project_manager.as_ref().unwrap().estimate; + assert!(est.is_some_and(|(a, b, n)| a > 0 && b == a && n == 2), "{est:?}"); + d.snapshot("media-project-manager"); + d.click("pm.ok"); + assert!(d.app().ui.project_manager.is_none()); let _ = std::fs::remove_dir_all(&root); } diff --git a/docs/control-protocol.md b/docs/control-protocol.md index 22349d22..4dc919fb 100644 --- a/docs/control-protocol.md +++ b/docs/control-protocol.md @@ -49,8 +49,8 @@ Project files, auto-save, crash recovery and preferences commands (`file.recover automation ids are listed in [project-files.md](project-files.md). That file also lists the media management commands and dialog ids: offline media and relinking (`media.findMissing`, `media.relink`, `media.autoRelink`, `media.search`, `media.makeOffline`, `media.status`, `linkMedia.*`), -proxies (`media.createProxies`, `media.attachProxies`, `media.toggleProxies`, `proxies.*`) and ingest -(`project.ingestSettings`). +proxies (`media.createProxies`, `media.attachProxies`, `media.toggleProxies`, `proxies.*`), ingest +(`project.ingestSettings`) and the Project Manager (`file.projectManager`, `pm.*`). **Trim mode** (`crates/engine/src/trim.rs`): `trim.selectEditPoint` / `trim.selectNearest` enter trim mode (the Program monitor becomes the Trim Monitor, ids `trimMonitor.*`); `trim.monitor` returns what diff --git a/docs/project-files.md b/docs/project-files.md index 8230ace7..7a049ed6 100644 --- a/docs/project-files.md +++ b/docs/project-files.md @@ -292,3 +292,28 @@ destination?, preset?}` (Project Settings ▸ Ingest) acts on every `file.import `file.import` reports this under `ingest`. +## Project Manager + +`file.projectManager` (File ▸ Project Manager…) makes a self-contained copy of a project: + +```json +{"destination": "/path/Folder", "mode": "collect|consolidate", "sequences": [id], "excludeUnused": true, + "handles": 30, "preset": "prores_lt", "includeProxies": true, "includePreviews": false, + "projectName": "…", "dryRun": false, "overwrite": false, "wait": false} +``` + +- **collect** copies every media file that the chosen sequences use (following nested sequences and + subclips), plus their proxies and render previews if asked. +- **consolidate** writes only the used range of each movie or audio file, plus `handles` frames on + each side, with a transcode preset (`prores_lt`, `prores_hq`, `h264`). It then re-bases the clips' + media time, so every edit, keyframe and marker stays in place. Stills are copied. +- **excludeUnused** drops media items and sequences that the chosen sequences don't use. +- **dryRun** returns the plan and the disk-space estimate (`originalBytes`, `resultBytes`, `files`) + without writing anything. + +The copy runs as a job. The project file (`/.fcproj`) is written last, with the +new paths and fingerprints, so it opens with no missing media. + +Dialog ids: `pm.seq.`, `pm.mode.`, `pm.preset.`, `pm.excludeUnused`, +`pm.handles`, `pm.includeProxies`, `pm.includePreviews`, `pm.destination`, `pm.browse`, +`pm.calculate`, `pm.sizes`, `pm.ok`, `pm.cancel`.