diff --git a/docs/fidelity/README.md b/docs/fidelity/README.md index 025997d4..5afafee4 100644 --- a/docs/fidelity/README.md +++ b/docs/fidelity/README.md @@ -3,6 +3,9 @@ The [AE 2026 replacement acceptance gates](ae-2026-acceptance.md) define the broader target across workflows, rendered output, work-project compatibility, performance and platforms. +[Object Matte](object-matte.md) has a comparison plan (clips, prompt, per-frame IoU and the other +metrics) but no measurement yet. + The first G1 comparison measures eight original scalar Rotation cases against a live After Effects 2026 session through AEsync 2.0.4: Easy Ease, overlapping influences, asymmetric ease, overshoot, descending values, equal endpoints, 100%/100% influence and 0.1%/100% influence. Each case has diff --git a/docs/fidelity/object-matte.md b/docs/fidelity/object-matte.md new file mode 100644 index 00000000..da4e3bd3 --- /dev/null +++ b/docs/fidelity/object-matte.md @@ -0,0 +1,73 @@ +# Object Matte: comparison plan (not measured yet) + +**Status: untested.** There is no After Effects 26.2+ comparison of Object Matte yet. Nothing on +this page is a measurement. It lists what to compare once an After Effects reference session +is available, so the first run can be repeated and scored without guesswork. + +EffectCraft side, as of #783: the Object Matte effect (`ec.matte.objectmatte`), the Object Matte +tool in the Composition viewer, `objectMatte.apply` / `.propagate` / `.freeze` / +`.status`, and segmentations kept in the disk cache. The point (and an optional box) is +segmented by Roto Brush's engine: MobileSAM when it is installed (Settings ▸ Roto Brush, with +the box as the model's box prompt), otherwise the classical graph cut. The result is tracked +frame by frame with optical flow, and the model re-prompts each frame from the carried matte. +Record which engine was active, because the two are separate cases. + +## Test clips + +All original, generated or openly licensed; none of them goes into the repository. + +1. **Synthetic disk**: the moving textured disk over a textured background from + `crates/engine/src/tests_roto.rs` (`draw(center(f))`, 320×240, 25 fps, 25 frames, radius + 40 px, about 4 px a frame to the right). Export it as a PNG sequence. Its ground truth is + analytic (`gt(center(f))`). +2. **Synthetic disk with a jump**: the same disk with `jumping(f)` (it jumps at frame 8). This + checks how each tool handles losing the object. +3. **Two objects**: the disk plus a second still, differently coloured square that touches it + on some frames, prompted once with a point only and once with a box around the disk. This + checks the box prompt. +4. **Real footage**: one openly licensed clip (CC0 or CC BY) with a person or object moving + against clutter, at least 1080p and 5 s long. Record its URL, licence and SHA-256 next to the + results, and keep the clip itself out of the repository. + +## Prompt and settings + +- Reference frame 0. The point at the object's centre, in layer pixels (clip 1: `[90, 110]`). + The box, where used, is `[centre − 55, centre + 55]` on each axis. +- Track forward to the last frame (Frames After = clip length − 1, Frames Before = 0). +- Matte refinement at its defaults on both sides: Feather 0, Contrast 0, Shift Edge 0, no + Invert. Output the matte itself (View ▸ Matte in EffectCraft, the alpha channel in After + Effects) as a full-resolution PNG sequence. +- Record the After Effects build, the OS, the GPU, and the exact click position in After + Effects. It is a viewer click, so convert it to layer pixels with the comp's zoom. + +Keep the After Effects mattes in gitignored `plan/aftereffects/ref/object-matte/`. Only an +aggregate results file (like `keyframe-ease.json`) and the comparison script may be committed. + +## Metrics (per frame, then mean / worst) + +- **IoU against After Effects**: binarize both mattes at alpha 0.5, then intersection over + union. This is the headline number. `effectcraft_track::roto::iou` computes it, and + `roto.status` already reports it for Roto Brush through `compareTo`; Object Matte needs the + same for exported mattes. +- **IoU against ground truth** for the synthetic clips, for both applications. It tells + "differs from After Effects" apart from "worse than After Effects". +- **Boundary error**: the mean and 95th-percentile distance (px) between the two mattes' 0.5 + contours, which shows edge disagreement that IoU dilutes on large objects. +- **Soft-edge error**: the mean absolute alpha difference inside a 3 px band around the After + Effects contour. +- **Temporal stability**: the frame-to-frame IoU change of each matte on clip 1, where the + object moves rigidly, so jitter shows. +- **Drift**: the frame where IoU against ground truth first drops below 0.9 (clips 1 and 2). +- **Time**: seconds per frame to track the clip, warm and cold, on the same machine, with the + disk cache on and off. + +Proposed pass bar (a maintainer decision, not yet agreed): mean IoU against After Effects +≥ 0.95 and worst ≥ 0.90 on clips 1 and 3, with MobileSAM. With the classical engine the numbers +are reported but not gated. + +## Reproduce (to write) + +The script does not exist yet. It should: generate the synthetic clips; run EffectCraft +headless (`effectcraft-cli run … objectMatte.apply … objectMatte.propagate {"wait":true}`, then +render the matte); drive After Effects through ExtendScript or AEsync to apply Object Matte with +the same prompt and render its alpha; then compute the metrics above into a JSON results file. diff --git a/docs/gaps.md b/docs/gaps.md index 577f7bd6..e9366c73 100644 --- a/docs/gaps.md +++ b/docs/gaps.md @@ -42,7 +42,7 @@ because 179 issues were closed with regression tests between 5 and 10 October. | Real-user experience | ≈ 60%, uneven by platform | (in G2) | 179 issues closed 5–10 October, nearly all with regression tests ([log below](#real-user-report-log-510-october)); 134 open, many from Linux and Windows. Most checking still happens on macOS | | Performance | unknown against After Effects, ≈ 35% | 50–100 | Internal numbers only (e.g. Advanced 3D 290 ms/frame at 1080p on the GPU on an M4 Pro). Users report stutter on cached frames (#595, #304), slow scrubbing (#329), late audio (#554). No 4K / hundreds-of-layers project tested | | Media formats | ≈ 75% | (in file formats) | H.264, ProRes, HEVC, AV1, VP9, image sequences and audio exist. The HEVC / AV1 encoders have no B-frames, multi-reference or SAO / CDEF, so files are larger than from mature encoders. Camera formats (BRAW, R3D, ProRes RAW, variable-frame-rate phone video) are unverified | -| AI-assisted tools | ≈ 35% | 40–80 | Roto Brush 2.0 / 3.0 with MobileSAM (M13.35) scores IoU 0.989 on the base frame of our synthetic moving-disc test and ≥ 0.980 over 20 propagated frames (classic: 0.973). Face tracking with MediaPipe Face Landmarker (M13.36) matches Google's own pipeline to 0.85 px on a test portrait. Neither is compared with After Effects; Object Matte (26.2) is missing | +| AI-assisted tools | ≈ 35% | 40–80 | Roto Brush 2.0 / 3.0 with MobileSAM (M13.35) scores IoU 0.989 on the base frame of our synthetic moving-disc test and ≥ 0.980 over 20 propagated frames (classic: 0.973). Face tracking with MediaPipe Face Landmarker (M13.36) matches Google's own pipeline to 0.85 px on a test portrait. Neither is compared with After Effects; Object Matte (26.2) has a first slice, not compared either | | Maturity | early | — | First commit 1 October 2026. ≈ 329,000 lines of almost entirely agent-written Rust, 2,704 tests, 313 GitHub issues. After Effects has around 30 years of edge cases behind it | ## Every known gap, ranked @@ -70,7 +70,7 @@ are grouped by area; the issue tracker has the rest. | 11 | **Text rendering and layout** | #453 CJK text in comps, #316 paragraph alignment inverted, #483 squeezed at 4:3, #417 soft at Fit, #583 font search, #550 toolbar font options | Text is in almost every motion graphic | 10–20 h | G2 · [ui-parity.md](ui-parity.md) | | 12 | **3D interaction and depth** | No 3D transform gizmos (#392, #347); Z position (#489); camera aperture (#260); reflections (#261); bevel options (#315); 3D issue reopened (#307) | Advanced 3D is After Effects' growth area since 2023 | 20–35 h | G2 · [ui-parity.md](ui-parity.md) | | 13 | **3D model formats and materials** | After Effects 26 imports FBX, USD/USDZ, STL besides glTF and OBJ (`usd_plugins`), and Substance 3D materials with mesh displacement (26.0–26.2); we read glTF and OBJ | Product shots and 3D scenes come as FBX/USDZ | 25–45 h | — · [file-format-parity.md](file-format-parity.md) | -| 14 | **Object Matte** (26.2, disk cache in 26.5) | First slice (#783): the Object Matte effect (Effect ▸ Matte) and `objectMatte.apply` turn a point (and optional box) on a reference frame into a matte and follow the object up to 600 frames each way, or segment every frame on its own; Roto Brush's engine, cache and model (MobileSAM from Settings ▸ Roto Brush, classical graph cut without it). Missing: the click / hover tool in the viewer, a background propagation job, the disk cache, comparison with After Effects | The new one-click roto | 10–20 h | G7 · [target-app-parity.md](target-app-parity.md) | +| 14 | **Object Matte** (26.2, disk cache in 26.5) | First slice (#783): the Object Matte effect (Effect ▸ Matte) and `objectMatte.apply` turn a point (and optional box) on a reference frame into a matte and follow the object up to 600 frames each way, or segment every frame on its own; Roto Brush's engine, cache and model (MobileSAM from Settings ▸ Roto Brush, classical graph cut without it). Since then: the Object Matte tool in the viewer (click / drag a box, hover crosshair), the box as the model's box prompt, background tracking with progress and cancel, Freeze, and segmentations kept in the disk cache (26.5). Missing: a comparison with After Effects ([plan](fidelity/object-matte.md)) | The new one-click roto | 10–20 h | G7 · [target-app-parity.md](target-app-parity.md) | | 15 | **Hardware video decode and encode** | No VideoToolbox, NVENC, Quick Sync or VA-API; #498 asks for it; FilmCraft has VideoToolbox / NVENC / VA-API paths to borrow | 4K playback and export speed | 25–45 h | G5/G6 · [hardware-parity.md](hardware-parity.md) | | 16 | **Localization depth** | Menus in 6 languages; panels in 4; effect names and ~1,650 effect parameter names untranslated; French, German, Korean, Italian, Russian (After Effects ships them) absent; #599 Hangul boxes on macOS | After Effects is used heavily in Japan, China, Korea, Brazil, Europe | 60–110 h + native review | G9 · [localization-parity.md](localization-parity.md) | | 17 | **Media Browser lists files import refuses** | `media_browser.rs` lists TGA, DPX, HDR, AVI, MXF; `media::is_importable` has none of them; #431 any image imports as a sequence; #324 MP3 goes silent | Confusing failures on common VFX formats (DPX, TGA) | 15–30 h | G6 · [file-format-parity.md](file-format-parity.md) | diff --git a/docs/target-app-parity.md b/docs/target-app-parity.md index f0ed90b6..c8ef95a0 100644 --- a/docs/target-app-parity.md +++ b/docs/target-app-parity.md @@ -122,7 +122,7 @@ compositing and effects dominate; audio is light). Hours are to full parity for | Text and typography | 7% | 94% | 50% | 20–35 | CJK text in comps renders wrongly (#453); paragraph alignment inverted (#316); squeezed text at 4:3 (#483); font search (#583); soft text at Fit (#417); variable-font filter (26.3) | | Effects and animation presets | 14% | 85% | 40% | 70–130 | All 306 effects exist, but rendered output is unmeasured against After Effects; 8 built-in presets against After Effects' 621; open effect bugs (#580 crash, #514, #526, #463, #369, #473); effect colour labels (26.5); see [effects-parity.md](effects-parity.md) | | 3D (Classic and Advanced) | 6% | 82% | 35% | 35–65 | No FBX/USD/STL models, no Substance materials, mesh displacement unverified; no 3D transform gizmos (#392, #347); Z position (#489); reflections (#261); camera aperture (#260); bevel options (#315) | -| Tracking and AI-assisted tools | 5% | 80% | 35% | 35–70 | Object Matte (26.2) is a first slice: an effect and command from a point / box prompt, no viewer tool or disk cache; Roto Brush (MobileSAM) and face tracking (MediaPipe) unmeasured against After Effects; camera-tracker view (#505) | +| Tracking and AI-assisted tools | 5% | 80% | 35% | 35–70 | Object Matte (26.2) has the effect, the viewer tool (click / box), background tracking, Freeze and the 26.5 disk cache, not yet compared with After Effects; Roto Brush (MobileSAM) and face tracking (MediaPipe) unmeasured against After Effects; camera-tracker view (#505) | | Preview and cache | 7% | 92% | 40% | 25–50 | Playback stutters on cached frames (#595, #304); slow scrubbing (#329); late audio (#554, #504, #335); no benchmark against After Effects | | Render queue and export | 7% | 85% | 55% | 25–45 | No hardware encoding; ProRes 4444 colour tag (#340); rendering over an existing file (#325); default macOS output location (#446); uncompressed single-file output (#372); HEVC/AV1 without B-frames | | Audio | 2% | 85% | 45% | 6–12 | Audio sync and crackle (#335); audio to keyframes depth | @@ -144,7 +144,7 @@ Feature depth (≈ 50%) feeds the overall *ready* number with the other dimensio | Stability | 45% | 40–80 | Open crash reports #580, #591, #501, #426, #341, #306, #234, #511; no CI runs the tests on pull requests | [gaps.md](gaps.md#g3-stability-and-a-green-main) | | Platforms | 65% | 30–60 | Releases for macOS, Windows x64/x86/ARM64, Linux, FreeBSD and the web (more than After Effects), but Linux and Windows users hit basic bugs | [gaps.md](gaps.md#g2-real-user-reliability-on-every-platform) | | Ecosystem and plug-ins | 20% | 40–80 | After Effects scripts and expressions mostly carry over; no After Effects SDK plug-ins, `.ffx` presets or `.mogrt`; our WebAssembly plug-in API has no third-party plug-ins yet | [plugins.md](plugins.md) | -| AI features | 35% | 40–80 (overlaps Tracking) | Roto Brush (MobileSAM) and face tracking (MediaPipe) as optional downloads; Object Matte (point / box prompt, same model) without its viewer tool or disk cache; the After Effects bundle ships Whisper, BERT and Marian tokenizers and a shot-cut model we have no equivalent of | [gaps.md](gaps.md) | +| AI features | 35% | 40–80 (overlaps Tracking) | Roto Brush (MobileSAM) and face tracking (MediaPipe) as optional downloads; Object Matte (point / box prompt, same model; viewer tool, background tracking, Freeze, disk cache) unmeasured against After Effects; the After Effects bundle ships Whisper, BERT and Marian tokenizers and a shot-cut model we have no equivalent of | [gaps.md](gaps.md) | | Agent control (beyond After Effects) | ahead | — | Every command is drivable over MCP, the control channel and the CLI | [agents.md](agents.md) | **Totals.** Summing the rows double-counts: AI overlaps Tracking, performance overlaps Preview,