From a8de199acac3c079be916bbed2ba3ad4f64797f6 Mon Sep 17 00:00:00 2001 From: vangel76 Date: Sun, 11 Oct 2026 14:07:22 +0200 Subject: [PATCH] #783: Object Matte fidelity plan; gaps and parity notes updated Problem: Object Matte had no After Effects comparison, and none can be made on this Linux machine (no After Effects). The gap notes still listed the viewer tool, background job and disk cache as missing. Fix: docs/fidelity/object-matte.md states plainly that nothing is measured yet and pins down the comparison for when an After Effects 26.2+ reference is available: four original or openly licensed clips (the synthetic moving disk from the engine tests, the disk with a jump, two objects with a box prompt, one CC0/CC BY real clip), the prompt and settings, where reference mattes stay (gitignored), and the metrics: per-frame IoU against After Effects (headline), IoU against ground truth, boundary distance, soft-edge error, temporal stability, drift and time per frame, plus a proposed pass bar left to the maintainers. Linked from docs/fidelity/README.md. docs/gaps.md (#14) and docs/target-app-parity.md now list what landed and leave the comparison as the one remaining item. Co-Authored-By: Claude Opus 5.5 --- docs/fidelity/README.md | 3 ++ docs/fidelity/object-matte.md | 73 +++++++++++++++++++++++++++++++++++ docs/gaps.md | 4 +- docs/target-app-parity.md | 4 +- 4 files changed, 80 insertions(+), 4 deletions(-) create mode 100644 docs/fidelity/object-matte.md diff --git a/docs/fidelity/README.md b/docs/fidelity/README.md index 025997d4..5afafee4 100644 --- a/docs/fidelity/README.md +++ b/docs/fidelity/README.md @@ -3,6 +3,9 @@ The [AE 2026 replacement acceptance gates](ae-2026-acceptance.md) define the broader target across workflows, rendered output, work-project compatibility, performance and platforms. +[Object Matte](object-matte.md) has a comparison plan (clips, prompt, per-frame IoU and the other +metrics) but no measurement yet. + The first G1 comparison measures eight original scalar Rotation cases against a live After Effects 2026 session through AEsync 2.0.4: Easy Ease, overlapping influences, asymmetric ease, overshoot, descending values, equal endpoints, 100%/100% influence and 0.1%/100% influence. Each case has diff --git a/docs/fidelity/object-matte.md b/docs/fidelity/object-matte.md new file mode 100644 index 00000000..da4e3bd3 --- /dev/null +++ b/docs/fidelity/object-matte.md @@ -0,0 +1,73 @@ +# Object Matte: comparison plan (not measured yet) + +**Status: untested.** There is no After Effects 26.2+ comparison of Object Matte yet. Nothing on +this page is a measurement. It lists what to compare once an After Effects reference session +is available, so the first run can be repeated and scored without guesswork. + +EffectCraft side, as of #783: the Object Matte effect (`ec.matte.objectmatte`), the Object Matte +tool in the Composition viewer, `objectMatte.apply` / `.propagate` / `.freeze` / +`.status`, and segmentations kept in the disk cache. The point (and an optional box) is +segmented by Roto Brush's engine: MobileSAM when it is installed (Settings ▸ Roto Brush, with +the box as the model's box prompt), otherwise the classical graph cut. The result is tracked +frame by frame with optical flow, and the model re-prompts each frame from the carried matte. +Record which engine was active, because the two are separate cases. + +## Test clips + +All original, generated or openly licensed; none of them goes into the repository. + +1. **Synthetic disk**: the moving textured disk over a textured background from + `crates/engine/src/tests_roto.rs` (`draw(center(f))`, 320×240, 25 fps, 25 frames, radius + 40 px, about 4 px a frame to the right). Export it as a PNG sequence. Its ground truth is + analytic (`gt(center(f))`). +2. **Synthetic disk with a jump**: the same disk with `jumping(f)` (it jumps at frame 8). This + checks how each tool handles losing the object. +3. **Two objects**: the disk plus a second still, differently coloured square that touches it + on some frames, prompted once with a point only and once with a box around the disk. This + checks the box prompt. +4. **Real footage**: one openly licensed clip (CC0 or CC BY) with a person or object moving + against clutter, at least 1080p and 5 s long. Record its URL, licence and SHA-256 next to the + results, and keep the clip itself out of the repository. + +## Prompt and settings + +- Reference frame 0. The point at the object's centre, in layer pixels (clip 1: `[90, 110]`). + The box, where used, is `[centre − 55, centre + 55]` on each axis. +- Track forward to the last frame (Frames After = clip length − 1, Frames Before = 0). +- Matte refinement at its defaults on both sides: Feather 0, Contrast 0, Shift Edge 0, no + Invert. Output the matte itself (View ▸ Matte in EffectCraft, the alpha channel in After + Effects) as a full-resolution PNG sequence. +- Record the After Effects build, the OS, the GPU, and the exact click position in After + Effects. It is a viewer click, so convert it to layer pixels with the comp's zoom. + +Keep the After Effects mattes in gitignored `plan/aftereffects/ref/object-matte/`. Only an +aggregate results file (like `keyframe-ease.json`) and the comparison script may be committed. + +## Metrics (per frame, then mean / worst) + +- **IoU against After Effects**: binarize both mattes at alpha 0.5, then intersection over + union. This is the headline number. `effectcraft_track::roto::iou` computes it, and + `roto.status` already reports it for Roto Brush through `compareTo`; Object Matte needs the + same for exported mattes. +- **IoU against ground truth** for the synthetic clips, for both applications. It tells + "differs from After Effects" apart from "worse than After Effects". +- **Boundary error**: the mean and 95th-percentile distance (px) between the two mattes' 0.5 + contours, which shows edge disagreement that IoU dilutes on large objects. +- **Soft-edge error**: the mean absolute alpha difference inside a 3 px band around the After + Effects contour. +- **Temporal stability**: the frame-to-frame IoU change of each matte on clip 1, where the + object moves rigidly, so jitter shows. +- **Drift**: the frame where IoU against ground truth first drops below 0.9 (clips 1 and 2). +- **Time**: seconds per frame to track the clip, warm and cold, on the same machine, with the + disk cache on and off. + +Proposed pass bar (a maintainer decision, not yet agreed): mean IoU against After Effects +≥ 0.95 and worst ≥ 0.90 on clips 1 and 3, with MobileSAM. With the classical engine the numbers +are reported but not gated. + +## Reproduce (to write) + +The script does not exist yet. It should: generate the synthetic clips; run EffectCraft +headless (`effectcraft-cli run … objectMatte.apply … objectMatte.propagate {"wait":true}`, then +render the matte); drive After Effects through ExtendScript or AEsync to apply Object Matte with +the same prompt and render its alpha; then compute the metrics above into a JSON results file. diff --git a/docs/gaps.md b/docs/gaps.md index 577f7bd6..e9366c73 100644 --- a/docs/gaps.md +++ b/docs/gaps.md @@ -42,7 +42,7 @@ because 179 issues were closed with regression tests between 5 and 10 October. | Real-user experience | ≈ 60%, uneven by platform | (in G2) | 179 issues closed 5–10 October, nearly all with regression tests ([log below](#real-user-report-log-510-october)); 134 open, many from Linux and Windows. Most checking still happens on macOS | | Performance | unknown against After Effects, ≈ 35% | 50–100 | Internal numbers only (e.g. Advanced 3D 290 ms/frame at 1080p on the GPU on an M4 Pro). Users report stutter on cached frames (#595, #304), slow scrubbing (#329), late audio (#554). No 4K / hundreds-of-layers project tested | | Media formats | ≈ 75% | (in file formats) | H.264, ProRes, HEVC, AV1, VP9, image sequences and audio exist. The HEVC / AV1 encoders have no B-frames, multi-reference or SAO / CDEF, so files are larger than from mature encoders. Camera formats (BRAW, R3D, ProRes RAW, variable-frame-rate phone video) are unverified | -| AI-assisted tools | ≈ 35% | 40–80 | Roto Brush 2.0 / 3.0 with MobileSAM (M13.35) scores IoU 0.989 on the base frame of our synthetic moving-disc test and ≥ 0.980 over 20 propagated frames (classic: 0.973). Face tracking with MediaPipe Face Landmarker (M13.36) matches Google's own pipeline to 0.85 px on a test portrait. Neither is compared with After Effects; Object Matte (26.2) is missing | +| AI-assisted tools | ≈ 35% | 40–80 | Roto Brush 2.0 / 3.0 with MobileSAM (M13.35) scores IoU 0.989 on the base frame of our synthetic moving-disc test and ≥ 0.980 over 20 propagated frames (classic: 0.973). Face tracking with MediaPipe Face Landmarker (M13.36) matches Google's own pipeline to 0.85 px on a test portrait. Neither is compared with After Effects; Object Matte (26.2) has a first slice, not compared either | | Maturity | early | — | First commit 1 October 2026. ≈ 329,000 lines of almost entirely agent-written Rust, 2,704 tests, 313 GitHub issues. After Effects has around 30 years of edge cases behind it | ## Every known gap, ranked @@ -70,7 +70,7 @@ are grouped by area; the issue tracker has the rest. | 11 | **Text rendering and layout** | #453 CJK text in comps, #316 paragraph alignment inverted, #483 squeezed at 4:3, #417 soft at Fit, #583 font search, #550 toolbar font options | Text is in almost every motion graphic | 10–20 h | G2 · [ui-parity.md](ui-parity.md) | | 12 | **3D interaction and depth** | No 3D transform gizmos (#392, #347); Z position (#489); camera aperture (#260); reflections (#261); bevel options (#315); 3D issue reopened (#307) | Advanced 3D is After Effects' growth area since 2023 | 20–35 h | G2 · [ui-parity.md](ui-parity.md) | | 13 | **3D model formats and materials** | After Effects 26 imports FBX, USD/USDZ, STL besides glTF and OBJ (`usd_plugins`), and Substance 3D materials with mesh displacement (26.0–26.2); we read glTF and OBJ | Product shots and 3D scenes come as FBX/USDZ | 25–45 h | — · [file-format-parity.md](file-format-parity.md) | -| 14 | **Object Matte** (26.2, disk cache in 26.5) | First slice (#783): the Object Matte effect (Effect ▸ Matte) and `objectMatte.apply` turn a point (and optional box) on a reference frame into a matte and follow the object up to 600 frames each way, or segment every frame on its own; Roto Brush's engine, cache and model (MobileSAM from Settings ▸ Roto Brush, classical graph cut without it). Missing: the click / hover tool in the viewer, a background propagation job, the disk cache, comparison with After Effects | The new one-click roto | 10–20 h | G7 · [target-app-parity.md](target-app-parity.md) | +| 14 | **Object Matte** (26.2, disk cache in 26.5) | First slice (#783): the Object Matte effect (Effect ▸ Matte) and `objectMatte.apply` turn a point (and optional box) on a reference frame into a matte and follow the object up to 600 frames each way, or segment every frame on its own; Roto Brush's engine, cache and model (MobileSAM from Settings ▸ Roto Brush, classical graph cut without it). Since then: the Object Matte tool in the viewer (click / drag a box, hover crosshair), the box as the model's box prompt, background tracking with progress and cancel, Freeze, and segmentations kept in the disk cache (26.5). Missing: a comparison with After Effects ([plan](fidelity/object-matte.md)) | The new one-click roto | 10–20 h | G7 · [target-app-parity.md](target-app-parity.md) | | 15 | **Hardware video decode and encode** | No VideoToolbox, NVENC, Quick Sync or VA-API; #498 asks for it; FilmCraft has VideoToolbox / NVENC / VA-API paths to borrow | 4K playback and export speed | 25–45 h | G5/G6 · [hardware-parity.md](hardware-parity.md) | | 16 | **Localization depth** | Menus in 6 languages; panels in 4; effect names and ~1,650 effect parameter names untranslated; French, German, Korean, Italian, Russian (After Effects ships them) absent; #599 Hangul boxes on macOS | After Effects is used heavily in Japan, China, Korea, Brazil, Europe | 60–110 h + native review | G9 · [localization-parity.md](localization-parity.md) | | 17 | **Media Browser lists files import refuses** | `media_browser.rs` lists TGA, DPX, HDR, AVI, MXF; `media::is_importable` has none of them; #431 any image imports as a sequence; #324 MP3 goes silent | Confusing failures on common VFX formats (DPX, TGA) | 15–30 h | G6 · [file-format-parity.md](file-format-parity.md) | diff --git a/docs/target-app-parity.md b/docs/target-app-parity.md index f0ed90b6..c8ef95a0 100644 --- a/docs/target-app-parity.md +++ b/docs/target-app-parity.md @@ -122,7 +122,7 @@ compositing and effects dominate; audio is light). Hours are to full parity for | Text and typography | 7% | 94% | 50% | 20–35 | CJK text in comps renders wrongly (#453); paragraph alignment inverted (#316); squeezed text at 4:3 (#483); font search (#583); soft text at Fit (#417); variable-font filter (26.3) | | Effects and animation presets | 14% | 85% | 40% | 70–130 | All 306 effects exist, but rendered output is unmeasured against After Effects; 8 built-in presets against After Effects' 621; open effect bugs (#580 crash, #514, #526, #463, #369, #473); effect colour labels (26.5); see [effects-parity.md](effects-parity.md) | | 3D (Classic and Advanced) | 6% | 82% | 35% | 35–65 | No FBX/USD/STL models, no Substance materials, mesh displacement unverified; no 3D transform gizmos (#392, #347); Z position (#489); reflections (#261); camera aperture (#260); bevel options (#315) | -| Tracking and AI-assisted tools | 5% | 80% | 35% | 35–70 | Object Matte (26.2) is a first slice: an effect and command from a point / box prompt, no viewer tool or disk cache; Roto Brush (MobileSAM) and face tracking (MediaPipe) unmeasured against After Effects; camera-tracker view (#505) | +| Tracking and AI-assisted tools | 5% | 80% | 35% | 35–70 | Object Matte (26.2) has the effect, the viewer tool (click / box), background tracking, Freeze and the 26.5 disk cache, not yet compared with After Effects; Roto Brush (MobileSAM) and face tracking (MediaPipe) unmeasured against After Effects; camera-tracker view (#505) | | Preview and cache | 7% | 92% | 40% | 25–50 | Playback stutters on cached frames (#595, #304); slow scrubbing (#329); late audio (#554, #504, #335); no benchmark against After Effects | | Render queue and export | 7% | 85% | 55% | 25–45 | No hardware encoding; ProRes 4444 colour tag (#340); rendering over an existing file (#325); default macOS output location (#446); uncompressed single-file output (#372); HEVC/AV1 without B-frames | | Audio | 2% | 85% | 45% | 6–12 | Audio sync and crackle (#335); audio to keyframes depth | @@ -144,7 +144,7 @@ Feature depth (≈ 50%) feeds the overall *ready* number with the other dimensio | Stability | 45% | 40–80 | Open crash reports #580, #591, #501, #426, #341, #306, #234, #511; no CI runs the tests on pull requests | [gaps.md](gaps.md#g3-stability-and-a-green-main) | | Platforms | 65% | 30–60 | Releases for macOS, Windows x64/x86/ARM64, Linux, FreeBSD and the web (more than After Effects), but Linux and Windows users hit basic bugs | [gaps.md](gaps.md#g2-real-user-reliability-on-every-platform) | | Ecosystem and plug-ins | 20% | 40–80 | After Effects scripts and expressions mostly carry over; no After Effects SDK plug-ins, `.ffx` presets or `.mogrt`; our WebAssembly plug-in API has no third-party plug-ins yet | [plugins.md](plugins.md) | -| AI features | 35% | 40–80 (overlaps Tracking) | Roto Brush (MobileSAM) and face tracking (MediaPipe) as optional downloads; Object Matte (point / box prompt, same model) without its viewer tool or disk cache; the After Effects bundle ships Whisper, BERT and Marian tokenizers and a shot-cut model we have no equivalent of | [gaps.md](gaps.md) | +| AI features | 35% | 40–80 (overlaps Tracking) | Roto Brush (MobileSAM) and face tracking (MediaPipe) as optional downloads; Object Matte (point / box prompt, same model; viewer tool, background tracking, Freeze, disk cache) unmeasured against After Effects; the After Effects bundle ships Whisper, BERT and Marian tokenizers and a shot-cut model we have no equivalent of | [gaps.md](gaps.md) | | Agent control (beyond After Effects) | ahead | — | Every command is drivable over MCP, the control channel and the CLI | [agents.md](agents.md) | **Totals.** Summing the rows double-counts: AI overlaps Tracking, performance overlaps Preview,