74 lines
4.5 KiB
Markdown
74 lines
4.5 KiB
Markdown
# Object Matte: comparison plan (not measured yet)
|
||
|
||
**Status: untested.** There is no After Effects 26.2+ comparison of Object Matte yet. Nothing on
|
||
this page is a measurement. It lists what to compare once an After Effects reference session
|
||
is available, so the first run can be repeated and scored without guesswork.
|
||
|
||
EffectCraft side, as of #783: the Object Matte effect (`ec.matte.objectmatte`), the Object Matte
|
||
tool in the Composition viewer, `objectMatte.apply` / `.propagate` / `.freeze` /
|
||
`.status`, and segmentations kept in the disk cache. The point (and an optional box) is
|
||
segmented by Roto Brush's engine: MobileSAM when it is installed (Settings ▸ Roto Brush, with
|
||
the box as the model's box prompt), otherwise the classical graph cut. The result is tracked
|
||
frame by frame with optical flow, and the model re-prompts each frame from the carried matte.
|
||
Record which engine was active, because the two are separate cases.
|
||
|
||
## Test clips
|
||
|
||
All original, generated or openly licensed; none of them goes into the repository.
|
||
|
||
1. **Synthetic disk**: the moving textured disk over a textured background from
|
||
`crates/engine/src/tests_roto.rs` (`draw(center(f))`, 320×240, 25 fps, 25 frames, radius
|
||
40 px, about 4 px a frame to the right). Export it as a PNG sequence. Its ground truth is
|
||
analytic (`gt(center(f))`).
|
||
2. **Synthetic disk with a jump**: the same disk with `jumping(f)` (it jumps at frame 8). This
|
||
checks how each tool handles losing the object.
|
||
3. **Two objects**: the disk plus a second still, differently coloured square that touches it
|
||
on some frames, prompted once with a point only and once with a box around the disk. This
|
||
checks the box prompt.
|
||
4. **Real footage**: one openly licensed clip (CC0 or CC BY) with a person or object moving
|
||
against clutter, at least 1080p and 5 s long. Record its URL, licence and SHA-256 next to the
|
||
results, and keep the clip itself out of the repository.
|
||
|
||
## Prompt and settings
|
||
|
||
- Reference frame 0. The point at the object's centre, in layer pixels (clip 1: `[90, 110]`).
|
||
The box, where used, is `[centre − 55, centre + 55]` on each axis.
|
||
- Track forward to the last frame (Frames After = clip length − 1, Frames Before = 0).
|
||
- Matte refinement at its defaults on both sides: Feather 0, Contrast 0, Shift Edge 0, no
|
||
Invert. Output the matte itself (View ▸ Matte in EffectCraft, the alpha channel in After
|
||
Effects) as a full-resolution PNG sequence.
|
||
- Record the After Effects build, the OS, the GPU, and the exact click position in After
|
||
Effects. It is a viewer click, so convert it to layer pixels with the comp's zoom.
|
||
|
||
Keep the After Effects mattes in gitignored `plan/aftereffects/ref/object-matte/`. Only an
|
||
aggregate results file (like `keyframe-ease.json`) and the comparison script may be committed.
|
||
|
||
## Metrics (per frame, then mean / worst)
|
||
|
||
- **IoU against After Effects**: binarize both mattes at alpha 0.5, then intersection over
|
||
union. This is the headline number. `effectcraft_track::roto::iou` computes it, and
|
||
`roto.status` already reports it for Roto Brush through `compareTo`; Object Matte needs the
|
||
same for exported mattes.
|
||
- **IoU against ground truth** for the synthetic clips, for both applications. It tells
|
||
"differs from After Effects" apart from "worse than After Effects".
|
||
- **Boundary error**: the mean and 95th-percentile distance (px) between the two mattes' 0.5
|
||
contours, which shows edge disagreement that IoU dilutes on large objects.
|
||
- **Soft-edge error**: the mean absolute alpha difference inside a 3 px band around the After
|
||
Effects contour.
|
||
- **Temporal stability**: the frame-to-frame IoU change of each matte on clip 1, where the
|
||
object moves rigidly, so jitter shows.
|
||
- **Drift**: the frame where IoU against ground truth first drops below 0.9 (clips 1 and 2).
|
||
- **Time**: seconds per frame to track the clip, warm and cold, on the same machine, with the
|
||
disk cache on and off.
|
||
|
||
Proposed pass bar (a maintainer decision, not yet agreed): mean IoU against After Effects
|
||
≥ 0.95 and worst ≥ 0.90 on clips 1 and 3, with MobileSAM. With the classical engine the numbers
|
||
are reported but not gated.
|
||
|
||
## Reproduce (to write)
|
||
|
||
The script does not exist yet. It should: generate the synthetic clips; run EffectCraft
|
||
headless (`effectcraft-cli run … objectMatte.apply … objectMatte.propagate {"wait":true}`, then
|
||
render the matte); drive After Effects through ExtendScript or AEsync to apply Object Matte with
|
||
the same prompt and render its alpha; then compute the metrics above into a JSON results file.
|