shifathosn
/
effectcraft
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
effectcraft/crates/segment
History
bflatastic 7579de21c6 M13.35: Roto Brush trained model — swappable segment module, MobileSAM in pure Rust 
- New L1 crate effectcraft-segment: the MaskModel interface (prompts → foreground probability),
  a registry of open-licensed models (licence must be MIT/Apache/BSD/…; URL, size, SHA-256),
  a PyTorch checkpoint reader (zip + pickle subset), dependency-free SHA-256, and MobileSAM
  (TinyViT-5M + SAM prompt encoder / two-way transformer / mask decoder) on rayon and ndarray's
  safe GEMM. Loads the official mobile_sam.pt as published; checked against candle
  (embedding 0.3% RMS, 99.9% equal mask pixels); ~0.7 s/frame encoder at 32 threads, cached
  per frame, ~0.1 s decoder.
- track::roto: segment_with / propagate_with prompt a model with the strokes or the flow-warped
  matte (box, core points, mask); its probability is a strong, unblurred prior for the same
  graph cut (strokes stay hard constraints); a failing model or one disagreeing with the flow
  (IoU < 0.5) is ignored. Prior gets a weight.
- effects::roto: Version 2.0 / 3.0 use the model chosen in Settings (set_model / model_for);
  the model id is part of the chain seed.
- engine: roto_models (roto.models, roto.model.select / install / download / remove), weights in
  the models folder (host::config_dir()/models, shared by the app, CLI and MCP), background
  loading, downloads through the system curl, verified; Settings ▸ Roto Brush page (prefs
  roto.model). host::config_dir moved from the desktop app.
- Tests: segment (kernels vs direct sums, checkpoint reader, SHA-256 vectors, registry), track
  (fake model: prior shapes the cut, failing / disagreeing models ignored; real weights with
  EFFECTCRAFT_MOBILESAM: base IoU 0.989, propagated ≥ 0.980 vs classic 0.973), engine (registry,
  verification, selection; real install + load), MCP.
- Docs: README, ROADMAP, parity (MSK-4 done: 90/2/0), preferences, agents, architecture,
  crates/segment/README.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 		2026-10-05 17:55:02 +07:00
..
src M13.35: Roto Brush trained model — swappable segment module, MobileSAM in pure Rust 2026-10-05 17:55:02 +07:00
Cargo.toml M13.35: Roto Brush trained model — swappable segment module, MobileSAM in pure Rust 2026-10-05 17:55:02 +07:00
README.md M13.35: Roto Brush trained model — swappable segment module, MobileSAM in pure Rust 2026-10-05 17:55:02 +07:00

README.md

effectcraft-segment

Trained segmentation models for Roto Brush, behind one swappable interface.

  • MaskModel: what Roto Brush asks of a model: given a frame and prompts (foreground / background points, a box, a prior mask), the foreground probability of every pixel. Roto Brush (effectcraft-track) only talks to this trait, so models can be added or swapped without touching it. The classical graph-cut segmenter stays built in as the fallback.
  • MODELS: the registry. Every entry must be open source under a licence compatible with EffectCraft's MIT OR Apache-2.0 (ALLOWED_LICENCES), with its homepage, licence URL, the official weights URL, size and SHA-256. load refuses anything else.
  • Weights are never bundled. They are downloaded on demand (or installed from a file) by the engine (roto.model.*, Settings ▸ Roto Brush) and verified against the registry's SHA-256 (sha256, dependency-free).
  • pt: reads PyTorch checkpoints (torch.save zip archives and the pickle subset state dicts use), so official weights load exactly as published.
  • mobilesam: MobileSAM in plain Rust: the TinyViT-5M image encoder and Segment Anything's prompt encoder and two-way-transformer mask decoder, on nn kernels (rayon, and ndarray's safe general_mat_mul, which picks AVX/FMA kernels at run time). No ML framework, no unsafe code.

Models

id Model Licence Weights Size
mobilesam MobileSAM (C. Zhang et al., 2023): Segment Anything with a 5M-parameter TinyViT encoder Apache-2.0 mobile_sam.pt from the official repository 40.7 MB

The implementation follows the published architectures (MobileSAM, TinyViT and Segment Anything papers and their Apache-2.0 reference code); its output was checked against an independent implementation (Hugging Face candle's, MIT/Apache-2.0) on the same weights: embeddings within 0.3% RMS (exact vs. tanh GELU) and 99.9% of mask pixels equal.

Adding a model

  1. Pick a model whose code and weights are open source under an allowed licence.
  2. Add a ModelInfo to MODELS (pin the official file's SHA-256 and size).
  3. Implement MaskModel for it and add its loader to load.
  4. Test it against a reference implementation; Roto Brush, Settings and the roto.model.* commands pick it up from the registry.