276 lines
19 KiB
Markdown
276 lines
19 KiB
Markdown
# EffectCraft on the web
|
||
|
||
`apps/effectcraft-web` runs the same engine and egui UI as the desktop app in the browser: the
|
||
workspace compiled to `wasm32-unknown-unknown`, started by eframe's web runner on WebGPU (WebGL2
|
||
fallback). It is a static site: one `.wasm`, its `wasm-bindgen` JavaScript glue, `index.html`, a
|
||
few small scripts (worker, audio worklet, service worker), a web manifest and icons. Nothing is
|
||
uploaded anywhere: projects, media, settings and renders stay in the browser (its private storage)
|
||
and on the user's machine.
|
||
|
||
## Build and run
|
||
|
||
| Tool | Version | Install |
|
||
|---|---|---|
|
||
| Rust target | `wasm32-unknown-unknown` | `rustup target add wasm32-unknown-unknown` |
|
||
| wasm-bindgen CLI | exactly the `wasm-bindgen` crate version (0.2.129) | `cargo install wasm-bindgen-cli --version 0.2.129 --locked` |
|
||
| wasm-opt (optional) | any recent binaryen | `brew install binaryen`; used by `cargo xtask web` when found |
|
||
|
||
```sh
|
||
cargo xtask web # release build → <target>/web/dist
|
||
cargo xtask web --dev # unoptimised build (faster to compile, slow to run)
|
||
cargo xtask web --serve 8765 # build, then serve dist on http://127.0.0.1:8765/
|
||
```
|
||
|
||
`<target>` is `$CARGO_TARGET_DIR` or `target`. `dist` holds `index.html`, `effectcraft_web.js` +
|
||
`effectcraft_web_bg.wasm` (+ `snippets/`, the browser glue `js/host.js`), `worker.js`,
|
||
`audio-worklet.js`, `sw.js` (its precache list and version filled in by the build),
|
||
`manifest.webmanifest`, `favicon.svg` and `icon-256.png` / `icon-512.png` (copied from
|
||
`assets/app-icon`). `--serve` is a tiny localhost-only static server that sends
|
||
`Cross-Origin-Opener-Policy` / `Cross-Origin-Embedder-Policy`, so the page is cross-origin
|
||
isolated like a production deployment should be. Any static server works
|
||
(`python3 -m http.server -d target/web/dist 8765`); it must serve `.wasm` as `application/wasm`.
|
||
Service workers need a secure context: `https://`, or `http://localhost` / `127.0.0.1`.
|
||
|
||
The release `.wasm` is about 38 MB before `wasm-opt` (code: the effects, codecs, the expression
|
||
engine and the UI; data, mostly the bundled fonts; function names for readable panics). Served
|
||
compressed it is a fraction of that, and the service worker caches it after the first visit.
|
||
|
||
On other targets the web crate holds only its portable storage model (`store.rs`, unit-tested by
|
||
`cargo test --workspace`); `cargo xtask wasm` (part of `cargo xtask ci`) checks the whole crate,
|
||
with every L0–L4 crate and `effectcraft-ui-egui`, for `wasm32-unknown-unknown`.
|
||
|
||
URL flags: `?empty` (a blank project; the session is not restored or recorded), `?demo` (the demo
|
||
project instead of the last session), `?home` (show the start screen), `?storage=indexeddb` /
|
||
`?storage=memory` (force a storage backend), `?noworkers` (render, analyse and draw viewer
|
||
frames on the page's thread), `?frameworkers=N` (frame workers, default 2; 0 = viewer frames on
|
||
the page's thread), `?nosw` (don't register the service worker).
|
||
|
||
## How the desktop pieces map to the browser
|
||
|
||
| Desktop | Web |
|
||
|---|---|
|
||
| frame render thread pool (`ui-egui/src/frames.rs`) | frame workers (below): `Frames::pump` hands queued frames to them, most important first (the viewer's frame, then prefetch); frames the GPU compositor can keep on the GPU render on the page. Without workers, queued frames render on the UI thread after each egui frame within 40 ms (24 ms while playing) |
|
||
| Render Queue on a background thread | a Web Worker running a second engine instance (below); progress streams back, the page never waits |
|
||
| tracker / mask tracker / Warp Stabilizer / 3D Camera Tracker / Roto Brush Freeze threads | the same workers; the analysed property group comes back as one undo step |
|
||
| `rayon` parallel loops (raster, effects, export batches) | rayon's global pool falls back to the calling thread when threads are unavailable: same code, serial |
|
||
| `std::time::Instant` / `SystemTime` (panic on wasm32) | `web-time` (re-exports `std::time` on native) |
|
||
| `std::fs` project reads/writes (`FsServices`) | `files::WebServices`: the virtual file table, persisted (below); saving also downloads the `.ecproj` |
|
||
| config directory: settings, shortcut presets, recent projects, the recovery sentinel | `store::WebConfig` in browser storage |
|
||
| auto-save folder | `/EffectCraft Auto-Save/` in browser storage (`ConfigStore::files`) |
|
||
| media reads (`MediaPool`, `probe`) | `files::WebImporter`: bytes from the file table, handed to `MediaPool::add_bytes` / `probe_bytes` |
|
||
| export writes (`effectcraft-export`) | the job's `sink` (`effectcraft_host::FileExporter { sink }`): files land in the table and download after the render; several files (an image sequence) download as one stored `.zip` |
|
||
| rfd file dialogs | `<input type=file>` for File ▸ Open / Import; drop files anywhere on the page (`.ecproj` opens, everything else imports); "Save As" / "Output To" pick a download name |
|
||
| Media Browser on the file system | browser storage and folders opened with the File System Access API (below) |
|
||
| system fonts | not scanned; the bundled fonts (Inter, Noto Serif, JetBrains Mono) are always there |
|
||
| TCP control channel / MCP | `window.effectcraft` (below) |
|
||
| cpal audio output | Web Audio (below) |
|
||
| GPU compositor on the desktop's wgpu device | the same compositor on eframe's WebGPU device (below) |
|
||
|
||
### Persistence
|
||
|
||
Everything that lives in files on the desktop lives in the browser's **Origin Private File
|
||
System** (OPFS); where OPFS can't be written from the page (older Safari), **IndexedDB**; with
|
||
neither (some private windows), memory only. `effectcraft.info().storage.backend` says which.
|
||
The page asks for persistent storage (`navigator.storage.persist()`), which browsers grant to
|
||
installed apps, so the store isn't evicted under storage pressure.
|
||
|
||
The engine's storage traits are synchronous and the browser's are not, so the app works on an
|
||
in-memory mirror (`store.rs`): every stored entry is read before the app starts, reads and writes
|
||
hit the mirror, and changes flush in the background (one write per changed key, in order; OPFS
|
||
writes are atomic: a swap file replaced on close). Keys:
|
||
|
||
| Key | What |
|
||
|---|---|
|
||
| `config/prefs.json`, `config/shortcuts.json` | Settings (with File ▸ Open Recent's list) and keyboard shortcut presets |
|
||
| `config/session.lock` | the crash-recovery sentinel |
|
||
| `config/session.ecproj`, `config/session.json` | the session snapshot: the open project and editor state (active comp, time, selection), written at most once a second while they change |
|
||
| `files/<path>` | the file table: imported media, projects saved with File ▸ Save / Save As or `saveToBrowser`, auto-saves |
|
||
|
||
On the next visit the app reopens the snapshot (unsaved changes stay unsaved: the project is
|
||
dirty), registers every stored media file with the media pool so its footage decodes, and File ▸
|
||
Open Recent reopens saved projects from storage. Without a snapshot (first visit) it opens the
|
||
demo project. Render outputs are not stored: they download.
|
||
|
||
### Background jobs: Web Workers
|
||
|
||
wasm threads need shared memory, which needs a nightly `build-std` toolchain, so this build is
|
||
single-threaded per instance. Long jobs run in **Web Workers** instead, each with its own instance
|
||
of the same module (the page compiles it once and posts the `WebAssembly.Module`, so nothing is
|
||
compiled twice) and its own memory (`crates/engine/src/offload.rs`, `src/worker.rs`):
|
||
|
||
1. The session's `Offload` serializes a `WorkerRequest`: the project JSON, the footage paths it
|
||
reads, and the job (`render` with the resolved queue items and output paths; `warp`, `camera`,
|
||
`track`, `maskTrack`, `rotoFreeze` with the target and analysis parameters).
|
||
2. `js/host.js` takes an idle worker from its pool (or starts one), sends it the footage bytes it
|
||
doesn't have yet, then the request.
|
||
3. The worker runs the job on a plain engine session, blocking, and posts `WorkerReply`s: render
|
||
progress (at most every 100 ms), item started/finished, rendered files (transferred), analysis
|
||
progress, the analysed property group, `failed`, `done`.
|
||
4. The page applies them every frame: `Session::poll_render` updates the queue items and the
|
||
progress bar, `Session::poll_offload` writes an analysis result as one undo step (the effect /
|
||
tracker / mask group is replaced; other edits made meanwhile are kept) and the downloads start.
|
||
|
||
Stopping a job terminates its worker (a worker can't be interrupted mid-frame): the item being
|
||
rendered becomes "User Stopped", an analysis writes nothing. Every job shows in Window ▸ Progress
|
||
(`jobs.list`) with its progress and a cancel button (`jobs.cancel`).
|
||
|
||
**Roto Brush propagation** runs in a job worker too (`WorkerJob::RotoPropagate`): the worker
|
||
streams the segmentations it computes back at most every 250 ms (`WorkerReply::Segs`: each
|
||
`FrameSeg` in a compact binary form, run-length coded planes, base64), the page puts them into its
|
||
segmentation cache and relays them to the frame workers. Stopping it keeps the frames propagated
|
||
so far.
|
||
|
||
**`wait: true`** (the default of `renderQueue.render` for agents, and an option of the analysis
|
||
and Roto Brush commands) no longer blocks the page: over `window.effectcraft` (the control
|
||
channel) such a command runs with `wait: false`, and the promise resolves when the job it started
|
||
ends, with what the blocking command returns (final render item statuses, the analysis status)
|
||
or rejects with the job's error (`ui-egui/src/control.rs`, `JobWaiter`). Engine commands executed
|
||
directly on the session (scripts) still block.
|
||
|
||
### Viewer frames: frame workers
|
||
|
||
Viewer frames (the frame on screen, RAM preview and prefetch) render in long-lived **frame
|
||
workers** (`src/frames.rs`, `crates/engine/src/remote.rs`), two by default. Each holds a replica
|
||
of the project:
|
||
|
||
1. Before a render, the page sends a worker the footage files it lacks, then a sync: the whole
|
||
project the first time, afterwards a **diff** of the serialized project from the revision the
|
||
worker holds (`remote::diff`: object members added / changed / removed, arrays element-wise
|
||
when their length is unchanged; a slider drag sends a few hundred bytes, not the project).
|
||
2. Then the render request (`FrameMsg::Render`: revision, comp, time, render options). The worker
|
||
renders on its CPU renderer and posts the frame back as premultiplied RGBA8 in a transferred
|
||
buffer (`FrameReply::Frame`); the page puts it in the frame cache.
|
||
3. One frame per worker at a time, so the page always picks what renders next (the viewer's frame
|
||
before prefetch); a request for a revision the worker does not hold is refused, a worker that
|
||
lost its replica asks for the whole project again (`FrameReply::Resync`).
|
||
|
||
Frames the WebGPU compositor can finish alone still render on the page (they stay on the GPU, no
|
||
readback); everything else (CPU-only effects, 3D runs, the Software Only renderer, WebGL2) goes to
|
||
the workers. If no worker is usable, frames render on the page's thread as before.
|
||
`effectcraft.info().frameWorkers` reports `{workers, alive, busy, rendered, failed, lastMs,
|
||
syncs, syncBytes}`.
|
||
|
||
**Why workers and not wasm threads:** shared-memory threads (SharedArrayBuffer + atomics, a
|
||
`wasm-bindgen-rayon`-style pool) need the standard library rebuilt with atomics (`build-std`,
|
||
nightly Rust); the repository builds on stable, so each worker runs its own engine instance with
|
||
its own memory. The page is still cross-origin isolated (`--serve` and the service worker send
|
||
COOP / COEP), so a threaded build can be added later without changing the deployment.
|
||
|
||
### Media Browser
|
||
|
||
Window ▸ Media Browser browses a virtual tree (`src/browse.rs`):
|
||
|
||
- **Browser Storage**: every file in browser storage (imported media, saved projects, files added
|
||
with **Add Files…** or **Upload Folder…**, plain file inputs that work in every browser). This is
|
||
the persistent recent list where folders can't be opened.
|
||
- **Folders**: **Open Folder…** (File System Access API, `showDirectoryPicker`, Chromium) lists a
|
||
folder from the user's disk; only the listing is read, and a file's bytes are read when it is
|
||
imported (then kept in browser storage). The folder handles are remembered in IndexedDB; after
|
||
a reload the folders come back if the browser still grants access, else **Reconnect <name>**
|
||
asks again.
|
||
|
||
`mediaBrowser.list` reports the places and these actions; `mediaBrowser.action {action}` runs one.
|
||
|
||
### Audio
|
||
|
||
Audio preview goes through **Web Audio**: an AudioWorklet (`audio-worklet.js`; a
|
||
ScriptProcessorNode where worklets are unavailable) plays interleaved stereo blocks. The mixdown
|
||
is the desktop's (`AudioPlayback`, `mix_comp`): without a feeder thread, `AudioPlayback::pump`
|
||
mixes ahead on every UI frame (≈ 350 ms queued) and the output device takes what keeps it
|
||
≈ 150 ms ahead of the playhead. A/V sync follows `AudioContext.currentTime`: the playback clock
|
||
is the frames played by the context clock minus its output latency, and the viewer shows the
|
||
frame under the audible sample, as on the desktop. The Audio panel's meters read the same feed.
|
||
|
||
Browsers start an AudioContext suspended until the user interacts with the page; the app resumes
|
||
it on the first pointer or key event, so audio plays from the first preview after any click or
|
||
key press.
|
||
|
||
### GPU
|
||
|
||
eframe starts on WebGPU where the browser has it (WebGL2 otherwise), and the GPU compositor
|
||
(`effectcraft-gpu`) runs on the same device: viewer frames are composited by compute shaders and
|
||
drawn straight from their texture, with no readback. Steps that need the CPU mid-render (3D runs,
|
||
adjustment layers, CPU-only effects) fall back to the CPU renderer for that frame, because the
|
||
page's thread can't wait for a GPU readback. The Info panel's pixel readout and the eyedroppers
|
||
read GPU frames back asynchronously (`Gpu::read_display_async`: the pixels arrive a frame
|
||
later). On WebGL2, or without a usable adapter, everything renders on the CPU.
|
||
`effectcraft.info().gpu` reports `{compositor, viewerOnGpu}`.
|
||
|
||
### Offline and install (PWA)
|
||
|
||
`manifest.webmanifest` (name, icons, standalone display) makes the app installable; `sw.js`
|
||
precaches every file of the build into a cache named after the build's hash, serves them
|
||
cache-first (the cached responses keep the server's COOP/COEP headers, so the page stays
|
||
cross-origin isolated offline), and drops older caches when a new build activates. After the
|
||
first visit the app loads without a network.
|
||
|
||
## `window.effectcraft`: the agent / test API
|
||
|
||
The control channel of the desktop app (`docs/control-protocol.md`) as promises. Results resolve
|
||
with the method's `result`, errors reject with the message.
|
||
|
||
| Call | |
|
||
|---|---|
|
||
| `effectcraft.request(method, params)` | any control method: `ui.inspect`, `ui.click`, `ui.key`, `ui.playback`, `ui.menu.invoke`, `render.frame`… |
|
||
| `effectcraft.execute(command, params)` | `engine.execute`: an engine command (same ids and params as MCP / `effectcraft-cli`) |
|
||
| `effectcraft.commands()` / `effectcraft.inspect()` | `engine.commands` / `ui.inspect` |
|
||
| `effectcraft.renderFrame(params)` | `render.frame` with `base64: true`: `{comp, time, width, height, png}` |
|
||
| `effectcraft.screenshot(params)` | `ui.screenshot`; without `path` the PNG comes back inline as `png` (base64) |
|
||
| `effectcraft.addFile(fileOrUrl, name?)` | put a `File`/`Blob` (or fetched URL) into the file table (stored) and open (`.ecproj`) or import it; resolves with its path |
|
||
| `effectcraft.files()` / `effectcraft.readFile(path)` | the file table `[{path, size}]` / a file's bytes (`Uint8Array`), e.g. a render |
|
||
| `effectcraft.saveToBrowser(path?)` | save the project to browser storage without downloading it (default: its path, or `/<name>.ecproj`); resolves with `{path, bytes}` |
|
||
| `effectcraft.listStored()` | `{backend, usage, quota, persisted, pending, files: [{path, size, modified}], config: [name]}` |
|
||
| `effectcraft.removeStored(path)` | delete a stored file |
|
||
| `effectcraft.flush()` | resolves once every change is written to browser storage |
|
||
| `effectcraft.info()` | graphics backend, `gpu`, `storage`, `audio` (`{state, sampleRate, backend, posted, played, underruns}`), `workers`, `restored`, `webgpu`, `serviceWorker`, version, `crossOriginIsolated`, load timings |
|
||
|
||
`window.effectcraftLoad` holds `{wasmMs, readyMs}` (module fetch + compile, and until the app runs).
|
||
|
||
```js
|
||
await effectcraft.execute("layer.newSolid", {color: "#ff8800"});
|
||
await effectcraft.request("ui.menu.invoke", {id: "renderQueue.add", params: {format: "gif", resolution: 0.25}});
|
||
await effectcraft.execute("renderQueue.render", {wait: false}); // renders in a worker; the GIF downloads
|
||
await effectcraft.saveToBrowser("/my-project.ecproj"); // File ▸ Open Recent has it after a reload
|
||
```
|
||
|
||
## Browser test
|
||
|
||
`apps/effectcraft-web/tests/smoke.mjs` drives headless Chrome over the DevTools protocol (Node ≥ 22,
|
||
no npm packages): load, the GPU path (WebGPU → GPU compositor, viewer on the GPU), the demo comp
|
||
in the viewer, `render.frame`, Render Queue GIF and PNG sequence (`.zip`) downloads, a background
|
||
render in a worker while measuring the page's event-loop gaps (must stay under 400 ms) and its
|
||
progress, the AudioContext starting on a (simulated) key press and a tone playing through it
|
||
with the meters moving, Save As and re-opening the saved project, persistence across a reload
|
||
(session snapshot, a stored project, imported media, a setting, Open Recent) and an offline
|
||
reload through the service worker. It writes screenshots and `report.json`:
|
||
|
||
```sh
|
||
cargo xtask web --serve 8765 &
|
||
node apps/effectcraft-web/tests/smoke.mjs --url http://127.0.0.1:8765/ --out target/web/smoke
|
||
```
|
||
|
||
It also checks the M13.10 paths: viewer frames rendered in frame workers while scrubbing with the
|
||
CPU renderer (project synced as diffs, event-loop gaps under 400 ms), a `wait: true` render that
|
||
replies when done while the page keeps running, and the Media Browser over browser storage.
|
||
|
||
Native unit tests cover the storage model (`apps/effectcraft-web/src/store.rs`: write
|
||
coalescing, file table, `ConfigStore` / `FileOps`, auto-save and crash recovery through the
|
||
browser store) and the worker protocol (`crates/engine/src/offload.rs`: serde round trips of
|
||
every request and reply, a render through an in-process offload; `tests_roto.rs`: Roto Brush
|
||
propagation through an offload streaming segmentations), the frame-worker protocol
|
||
(`crates/engine/src/remote.rs`: diff / patch, a replica following edits renders the same pixels;
|
||
`crates/ui-egui/tests/ui_web_offload.rs`: the frame cache through a remote renderer, `wait: true`
|
||
deferred until the offloaded job ends) and the Media Browser's virtual tree (`media_browser.rs`,
|
||
`tests_panels.rs`).
|
||
|
||
## Gaps
|
||
|
||
- No shared-memory threads: each worker is a separate engine instance (memory per worker: the
|
||
project replica and the footage it reads).
|
||
- GPU effects run only in the desktop GPU path: in the browser, effects run on the CPU (inside
|
||
workers for renders) because the page's thread can't wait for mid-render readbacks.
|
||
- No disk cache (Settings ▸ Media & Disk Cache) in the browser: the layer cache stays in memory.
|
||
- Cancelling an analysis drops its partial result (the worker is terminated).
|
||
- Stored media count against the origin's storage quota; `effectcraft.listStored()` shows usage,
|
||
`removeStored` frees it (there is no storage manager in the UI yet).
|