195 lines
20 KiB
Markdown
195 lines
20 KiB
Markdown
# Control protocol & MCP
|
||
|
||
## Desktop control channel
|
||
`filmcraft --control 9876` (or `FILMCRAFT_CONTROL_PORT`) listens on `127.0.0.1:<port>` (loopback only).
|
||
One JSON request per line → one JSON reply per line. The app opts out of macOS App Nap
|
||
(`apps/filmcraft/src/app_nap.rs`): a hidden window would otherwise drop the whole process to
|
||
background priority, and on a busy machine it would stop answering.
|
||
|
||
**Only requests are read.** Every line must be a JSON object with a string `method` (`id` and `params`
|
||
are optional; blank lines are skipped). Anything else gets one error reply
|
||
(`{"ok": false, "error": "… closing the connection"}`) and the server **closes the connection**, so
|
||
nothing sent after it on that connection runs. That covers text that isn't JSON, a JSON array or number,
|
||
an object without `method`, invalid UTF-8, and a line longer than 4 MiB. An HTTP request (for example a
|
||
web page's cross-origin `fetch` to `127.0.0.1:<port>`) therefore can't smuggle a command in its body:
|
||
its request line is rejected first. At most 16 connections are served at once; further ones get an error
|
||
line and are closed. Clients that get an error reply should reconnect. The port has no authentication,
|
||
so only enable it while you use it. Transport: `apps/filmcraft/src/control_server.rs`.
|
||
|
||
```json
|
||
{"id": 1, "method": "engine.execute", "params": {"command": "sequence.addEdit", "params": {"seconds": 3}}}
|
||
{"id": 1, "ok": true, "result": {"cuts": 2}}
|
||
```
|
||
|
||
Methods (handlers in `crates/ui-egui/src/control.rs`):
|
||
|
||
| Method | Params | |
|
||
|---|---|---|
|
||
| `engine.execute` / `ui.menu.invoke` | `{command, params}` | any engine or UI command id |
|
||
| `engine.commands` / `ui.menu.list` | – | command registry / menu tree |
|
||
| `ui.inspect` | – | UI state (tool, workspace, dock, timeline view, playback, fps…) |
|
||
| `ui.elements` | `{prefix?}` | every on-screen interactive element: id, label, rect |
|
||
| `ui.set` | `{tool, workspace, mode, theme, focused, playbackRes, timeline:{pps,scroll,fit}, program:{…}, source:{…}, clipDialog:{param: value}, menuDialog:{param: value}, panels:{scopes, timecode, reference, events, progress}, export:{…}, keyframeSelection:[…]}` | `keyframeSelection` replaces the Effect Controls keyframe selection (`[{clip, effect, param, mask?, time}]`, `time` in media ticks; read back as `ui.inspect` → `ui.keyframe_selection`; a click on a lane keyframe selects just that one); `program` / `source` merge fields into the monitor view state (`res`, `paused_res`, `high_quality`, `display`, `zoom`, `pan`, `show_rulers`, `show_guides`, `lock_guides`, `snap`, `guides`, `compare_ref`, …; see [monitors.md](monitors.md)); `clipDialog` sets fields of the open Edit / Clip / File dialog (Paste Attributes, Make Subclip, Frame Hold Options, …); `menuDialog` those of the M3.11 dialogs (Find, Create Search Bin, Project Settings, Scene Edit Detection, Simplify Sequence, Automate to Sequence…; see `panels::menu_dialogs`); `panels` merges into the panel settings (`ui.inspect` → `ui.panels`; see `panels::panel_state`): Lumetri Scopes `{shown: ["vectorscopeYuv"|"vectorscopeHls"|"histogram"|"parade"|"waveform"], waveformType, paradeType, colorSpace, brightness, scale, clamp, targets}`, Timecode `{rows: [{source, mode, display}], showName}`, Reference Monitor `{ganged, time, display: "composite"|"scopes", scopes}`, Events `{level, selected}`, Progress `{showFinished}` `export` deep-merges into Export mode state (`preset` — applied first —, `settings` (ExportSettings JSON), `fileName`, `location`, `range`, `customStart`, `customEnd`, `openSections`, `manager` (Preset Manager: `query`, `favoritesOnly`, `selected`, `saveName`), `quickOpen`, `quickPath`, `quickPreset`); `program` / `source` merge fields into the monitor view state (`res`, `paused_res`, `high_quality`, `display`, `zoom`, `pan`, `show_rulers`, `show_guides`, `lock_guides`, `snap`, `guides`, `compare_ref`, …; see [monitors.md](monitors.md)); `clipDialog` sets fields of the open Edit / Clip / File dialog (Paste Attributes, Make Subclip, Frame Hold Options, …); `menuDialog` those of the M3.11 dialogs (Find, Create Search Bin, Project Settings, Scene Edit Detection, Simplify Sequence, Automate to Sequence…; see `panels::menu_dialogs`); `projectPanel` merges into the Project panel's view state (`bin` shown in place, `tabs` (bins in a tab or window: `{bin, floating, view, iconSize, nav}`), `activeTab`, `selectedBin`, `rename: {item|bin, text}`, `dialog` (`{"metadataDisplay": {columns, filter}}`, `{"savePresetAs": {name}}`, `{"managePresets": {selected, name}}`, `{"freeformOptions": {options}}`, `{"saveArrangement": {name, bin}}`)); `mediaBrowser` into the Media Browser's (`expanded`, `pathEdit`, `editColumns`, `treeWidth`). View settings, columns and presets are engine commands (`project.view.set`, `project.columns.set`, `project.viewPreset.*`, `mediaBrowser.settings`) |
|
||
| `ui.panel.show` / `ui.panel.close` | `{panel}` | |
|
||
| `ui.click` / `ui.move` | `{id}` or `{x,y}`, `button`, `count`, `modifiers` | synthetic pointer input |
|
||
| `ui.drag` | `{from, to, steps, modifiers}` | press–move–release |
|
||
| `ui.scroll` | `{id|x,y, dx, dy, modifiers}` | wheel / trackpad |
|
||
| `ui.key` / `ui.type` | `{key}` (`Cmd+K`, `Space`…; off macOS `Ctrl+K` is the same key) / `{text}` | keyboard |
|
||
| `ui.timeline.hit` / `ui.timeline.locate` | `{x,y,modifiers?}` / `{clip, edge?}` | timeline hit-testing. `hit` reports a clip's `edge` (`In`/`Out`) within 7 px of it (a third of a narrow clip); just left of a cut that is the left clip's Out, from the cut rightwards the right clip's In. `kind` is the trim a press of the current tool with `modifiers` starts there (`trim`, `ripple`, `roll`, `rateStretch`, `remix`), `volume` on an audio clip's Volume line (Selection and Pen tools), or null |
|
||
| `ui.playback` | `{action: play|stop|toggle, speed?}` | |
|
||
| `ui.screenshot` | `{path?, panel?}` | PNG of the window or one panel; fails after 10 s when no frame is presented (window hidden, display asleep) |
|
||
| `ui.resize`, `app.quit` | `{width,height}` / `{}` | resize or close the native window; `app.quit` quits without the save prompt (unsaved changes stay in the recovery journal), while the command `app.quit` asks first |
|
||
| `ui.focus` | `{}` / `{id}` | empty parameters activate the native window; a widget ID requests egui focus without OS activation |
|
||
| `perf.stats` | – | performance counters (also the command `perf.stats`, so `engine.execute` and MCP `command_run` reach it; headless sessions return the engine part). `decode`: GOP-cache requests / hits / `cacheHitRate`, decoder seeks, samples decoded and skipped while catching up, `draftFrames` (decoded in draft mode), `h264Threads` (frame-threading workers per H.264 decoder), `liveDecoders` (sources holding a decoder now), `cacheMB` / `cacheBudgetMB` (decoded frames all sources hold, and the shared budget), `planePoolMB` / `planesReused` (idle buffers of the frame pool, the planes of evicted frames and the compositor's float images, which a standalone export frees when it ends; and buffers taken from it instead of allocated), decoder ms (total and per sample), `framesDecoded`, `hardware` (Settings ▸ Playback ▸ Hardware decoding: `enabled`, pictures from hardware decoders `frames` vs `softwareFrames`, `sessions`, `declined` streams, mid-stream `fallbacks`); `playback`: playing, shown / dropped frames and `dropRate` of the current or last play, resolution, `draftDecode` (Settings ▸ Playback ▸ Draft decoding), `audio` (desktop: sound is mixed 200 ms ahead of the device; device `callbacks`, `underruns` and `missingFrames` played as silence because the mixer fell behind, `minLeadMs` the least sound left buffered); `frames`: frame-worker jobs, cancelled, `requestHitRate`, `decodeMs` / `renderMs` (mean, p50, p95 of the last 256 jobs), queue, render-cost estimate, cache use; `monitor`: the `frame` and `revision` of the picture the Program monitor shows against `playheadFrame` and `projectRevision` (how far a drag or a scrub is ahead of the picture); `ui`: fps and frame ms; `process`: CPU seconds; `media`, `jobs`. Counters are cumulative: diff two readings to measure an interval ([performance.md](performance.md)) |
|
||
|
||
**Timeline thumbnails.** `ui.set {timeline: {thumbnails}}` sets the Timeline wrench menu's Show Video
|
||
Thumbnails: `off`, `head`, `headAndTail` or `continuous`.
|
||
|
||
**Selection or explicit targets.** A command that acts on the selection is "not available right now" when
|
||
nothing is selected. When `params` name the targets under a key the command documents (`clips` /
|
||
`clip`: timeline clips of the active sequence, `items` / `item`: project items), enablement is
|
||
checked against those instead and the selection is left as it is; the command's other conditions
|
||
still apply. `engine.commands` and the menus always show the selection's enablement.
|
||
|
||
**Focus.** Driving the app never steals the user's keyboard. Started with `--control`, the app opens
|
||
without activating itself. UI requests and screenshots that need a rendered frame bring the window
|
||
forward without making it key (macOS `orderFrontRegardless`), so the user's typing keeps going to
|
||
the app they are in. Only an explicit `ui.focus {}` activates FilmCraft. Injected keys work on
|
||
focused widgets even when the native window is in the background.
|
||
|
||
`ui.focus {id}` targets an enabled, visible widget registered with a real egui response. It rejects
|
||
unknown IDs, non-widget regions and hidden tool aliases. The reply acknowledges the request;
|
||
`ui.inspect.keyboardFocus` reports the actual widget ID, matching element IDs, label and active
|
||
focus. This is separate from `ui.focused`, which identifies the active panel. Tools, dock tabs,
|
||
tab menus and divider handles expose widget IDs through `ui.elements`. Tab/Shift-Tab move widget
|
||
focus; Space/Enter activate focused buttons and tabs, with visible focus outlines.
|
||
|
||
For unattended validation, start with `--control <port> --control-input-only`. This opt-in mode
|
||
accepts widget input through control requests and excludes native keyboard, pointer, file-drop
|
||
and menu commands. Window close/quit and screenshots remain available. Ordinary launches and
|
||
`--control` alone keep native input enabled. `ui.inspect.input` distinguishes `mode`, the observed
|
||
`nativeWindowFocused`, and egui's `effectiveWindowFocused`; effective focus does not activate the
|
||
OS window or prove physical-keyboard delivery.
|
||
|
||
Element ids are stable, e.g. `timeline.clip.<id>`, `timeline.track.V1.lock`, `tools.Razor`,
|
||
`project.item.<id>`, `effects.item.gaussian_blur`, `panel.tab.Timeline`, `program.transport.playback.toggle`.
|
||
|
||
Audio mixing: `mixer.*` commands (strips `"A1"`, `"S1"`, `"Mix"` or ids; lanes `volume`, `pan`,
|
||
`mute`, `send.<i>.level`, `fx.<slot>.<param>`): `mixer.inspect`, `mixer.setStrip` (volume, pan, mute,
|
||
solo, record arm, solo safe, mode Off/Read/Latch/Touch/Write, output, input map, channels),
|
||
`mixer.setValue`, `mixer.touch` / `mixer.release` (a fader gesture; recorded during an automation pass),
|
||
`mixer.recordStart` / `mixer.recordStop` (playback runs them), `mixer.addSubmix`, `mixer.deleteSubmix`,
|
||
`mixer.addInsert` / `removeInsert` / `setInsert`, `mixer.addSend` / `setSend` / `removeSend`,
|
||
`mixer.setKeyframe` / `deleteKeyframe` / `moveKeyframe` / `clearLane`, `mixer.writeAutomation`;
|
||
`clipMixer.set`; `clip.audioGain {clips?, mode: set|adjust|normalizeMax|normalizeAll, db}`, `clip.audioPeak`;
|
||
`effects.setDefaultTransition`. UI ids: `mixer.<A1|S1|Mix>.<fader|value|pan|panValue|mode|mute|solo|
|
||
record|soloSafe|output|input|fx.<n>|send.<n>|meter|name>` (popup entries below them, e.g.
|
||
`mixer.A1.mode.Touch`, `mixer.A1.fx.0.studio_reverb`), `mixer.showEffects`, `mixer.transport.<cmd>`,
|
||
`clipMixer.A1.<fader|pan|mute|solo|keyframe|value>`, `timeline.track.A1.keyframes[.<lane>|.clip]`,
|
||
`timeline.track.A1.lane[.kf.<n>]`, `audioGain.<set|adjust|normalizeMax|normalizeAll|ok|cancel|peak>`.
|
||
|
||
Essential Sound: `essentialSound.inspect`, `essentialSound.setType {type: dialogue|music|sfx|ambience}`,
|
||
`essentialSound.clearType`, `essentialSound.set {key, value}` or `{values: {key: value}}` (keys such as
|
||
`repair.noise.on`, `repair.noise.amount` (0–10), `repair.humHz`, `clarity.eqPreset`, `clarity.enhanceTone`,
|
||
`creative.reverbPreset`, `ducking.against`, `ducking.reduceDb`, `ducking.fadeS`, `pan.value`,
|
||
`volume.levelDb`, `mute`, section switches `<section>.enabled`), `essentialSound.applyPreset {preset, type?}`,
|
||
`essentialSound.savePreset {name}`, `essentialSound.deletePreset {name}`, `essentialSound.autoMatch {target?}`,
|
||
`essentialSound.generateDucking`; all take `clips` (default: the selected audio clips). UI ids:
|
||
`essentialSound.tab.<Browse|Edit>`, `essentialSound.type.<Dialogue|Music|SFX|Ambience>`,
|
||
`essentialSound.clearType`, `essentialSound.preset[.save|.delete|.name|.ok]`,
|
||
`essentialSound.section.<Name>[.toggle]`, a row per setting key (`essentialSound.repair.noise.on`,
|
||
`essentialSound.repair.noise.amount`, `essentialSound.repair.humHz.<i>`), `essentialSound.autoMatch`,
|
||
`essentialSound.ducking.against.<Type>`, `essentialSound.generateDucking`, `essentialSound.volume.on`,
|
||
`essentialSound.volume.levelDb`, `essentialSound.mute`, `essentialSound.browse.<Type>.<preset>`.
|
||
|
||
Project files, auto-save, crash recovery and preferences commands (`file.recover`, `prefs.set`, …) and their
|
||
automation ids are listed in [project-files.md](project-files.md). That file also lists the media
|
||
management commands and dialog ids: offline media and relinking (`media.findMissing`,
|
||
`media.relink`, `media.autoRelink`, `media.search`, `media.makeOffline`, `media.status`, `linkMedia.*`),
|
||
proxies (`media.createProxies`, `media.attachProxies`, `media.toggleProxies`, `proxies.*`), ingest
|
||
(`project.ingestSettings`) and the Project Manager (`file.projectManager`, `pm.*`).
|
||
|
||
**Moving clips**: `timeline.move {moves: [{clip, track, time}], insert?, linked?}` → `{moved: [clip ids]}`.
|
||
With Linked Selection on (the default; `linked` overrides the toggle for one call) the linked
|
||
partners of a listed clip that are not listed themselves move by the same time offset and stay on
|
||
their own tracks, so picture and sound stay in sync. Like a listed clip, a partner overwrites
|
||
what is at its destination (or inserts, with `insert`). A partner on a locked track stays where it
|
||
is, and so does one whose position would not change (the listed clip only changed track). `moved`
|
||
lists every clip that moved, partners included. No clip is moved before the sequence
|
||
start: if one would be, all clips of the call land later by the same amount (their spacing is
|
||
kept). A `time` beyond the representable range is an "invalid parameters" error. The timeline
|
||
panel's drag passes `linked: false`, because it already lists exactly the clips it moves. With
|
||
`copy: true` (Option-drag in the timeline) the clips stay where they are and copies land at the
|
||
destinations instead (linked partners are copied too, unless `linked: false`); the result lists
|
||
the new clips as `copied`, they are selected, and copies of linked clips are linked to each other.
|
||
|
||
**Trim mode** (`crates/engine/src/trim.rs`): `trim.selectEditPoint` / `trim.selectNearest` enter trim
|
||
mode (the Program monitor becomes the Trim Monitor, ids `trimMonitor.*`); `trim.monitor` returns what
|
||
it shows. Dynamic trimming takes an explicit clock (seconds) so it is deterministic: `trim.shuttle
|
||
{direction, slow?, clock}` (J/L), `trim.tick {clock}`, `trim.shuttleStop {clock?}` (K, one undo step),
|
||
`trim.cancelDynamic` (Esc), `trim.playAround {clock, loop?}` (Space / Shift+K).
|
||
|
||
**Replace With Clip**: `clip.replaceFromSource`, `clip.replaceFromSourceMatchFrame` and
|
||
`clip.replaceFromBin {clips?, item?, keepSourceIn?}` swap the media of the clips and return
|
||
`{clips: [ids], item, short: [{clip, shortByFrames}]}`. `short` lists the replaced clips that now
|
||
run past the end of their media (the renderer holds the last frame there) with the number of
|
||
sequence frames that have no media, at most the clip's length; it is empty when the new media
|
||
covers every clip. The edit is the same either way. From Bin starts each clip at the new item's In point; with
|
||
`keepSourceIn: true` each clip keeps its own source In instead. A subclip that restricts trims has
|
||
no media outside its range, so there a kept In is moved to the subclip's first or last frame, and
|
||
the result's `sourceInClamped: [ids]` (present with `keepSourceIn`) names the clips it was moved for.
|
||
|
||
**Colour** (`crates/engine/src/color.rs`): `sequence.colorSettings {workingSpace: rec709|rec2100-pq|
|
||
rec2100-hlg, wideGamut, autoToneMap}`, `clip.interpretFootage {items?, colorSpace?: auto|<id>, pixelAspect?: [num, den]|file}`,
|
||
`color.spaces`, `media.colorInfo {item}`; LUTs: `lut.import {path, name?}`, `lut.list`,
|
||
`lut.remove {id}`, `lut.export {lut, path, format?}`, `lumetri.setInputLut` / `lumetri.setLook
|
||
{clip?, lut: lib:<id>|builtin:<id>|"", path?}`, `lumetri.setSection {clip?, section, on?}`,
|
||
`lumetri.applyMatch {clip?, referenceTime|referenceFrame|referenceTimecode, faceDetection?}`. Without
|
||
params the menu entries open dialogs (ids `colorDialog.space.<id>`, `colorDialog.working.<id>`,
|
||
`colorDialog.wideGamut`, `colorDialog.autoToneMap`, `colorDialog.ok|cancel`). Lumetri panel ids:
|
||
`lumetri.input_lut`, `lumetri.look_lut`, `lumetri.switch.<section>`, `lumetri.section.<section>`,
|
||
`lumetri.match.*`; HDR scopes: `scopes.hdrWaveform`. `file.exportMedia {sdr: true}` exports an HDR
|
||
sequence as tone-mapped SDR.
|
||
|
||
**Keyboard shortcuts** (`crates/engine/src/shortcuts.rs`): `shortcuts.list {query?, panel?}`,
|
||
`shortcuts.get`, `shortcuts.set {command, keys, panel?, add?, keepConflicts?}`, `shortcuts.clear`,
|
||
`shortcuts.undo` / `shortcuts.redo`, `shortcuts.conflicts {platform?}`, `shortcuts.forKey {key}`,
|
||
`shortcuts.resolve {keys, panel?}`, `shortcuts.presets`, `shortcuts.loadPreset` / `savePreset` /
|
||
`deletePreset {name}`, `shortcuts.export` / `import {path}`, `shortcuts.audit`. Keys use `Cmd` (⌘ /
|
||
Ctrl), `Ctrl` (macOS ⌃), `Alt`, `Shift`. The dialog (Edit ▸ Keyboard Shortcuts…, ⌥⌘K) uses ids
|
||
`shortcuts.*` (`shortcuts.key.K`, `shortcuts.cell.<command>`, `shortcuts.ok`, …).
|
||
|
||
## MCP
|
||
`filmcraft-cli mcp` serves MCP on stdio: headless (in-process session; `--demo` / `--project p.fcproj`)
|
||
or `--bridge 127.0.0.1:9876` to drive the running app. Tools: `command_list`, `command_run`,
|
||
`command_batch`, `doc_inspect`, `render_preview`, `project_inspect`, `sequence_inspect`, `media_import`,
|
||
`render_frame`, `ui_inspect`, `ui_elements`, `ui_click`, `ui_drag`, `ui_key`, `ui_type`, `ui_screenshot`,
|
||
`ui_control`; resources `filmcraft://document` and `filmcraft://commands`. `.mcp.json` registers both.
|
||
Annotations, argument checking, errors and export progress / cancellation:
|
||
[agents.md § Conventions](agents.md#conventions).
|
||
|
||
Time is in ticks (254 016 000 000 per second); commands also accept `seconds`, `frame` or `timecode`.
|
||
|
||
## Panel docking commands
|
||
|
||
`ui.menu.invoke` and `engine.execute` route these frontend commands:
|
||
|
||
| Command | Parameters | Result |
|
||
| --- | --- | --- |
|
||
| `window.floatPanel` | `{panel?, rect?: [x,y,width,height]}` | Move an existing panel into a floating group. Omitted panel uses the focused panel; omitted rect uses a centered 420×320 group. |
|
||
| `window.dockPanel` | `{panel?, anchor?, zone?: "center"\|"left"\|"right"\|"top"\|"bottom", before?}` | Move into the anchor's tab group or a side split; `before` names an existing destination tab for precise ordering. Floating destinations accept tabs only. Without `anchor`, a side `zone` docks the panel along that edge of the whole workspace, at a quarter of its width or height. |
|
||
| `window.closePanel` | `{panel?}` | Close a docked or floating panel without closing its project documents. |
|
||
|
||
Invalid source/target identities and non-finite or out-of-range geometry return an error with
|
||
the layout unchanged. `ui.inspect.ui.floating` reports floating groups and their logical-point
|
||
rectangles. `panel.drag.<Panel>` locates each panel's header grip; `panel.float.<index>.move`
|
||
and `.resize` locate floating geometry handles. Named workspace saves include these groups.
|
||
|
||
`window.dockAction` accepts `{action}` using the shared docking action's JSON form. It
|
||
covers tab moves and ordering, split and floating geometry, activation, and closing.
|
||
For example, `{"action":{"ResizeSplit":{"path":[],"size":{"FixedFirst":360}}}}`
|
||
resizes the root split; path entries use `false` for the first child and `true` for
|
||
the second. `{"action":{"MoveFloating":{"panel":"Info","rect":[45,65,420,320]}}}`
|
||
repositions and resizes the floating group containing Info. Invalid actions leave the
|
||
layout unchanged. Pointer gestures dispatch through this same catalog command.
|