shifathosn
/
filmcraft
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
filmcraft/docs/agents.md

10 KiB
Raw Blame History

Agents

FilmCraft is built to be driven by AI agents, and largely built by them. Part 1 covers using FilmCraft from an agent. Part 2 covers developing FilmCraft as an agent.

Part 1: driving FilmCraft

1. MCP server

filmcraft-cli mcp serves MCP on stdio. It has two modes:

Mode Command What it drives
Headless filmcraft-cli mcp --demo or --project p.fcproj (neither = empty project) an in-process engine session, no window
Bridge filmcraft-cli mcp --bridge 127.0.0.1:9876 the running desktop app started with filmcraft --control 9876

Claude Code. The repository's .mcp.json registers both servers (filmcraft = bridge, filmcraft-headless = demo). Both point at target/release/filmcraft-cli, so build it first:

cargo build --release -p filmcraft-cli -p filmcraft
./target/release/filmcraft --control 9876 &      # only for the bridge server
claude                                           # approve the project MCP servers when asked

To register the server by hand, or for another MCP client, use the absolute binary path:

claude mcp add filmcraft-headless -- /abs/path/filmcraft/target/release/filmcraft-cli mcp --demo

{ "mcpServers": { "filmcraft": { "command": "/abs/path/filmcraft-cli", "args": ["mcp", "--bridge", "127.0.0.1:9876"] } } }

Tools

Tool Modes Purpose
command_list both every command: id, label, menu, shortcut, params, enabled now (filter, enabled_only)
command_run both run a command {id, params}; edits are undoable
project_inspect both bins and items with ids, types, durations; active sequence
sequence_inspect both the active sequence: tracks, clips (ticks and frames), effects, transitions, markers, playhead, selection
media_import both import files by absolute path (text, one path per line)
render_frame both PNG of the program frame at seconds (headless renders; bridge screenshots the Program monitor)
ui_inspect bridge UI state: tool, workspace, panels, zoom, playback, fps
ui_elements bridge on-screen interactive elements with id, label and rect (prefix filter)
ui_click bridge click by id or x,y; button, count, modifiers
ui_drag bridge press–move–release between points or element ids
ui_key bridge key or shortcut: Space, Cmd+K, Shift+Delete
ui_type bridge type text into the focused field
ui_screenshot bridge PNG of the window or one panel
ui_control bridge call any control-channel method directly (ui.set, ui.scroll, ui.timeline.locate, …)

Typical loop: project_inspect / sequence_inspect → get ids → command_run → render_frame or ui_screenshot → look at the result → edit.undo if it's wrong.

2. Control channel

filmcraft --control 9876 (or FILMCRAFT_CONTROL_PORT=9876) listens on 127.0.0.1 only. Send one JSON request per line and get one JSON reply per line. Full method table: control-protocol.md.

{"id":1,"method":"engine.commands"}
{"id":2,"method":"engine.execute","params":{"command":"file.openDemoProject"}}
{"id":3,"method":"engine.execute","params":{"command":"playhead.set","params":{"seconds":2}}}
{"id":4,"method":"engine.execute","params":{"command":"sequence.addEdit"}}
{"id":5,"method":"engine.execute","params":{"command":"effects.apply","params":{"effect":"Gaussian Blur"}}}
{"id":6,"method":"ui.elements","params":{"prefix":"tools."}}
{"id":7,"method":"ui.click","params":{"id":"tools.Razor"}}
{"id":8,"method":"ui.key","params":{"key":"Cmd+Z"}}
{"id":9,"method":"ui.screenshot","params":{"path":"/tmp/timeline.png","panel":"Timeline"}}

Replies look like {"id":4,"ok":true,"result":{"cuts":2}} or {"id":7,"ok":false,"error":"…"}. Minimal client:

import json, socket
s = socket.create_connection(("127.0.0.1", 9876)); f = s.makefile("rw")
def call(method, **params):
    f.write(json.dumps({"id": 1, "method": method, "params": params}) + "\n"); f.flush()
    return json.loads(f.readline())
print(call("engine.execute", command="sequence.inspect"))

Notes:

  • Time is in ticks (254 016 000 000 per second). Commands also accept seconds, frame or timecode.
  • engine.execute also runs UI-only commands (tool.razor, playback.toggle, window.workspace.color, window.panel.<name>).
  • Element ids come from the previous frame. If an element is missing right after a layout change, the app retries on later frames before giving up.
  • Without a window, filmcraft-cli run script.jsonl (lines of {"id":"…","params":{…}}) runs the same commands headlessly. See §2b for the rest of the CLI.

2b. Command-line interface

Every command is also one shell call away. Options go anywhere; output is JSON; exit status is 0 on success, 1 when a command fails and 2 on a usage error. filmcraft-cli help prints the reference.

filmcraft-cli commands razor                     # find ids (add --json for machine output)
filmcraft-cli describe timeline.razor            # one command: menu, shortcut, params, enabled now
filmcraft-cli --demo inspect sequence            # project / sequence as JSON
filmcraft-cli --project p.fcproj --save exec timeline.razor seconds=3.5
filmcraft-cli --project p.fcproj exec effects.apply '{"effect":"Gaussian Blur"}'
filmcraft-cli --project p.fcproj --save import a.mov b.wav
filmcraft-cli --project p.fcproj export out.mp4  # format from the extension; waits for the job
filmcraft-cli --project p.fcproj export out --preset "YouTube 1080p Full HD" --start 0 --end 10
filmcraft-cli export --list-presets prores       # built-in + user presets (--data-dir for another library)
filmcraft-cli --project p.fcproj exec export.queue.add preset="Apple ProRes 422 HQ" path=renders/ start=true wait=true
echo '{"id":"file.newBin","params":{"name":"Selects"}}' | filmcraft-cli --project p.fcproj --save run -
filmcraft-cli --bridge 127.0.0.1:9876 exec window.workspace.color   # the running app

key=value values are parsed as JSON when they can be (3.5, true, [1,2]), otherwise taken as strings; dotted keys nest (color.r=1). run prints one JSON line per command ({"line","id","ok","result"|"error"}) and stops at the first failure unless --keep-going. --save writes back to --project; --save-as path writes elsewhere.

3. Verifying UI work

For any UI change, look at the result:

  1. cargo run --release -p filmcraft -- --control 9876
  2. Drive the feature through the control channel or the MCP bridge: commands, then clicks and drags by automation id.
  3. Assert on ui.inspect, ui.elements and sequence.inspect.
  4. Take ui.screenshot (the whole window and the panel you changed) and open the PNG.

Screenshots you commit must follow AGENTS.md §1: FilmCraft only, openly licensed media, sidecar and ATTRIBUTION.md entry.

Part 2: developing FilmCraft

4. Orientation, in this order

  1. AGENTS.md: absolute rules (assets, clean-room, licences).
  2. CLAUDE.md: working instructions and non-negotiables.
  3. ROADMAP.md: read the honest assessment and Where we are lacking first, then milestones, what's done and what's running.
  4. architecture.md, then the README and tests of the crate you'll touch.
  5. contributing.md (how to add things, gates) and testing.md.

Maintainers also keep a local planning folder, plan/. It is gitignored and not in the repository, and holds the task-level plan, the status file and behaviour reference notes. If you don't have it, everything you need to contribute is in the public docs above. Ask a maintainer for a task id.

5. Autonomous work loop

  1. Orient. Pick the next task: the next unchecked task in the maintainer status file, or an open ROADMAP item. Prefer the gaps in ROADMAP's Where we are lacking (speed, correctness on real media, measurement, Windows/Linux) over adding more checklist items. Read the relevant architecture section and crate README.
  2. Plan tests first. Write down the acceptance test before writing code.
  3. Implement and test.
  4. Verify. Run all gates (cargo xtask ci). For UI work, run the app with --control, drive it and look at the screenshots (§3).
  5. Record. Update the crate README (behaviour decisions, test results, limitations) and ROADMAP.md when a milestone moves. Commit as M<n>.<k>: …, then move on to the next task.

For parallel agents, use one git worktree and one CARGO_TARGET_DIR per agent, and keep each crate with one owner (contributing.md §5).

6. Decision rules

Don't block on choices you can make yourself:

  • Behaviour. Match the publicly documented and observable behaviour of professional editors (help pages, published shortcut lists, using the app). Never inspect application internals (AGENTS.md §2). If behaviour is undocumented, choose what editors most commonly expect and record the choice in the crate README.
  • Ask a human only about:
    • licences outside the allowlist;
    • spending money;
    • publishing or pushing;
    • personal data;
    • destructive git operations;
    • anything AGENTS.md leaves unclear ("when in doubt, leave it out").

7. Definition of Done

A feature is done when:

  • its command(s) are registered with label, menu path, shortcut (if any) and a parameter doc;
  • engine tests cover it, including undo/redo and the disabled case;
  • the UI is wired: menu, panel control and shortcut;
  • every new interactive widget has an automation id;
  • you have driven it through the control channel and reviewed a screenshot;
  • all gates pass (cargo xtask ci), and new assets have sidecars;
  • the crate README and ROADMAP.md are updated where relevant;
  • it is committed with its task id.

Reporting progress honestly

ROADMAP.md keeps two numbers: the feature checklist (does it exist?) and ready for real work (does it hold up?). When you update either:

  • say how a number was obtained: measured (a test, a diff, a benchmark you ran) or estimated;
  • count an approximation, a stub or a partly wired setting as such, never as done;
  • record known bugs and limitations in the crate README and the honest assessment, not only what works;
  • benchmarks state the machine, the load average and before/after on the same build;
  • don't raise a percentage without evidence a reviewer can rerun.