84 lines
5.3 KiB
Markdown
84 lines
5.3 KiB
Markdown
# Transcripts and text-based editing
|
|
|
|
FilmCraft's Text panel ▸ **Transcript** tab shows the dialogue of the open sequence as text. Select
|
|
words to mark In/Out, then extract or lift them; remove filler words and long pauses in one step;
|
|
turn the transcript into captions. Every action is an engine command (`transcript.*`), so the CLI,
|
|
the control channel and MCP agents can do the same.
|
|
|
|
## Model
|
|
|
|
- A **transcript** belongs to a media item (`Project::transcripts`, saved in the `.fcproj` since
|
|
schema v9). It lists **words** with media-time bounds (`Tick`s), an optional speaker index and a
|
|
confidence, plus the speaker names and the language. Because the times are media time, the
|
|
transcript stays valid however the clip is trimmed, moved, sped up or reused.
|
|
- The **sequence transcript** is derived, never stored (`filmcraft_edit::transcript::sequence_words`):
|
|
audio tracks are read top first; a word is heard through the first enabled clip whose range
|
|
covers the word's midpoint (duplicates of the same dialogue on lower tracks read once); disabled,
|
|
reversed and frame-hold clips contribute nothing.
|
|
- Speaker names come from the clip transcripts, so renaming "Speaker 1" in every transcript renames
|
|
it across the sequence.
|
|
|
|
## Commands
|
|
|
|
| Command | What it does |
|
|
|---|---|
|
|
| `transcript.generate` | Transcribe media items (`items`, else the Project selection, else the media of the sequence's audio clips). Params: `model` (default `whisper-base`), `language` (`auto` = detect), `diarize`, `maxSpeakers`. One undo step. |
|
|
| `transcript.set` | Store a transcript you bring (JSON: `language`, `speakers`, `words` with `text`/`start`/`end`/`speaker`); it is sorted and made well formed. |
|
|
| `transcript.delete` | Remove transcripts. |
|
|
| `transcript.inspect` | The sequence transcript: words (index, text, sequence times, speaker, clip), paragraphs, speakers, the word at the playhead. |
|
|
| `transcript.search` | Word-index ranges matching a phrase (case and punctuation ignored; the last word may be a prefix). |
|
|
| `transcript.select` | Mark In/Out around words `from..=to` (frame-snapped outward) and move the playhead there. |
|
|
| `transcript.extract` / `transcript.lift` | Extract (ripple) or lift the words' frames on the targeted tracks. |
|
|
| `transcript.renameSpeaker` | Rename a speaker by name (every transcript) or by index in one `item`. |
|
|
| `transcript.removeFillers` | Ripple-delete filler words (`fillers`, default um/uh/erm/…; phrases such as "you know" allowed). |
|
|
| `transcript.removePauses` | Ripple-delete pauses longer than `minSeconds`, keeping `keepSeconds` of air on both sides. |
|
|
| `transcript.createCaptions` | Lay the words out as captions on a new caption track (`maxChars`, `lines`, `minSeconds`, `maxSeconds`, `gapFrames`). |
|
|
| `transcript.models` / `transcript.downloadModel` | List the speech models (size, licence, installed) / download one. |
|
|
|
|
## Speech recognition
|
|
|
|
Recognition goes through the `Transcriber` trait (`crates/speech`). The built-in recogniser is
|
|
OpenAI's Whisper, run in pure Rust on [candle](https://github.com/huggingface/candle) on the CPU,
|
|
with timestamp decoding, language detection and word times from cross-attention alignment (see the
|
|
`filmcraft_speech::whisper` module docs). Speakers are labelled by clustering per-chunk MFCC
|
|
statistics (`filmcraft_speech::diarize`); no model is involved.
|
|
|
|
Both are **optional features**, off by default and never built for the web:
|
|
|
|
- `whisper` (on `filmcraft-speech`, `filmcraft-engine`, and the `filmcraft` / `filmcraft-cli`
|
|
apps, where it also enables downloads): candle inference.
|
|
- `download` (`speech-download` on the engine): HTTPS downloads with rustls + RustCrypto and the
|
|
operating system's certificate verifier.
|
|
|
|
Without `whisper`, `transcript.generate` fails with "speech-to-text is not available in this build";
|
|
transcripts can still be imported with `transcript.set` and edited with every other command. Hosts
|
|
and tests can install any recogniser in `Session::transcriber`.
|
|
|
|
### Models
|
|
|
|
Weights are **never** bundled or committed. They are downloaded on request into
|
|
`<data dir>/models/<id>/` (see `filmcraft_engine::autosave::default_data_dir`), each file pinned to a
|
|
revision of OpenAI's Hugging Face repositories and checked against its SHA-256:
|
|
|
|
| Id | Languages | Licence |
|
|
|---|---|---|
|
|
| `whisper-tiny` | multilingual | MIT (OpenAI); HF conversion Apache-2.0 |
|
|
| `whisper-base` (default) | multilingual | MIT (OpenAI); HF conversion Apache-2.0 |
|
|
| `whisper-small` | multilingual | MIT (OpenAI); HF conversion Apache-2.0 |
|
|
|
|
### Testing
|
|
|
|
Unit and engine tests use a fake `Transcriber` (`FixedTranscriber`), so CI needs no model. The
|
|
end-to-end test `crates/speech/tests/whisper_model.rs` (feature `whisper`) runs only when weights are
|
|
in `target/models/<id>/` (or `$FILMCRAFT_MODELS_DIR`) and speech samples (mono 16 kHz f32 with a
|
|
reference `.txt`) are in `target/fixtures/speech/`; it reports the word error rate and otherwise
|
|
prints SKIPPED. Measured on 2026-10-01: `whisper-tiny`, English, 12.2 % WER over 797 words of
|
|
the local speech samples (LibriSpeech read speech and dialogue clips) (about 9.5 minutes for the run in a release build on an
|
|
Apple-silicon laptop CPU).
|
|
|
|
## Limits
|
|
|
|
- Transcription runs synchronously inside the command (no background job or progress bar yet).
|
|
- Track items that refer to a subclip are looked up by the subclip's id, so a transcript made for
|
|
the parent media is not shown through subclip clips yet.
|