M7.9: Text to Speech narrations (panel, tts.* commands, built-in voice)
Write what a narrator says, pick a voice, pitch and pace, and add it to
the timeline. Selecting a narration clip loads it back into the panel;
Save regenerates it in place as one undo step, keeping the clip's start
and length (longer speech kept whole in the file, shorter speech padded
with silence).
- filmcraft-tts (L2): Voice trait, voice catalogue, pause markers
([pause 1s]), and an original formant synthesizer (Basic Female/Male;
robotic placeholder until the neural voices land).
- project: Narration + Project::narrations (schema v13, no-op migration).
- engine: tts.voices/preview/create/edit/inspect; narrations never
overwrite clips (free targeted track, free track, else a new track).
- ui: Text to Speech panel beside Properties, preview playback,
click-to-edit, automation ids.
Gates: fmt, clippy -D warnings, layers, assets and wasm (rustup
toolchain) pass. Tests: all pass except three that fail identically on
main (61238bf) on this machine: shortcuts_tests::assign_reassigns_
conflicts_and_undoes, gpu fx_tests::gpu_effects_match_cpu_exactly, and
the frame pool test, which passes when run alone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Manuel Camacho
parent
207b179428
commit
66ba5bbb3c
|
|
@ -1874,6 +1874,7 @@ dependencies = [
|
|||
"filmcraft-testkit",
|
||||
"filmcraft-text",
|
||||
"filmcraft-time",
|
||||
"filmcraft-tts",
|
||||
"image",
|
||||
"log",
|
||||
"serde",
|
||||
|
|
@ -2212,6 +2213,14 @@ dependencies = [
|
|||
"serde",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "filmcraft-tts"
|
||||
version = "0.2.1"
|
||||
dependencies = [
|
||||
"serde",
|
||||
"thiserror 2.0.21",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "filmcraft-ui-egui"
|
||||
version = "0.3.0"
|
||||
|
|
@ -2236,6 +2245,7 @@ dependencies = [
|
|||
"filmcraft-scopes",
|
||||
"filmcraft-text",
|
||||
"filmcraft-time",
|
||||
"filmcraft-tts",
|
||||
"image",
|
||||
"log",
|
||||
"pollster 0.4.0",
|
||||
|
|
|
|||
|
|
@ -44,6 +44,7 @@ filmcraft-interchange = { path = "crates/interchange" }
|
|||
filmcraft-text = { path = "crates/text" }
|
||||
filmcraft-captions = { path = "crates/captions" }
|
||||
filmcraft-speech = { path = "crates/speech" }
|
||||
filmcraft-tts = { path = "crates/tts" }
|
||||
filmcraft-render = { path = "crates/render" }
|
||||
filmcraft-gpu = { path = "crates/gpu" }
|
||||
filmcraft-scopes = { path = "crates/scopes" }
|
||||
|
|
|
|||
|
|
@ -35,6 +35,7 @@ filmcraft-format = { workspace = true }
|
|||
filmcraft-interchange = { workspace = true }
|
||||
filmcraft-captions = { workspace = true }
|
||||
filmcraft-speech = { workspace = true }
|
||||
filmcraft-tts = { workspace = true }
|
||||
serde = { workspace = true }
|
||||
serde_json = { workspace = true }
|
||||
thiserror = { workspace = true }
|
||||
|
|
|
|||
|
|
@ -2510,6 +2510,7 @@ fn build() -> Vec<CommandSpec> {
|
|||
v.extend(crate::scopes::commands());
|
||||
v.extend(crate::remix::commands());
|
||||
v.extend(crate::voiceover::commands());
|
||||
v.extend(crate::narration::commands());
|
||||
v.extend(crate::keyboard::commands());
|
||||
v.extend(crate::project_panel::commands());
|
||||
v.extend(crate::media_browser::commands());
|
||||
|
|
|
|||
|
|
@ -28,6 +28,7 @@ pub mod media_browser;
|
|||
pub mod media_pool;
|
||||
pub mod mixer;
|
||||
pub mod multicam;
|
||||
pub mod narration;
|
||||
pub mod panels;
|
||||
pub mod perf;
|
||||
pub mod presets;
|
||||
|
|
@ -347,6 +348,8 @@ pub struct Session {
|
|||
pub mcrec: multicam::Recorder,
|
||||
/// Voice-over recording: the input device and the take in progress.
|
||||
pub voiceover: voiceover::VoiceOver,
|
||||
/// The last `tts.preview` result, for the host to play (Text to Speech ▸ Preview).
|
||||
pub tts_preview: Option<Arc<filmcraft_tts::Audio>>,
|
||||
/// Dynamic (J/K/L) trimming and trim-mode loop playback in progress.
|
||||
pub trim_play: trim::TrimPlayback,
|
||||
/// Keyboard shortcuts (active bindings, presets; `shortcuts.*` commands).
|
||||
|
|
@ -449,6 +452,7 @@ impl Session {
|
|||
mixrec: Default::default(),
|
||||
mcrec: Default::default(),
|
||||
voiceover: Default::default(),
|
||||
tts_preview: None,
|
||||
trim_play: Default::default(),
|
||||
shortcuts: shortcuts::Shortcuts::new(),
|
||||
offline: Default::default(),
|
||||
|
|
@ -1054,6 +1058,8 @@ mod mixer_tests;
|
|||
#[cfg(test)]
|
||||
mod multicam_tests;
|
||||
#[cfg(test)]
|
||||
mod narration_tests;
|
||||
#[cfg(test)]
|
||||
mod nest_editing_tests;
|
||||
#[cfg(test)]
|
||||
mod nest_fidelity_tests;
|
||||
|
|
|
|||
|
|
@ -0,0 +1,360 @@
|
|||
//! Text to Speech (narrations): write what the narrator says, pick a voice, and get an audio clip.
|
||||
//!
|
||||
//! | command | does |
|
||||
//! |---|---|
|
||||
//! | `tts.voices` | languages, voices, vocal pitches, pace limits and the pause marker |
|
||||
//! | `tts.preview` | synthesize without changing the project (`sample: true` speaks the voice's sample sentence); the host plays [`Session::tts_preview`]; `path` also writes a WAV |
|
||||
//! | `tts.create` | synthesize, write `Narration <n>.wav`, import it and place it at the playhead (one undo step) |
|
||||
//! | `tts.edit` | change a narration clip's script or voice settings and regenerate it in place (one undo step) |
|
||||
//! | `tts.inspect` | the narration settings of a clip or item |
|
||||
//!
|
||||
//! **Where a new narration goes.** At `time` (default: the playhead) on `track` if given (it must
|
||||
//! be free there), else on the first targeted audio track that is free for the narration's length,
|
||||
//! else the first free audio track, else on a new audio track. Existing clips are never overwritten.
|
||||
//!
|
||||
//! **Editing keeps the clip's length** (user decision, 2026-10-07). The clip keeps its start,
|
||||
//! duration and speed. When the new speech is longer than the clip, the clip still ends where it
|
||||
//! did but the file holds the whole speech, so trimming the clip's end out reveals the rest. When it
|
||||
//! is shorter, the file is padded with silence to the clip's length. The clip's source In is reset
|
||||
//! to the start of the new speech. Other clips (including other clips of the old narration) are not
|
||||
//! changed. Each edit writes a new file; the old item and its narration stay, so undo is exact.
|
||||
//!
|
||||
//! Files are named `Narration <n>.wav` (mono 32-bit float at the voice's rate, 24 kHz) and saved
|
||||
//! where voice-over recordings go (Scratch Disks ▸ Captured, else next to the project, else
|
||||
//! `Narrations` in the data or temporary directory); an edit writes next to the previous file.
|
||||
|
||||
use std::sync::Arc;
|
||||
|
||||
use serde_json::{Value, json};
|
||||
|
||||
use filmcraft_project::{ClipId, ItemId, ItemKind, MediaRef, Narration, TrackId, TrackKind, VocalPitch};
|
||||
use filmcraft_time::{Tick, TimeRange};
|
||||
use filmcraft_tts as tts;
|
||||
|
||||
use crate::commands::{CommandSpec, always, bad, has_seq, str_p, time_p, u64_p};
|
||||
use crate::{EngineError, Result, Session};
|
||||
|
||||
/// Item name prefix of generated narrations.
|
||||
const BASE_NAME: &str = "Narration";
|
||||
|
||||
/// The narration of a project item.
|
||||
pub fn narration_of(s: &Session, item: ItemId) -> Option<&Narration> {
|
||||
s.project.narrations.get(&item)
|
||||
}
|
||||
|
||||
/// The narration clip a command works on: `clip`, else the first selected clip whose item is a
|
||||
/// narration.
|
||||
pub fn target_clip(s: &Session, p: &Value) -> Option<ClipId> {
|
||||
let seq = s.active_sequence()?;
|
||||
let is_narration = |c: ClipId| seq.find_item(c).is_some_and(|(_, it)| s.project.narrations.contains_key(&it.item));
|
||||
match u64_p(p, "clip") {
|
||||
Some(c) => Some(ClipId(c)).filter(|c| is_narration(*c)),
|
||||
None => s.state.selection.iter().copied().find(|c| is_narration(*c)),
|
||||
}
|
||||
}
|
||||
|
||||
fn has_narration_clip(s: &Session) -> std::result::Result<(), String> {
|
||||
has_seq(s)?;
|
||||
target_clip(s, &Value::Null).map(|_| ()).ok_or_else(|| "select a narration clip".into())
|
||||
}
|
||||
|
||||
/// Script and settings from params, on top of `base`.
|
||||
fn settings(cmd: &str, p: &Value, base: Option<&Narration>) -> Result<Narration> {
|
||||
// a setting of the wrong type is an error, never silently replaced by a default
|
||||
for key in ["text", "voice", "pitch"] {
|
||||
if p.get(key).is_some_and(|v| !v.is_string()) {
|
||||
return Err(bad(cmd, format!("`{key}` must be a string")));
|
||||
}
|
||||
}
|
||||
let text = match str_p(p, "text") {
|
||||
Some(t) => t.to_string(),
|
||||
None => base.map(|b| b.text.clone()).ok_or_else(|| bad(cmd, "`text` is required"))?,
|
||||
};
|
||||
let voice = str_p(p, "voice").map(str::to_string).or_else(|| base.map(|b| b.voice.clone())).unwrap_or_else(|| tts::default_voice_id().to_string());
|
||||
let info = tts::voices().into_iter().find(|v| v.id == voice).ok_or_else(|| bad(cmd, format!("unknown voice `{voice}` (see tts.voices)")))?;
|
||||
let pitch = match str_p(p, "pitch") {
|
||||
Some(v) => VocalPitch::from_id(v).ok_or_else(|| bad(cmd, format!("unknown pitch `{v}` (extraLow, low, default, high, extraHigh)")))?,
|
||||
None => base.map(|b| b.pitch).unwrap_or_default(),
|
||||
};
|
||||
let pace = match p.get("pace") {
|
||||
Some(v) => v.as_f64().ok_or_else(|| bad(cmd, "`pace` must be a number"))?,
|
||||
None => base.map_or(1.0, |b| b.pace),
|
||||
};
|
||||
let n = Narration { text, language: info.language.to_string(), voice, pitch, pace, speech_duration: Tick::ZERO };
|
||||
n.check().map_err(|e| bad(cmd, e))?;
|
||||
Ok(n)
|
||||
}
|
||||
|
||||
fn synthesize(cmd: &str, n: &Narration) -> Result<tts::Audio> {
|
||||
let voice = tts::voice(&n.voice).map_err(|e| bad(cmd, e.to_string()))?;
|
||||
let params = tts::Params { semitones: n.pitch.semitones(), pace: n.pace };
|
||||
voice.synthesize(&n.text, ¶ms).map_err(|e| bad(cmd, e.to_string()))
|
||||
}
|
||||
|
||||
fn voices(_s: &mut Session, _p: &Value) -> Result<Value> {
|
||||
Ok(json!({
|
||||
"languages": tts::LANGUAGES.iter().map(|(id, name)| json!({"id": id, "name": name})).collect::<Vec<_>>(),
|
||||
"voices": tts::voices(),
|
||||
"defaultVoice": tts::default_voice_id(),
|
||||
"pitches": VocalPitch::ALL.iter().map(|p| json!({"id": p.id(), "label": p.label(), "semitones": p.semitones()})).collect::<Vec<_>>(),
|
||||
"pace": {"min": tts::MIN_PACE, "max": tts::MAX_PACE, "default": 1.0},
|
||||
"pauseMarker": tts::script::PAUSE_MARKER,
|
||||
"maxTextBytes": tts::MAX_TEXT_BYTES,
|
||||
"sampleSentence": tts::SAMPLE_SENTENCE,
|
||||
}))
|
||||
}
|
||||
|
||||
fn preview(s: &mut Session, p: &Value) -> Result<Value> {
|
||||
let cmd = "tts.preview";
|
||||
let base = target_clip(s, p).and_then(|c| s.active_sequence()?.find_item(c).map(|(_, it)| it.item)).and_then(|i| s.project.narrations.get(&i).cloned());
|
||||
let mut q = p.clone();
|
||||
if q.get("sample").and_then(Value::as_bool).unwrap_or(false) {
|
||||
q["text"] = json!(tts::SAMPLE_SENTENCE);
|
||||
}
|
||||
let n = settings(cmd, &q, base.as_ref())?;
|
||||
let audio = synthesize(cmd, &n)?;
|
||||
let seconds = audio.seconds();
|
||||
let samples = audio.samples.len();
|
||||
let rate = audio.sample_rate;
|
||||
let mut path = Value::Null;
|
||||
if let Some(dest) = str_p(p, "path") {
|
||||
let bytes = crate::voiceover::write_wav_f32(&audio.samples, rate);
|
||||
s.services.write_file(dest, &bytes).map_err(|e| EngineError::Other(format!("{dest}: {e}")))?;
|
||||
path = json!(dest);
|
||||
}
|
||||
s.tts_preview = Some(Arc::new(audio));
|
||||
Ok(json!({"seconds": seconds, "samples": samples, "sampleRate": rate, "voice": n.voice, "path": path}))
|
||||
}
|
||||
|
||||
/// Write `samples` as a new WAV and import it (its own undo step; callers collapse history).
|
||||
/// `near`: an edit writes next to the narration's previous file unless `dir` is given.
|
||||
fn write_and_import(s: &mut Session, p: &Value, samples: &[f32], rate: u32, near: Option<ItemId>) -> Result<(ItemId, String)> {
|
||||
let beside = near.filter(|_| p.get("dir").is_none()).and_then(|i| match &s.project.items.get(&i)?.kind {
|
||||
ItemKind::Media(m) => match &m.media {
|
||||
MediaRef::File { path } => std::path::Path::new(path).parent().filter(|d| !d.as_os_str().is_empty()).map(|d| d.to_string_lossy().into_owned()),
|
||||
MediaRef::Generator(_) => None,
|
||||
},
|
||||
_ => None,
|
||||
});
|
||||
let dir = beside.unwrap_or_else(|| crate::voiceover::media_dir(s, p, "Narrations"));
|
||||
let path = crate::voiceover::unique_wav_path(s, &dir, BASE_NAME);
|
||||
if !cfg!(target_arch = "wasm32") {
|
||||
std::fs::create_dir_all(&dir).map_err(|e| EngineError::Other(format!("{dir}: {e}")))?;
|
||||
}
|
||||
let bytes = crate::voiceover::write_wav_f32(samples, rate);
|
||||
s.services.write_file(&path, &bytes).map_err(|e| EngineError::Other(format!("{path}: {e}")))?;
|
||||
let item = crate::commands::import_bytes(s, &path, bytes.into(), None)?;
|
||||
Ok((item, path))
|
||||
}
|
||||
|
||||
fn track_is_free(s: &Session, track: TrackId, range: TimeRange) -> bool {
|
||||
s.active_sequence()
|
||||
.and_then(|q| q.audio_tracks.iter().find(|t| t.id == track))
|
||||
.is_some_and(|t| !t.locked && !t.items.iter().any(|it| it.start < range.end() && range.start < it.end()))
|
||||
}
|
||||
|
||||
/// The track a new narration of `range` goes to (None: add a track).
|
||||
fn placement(s: &Session, p: &Value, range: TimeRange) -> Result<Option<TrackId>> {
|
||||
let cmd = "tts.create";
|
||||
let seq = s.active_sequence().ok_or(EngineError::NoSequence)?;
|
||||
if let Some(v) = p.get("track") {
|
||||
let id =
|
||||
crate::mixer::strip_ref(seq, v).filter(|id| seq.audio_tracks.iter().any(|t| t.id == *id)).ok_or_else(|| bad(cmd, format!("no audio track {v}")))?;
|
||||
if !track_is_free(s, id, range) {
|
||||
return Err(bad(cmd, format!("audio track {v} is locked or has clips there")));
|
||||
}
|
||||
return Ok(Some(id));
|
||||
}
|
||||
let targeted = s.targeting().targeted;
|
||||
let order = seq.audio_tracks.iter().filter(|t| targeted.contains(&t.id)).chain(seq.audio_tracks.iter().filter(|t| !targeted.contains(&t.id)));
|
||||
Ok(order.map(|t| t.id).find(|id| track_is_free(s, *id, range)))
|
||||
}
|
||||
|
||||
fn create(s: &mut Session, p: &Value) -> Result<Value> {
|
||||
let cmd = "tts.create";
|
||||
let seq_id = s.state.active_sequence.ok_or(EngineError::NoSequence)?;
|
||||
let mut n = settings(cmd, p, None)?;
|
||||
let at = time_p(s, p, "").unwrap_or(s.playhead());
|
||||
if at < Tick::ZERO {
|
||||
return Err(bad(cmd, "the start time cannot be negative"));
|
||||
}
|
||||
let audio = synthesize(cmd, &n)?;
|
||||
let rate = audio.sample_rate.max(1);
|
||||
let dur = Tick::from_units(audio.samples.len() as i64, i64::from(rate));
|
||||
n.speech_duration = dur;
|
||||
let range = TimeRange::new(at, dur);
|
||||
let track = placement(s, p, range)?;
|
||||
let n0 = s.history.undo.len();
|
||||
let result = (|| -> Result<Value> {
|
||||
let track = match track {
|
||||
Some(t) => t,
|
||||
None => {
|
||||
let before: Vec<TrackId> = s.active_sequence().map(|q| q.audio_tracks.iter().map(|t| t.id).collect()).unwrap_or_default();
|
||||
s.execute("sequence.addTracks", json!({"video": 0, "audio": 1, "audioAfter": "last"}))?;
|
||||
s.active_sequence()
|
||||
.and_then(|q| q.audio_tracks.iter().map(|t| t.id).find(|id| !before.contains(id)))
|
||||
.ok_or_else(|| EngineError::Other("could not add an audio track".into()))?
|
||||
}
|
||||
};
|
||||
let (item, path) = write_and_import(s, p, &audio.samples, rate, None)?;
|
||||
let seq_rate = s.project.sequence(seq_id).map(|q| q.settings.frame_rate).unwrap_or_default();
|
||||
let narration = n.clone();
|
||||
let clip = s.edit("New Narration", |pr, st| {
|
||||
let mut it = pr
|
||||
.make_track_item(item, TrackKind::Audio, at, TimeRange::new(Tick::ZERO, dur), seq_rate)
|
||||
.ok_or(EngineError::Other("the narration is not importable".into()))?;
|
||||
it.duration = dur;
|
||||
let id = it.id;
|
||||
let seq = pr.sequence_mut(seq_id).ok_or(EngineError::NoSequence)?;
|
||||
let t = seq.audio_tracks.iter_mut().find(|t| t.id == track).ok_or_else(|| bad(cmd, "the narration track was deleted"))?;
|
||||
let pos = t.items.partition_point(|x| x.start <= at);
|
||||
t.items.insert(pos, it);
|
||||
seq.check().map_err(EngineError::Other)?;
|
||||
pr.narrations.insert(item, narration);
|
||||
st.selection = vec![id];
|
||||
Ok(id)
|
||||
})?;
|
||||
let seq = s.project.sequence(seq_id).ok_or(EngineError::NoSequence)?;
|
||||
Ok(json!({
|
||||
"item": item.0,
|
||||
"clip": clip.0,
|
||||
"track": crate::mixer::strip_label(seq, track),
|
||||
"start": at.0,
|
||||
"duration": dur.0,
|
||||
"speechDuration": dur.0,
|
||||
"path": path,
|
||||
"sampleRate": rate,
|
||||
}))
|
||||
})();
|
||||
crate::clip_ops::collapse_history(s, n0, "New Narration");
|
||||
if result.is_err() && s.history.undo.len() > n0 {
|
||||
// a step failed after the track was added or the file imported: roll back to before
|
||||
let _ = s.undo();
|
||||
s.history.redo.clear();
|
||||
}
|
||||
result
|
||||
}
|
||||
|
||||
/// Samples of media a clip of `duration` at `speed` plays, rounded up (exact for 100 %).
|
||||
fn needed_samples(duration: Tick, speed: f64, rate: u32) -> usize {
|
||||
let rate = i64::from(rate.max(1));
|
||||
let media = if speed.is_finite() && speed != 0.0 && (speed.abs() - 1.0).abs() > 1e-12 {
|
||||
// speed is a ratio, not edit time: scale the exact tick count
|
||||
Tick::from_seconds_f64(duration.seconds() * speed.abs())
|
||||
} else {
|
||||
duration
|
||||
};
|
||||
let mut n = media.to_units_floor(rate);
|
||||
if Tick::from_units(n, rate) < media {
|
||||
n = n.saturating_add(1);
|
||||
}
|
||||
let cap = (tts::MAX_SECONDS * 2.0) as i64 * rate;
|
||||
n.clamp(0, cap) as usize
|
||||
}
|
||||
|
||||
fn edit(s: &mut Session, p: &Value) -> Result<Value> {
|
||||
let cmd = "tts.edit";
|
||||
let seq_id = s.state.active_sequence.ok_or(EngineError::NoSequence)?;
|
||||
let clip = target_clip(s, p).ok_or_else(|| bad(cmd, "select a narration clip (or pass `clip`)"))?;
|
||||
let seq = s.active_sequence().ok_or(EngineError::NoSequence)?;
|
||||
let (track, it) = seq.find_item(clip).ok_or_else(|| bad(cmd, "no such clip"))?;
|
||||
if seq.audio_tracks.iter().any(|t| t.id == track && t.locked) {
|
||||
return Err(bad(cmd, "the clip's track is locked"));
|
||||
}
|
||||
let (old_item, clip_dur, speed) = (it.item, it.duration, it.speed);
|
||||
let base = s.project.narrations.get(&old_item).cloned().ok_or_else(|| bad(cmd, "the clip is not a narration"))?;
|
||||
let mut n = settings(cmd, p, Some(&base))?;
|
||||
let audio = synthesize(cmd, &n)?;
|
||||
let rate = audio.sample_rate.max(1);
|
||||
let speech = audio.samples.len();
|
||||
n.speech_duration = Tick::from_units(speech as i64, i64::from(rate));
|
||||
// the clip keeps its length: pad the file with silence to what the clip plays
|
||||
let needed = needed_samples(clip_dur, speed, rate);
|
||||
let mut samples = audio.samples;
|
||||
if needed > samples.len() {
|
||||
samples.try_reserve(needed - samples.len()).map_err(|e| bad(cmd, format!("unable to allocate the narration: {e}")))?;
|
||||
samples.resize(needed, 0.0);
|
||||
}
|
||||
let file_len = samples.len();
|
||||
let n0 = s.history.undo.len();
|
||||
let result = (|| -> Result<Value> {
|
||||
let (item, path) = write_and_import(s, p, &samples, rate, Some(old_item))?;
|
||||
let name = s.project.items.get(&item).map(|i| i.name.clone()).unwrap_or_default();
|
||||
let narration = n.clone();
|
||||
s.edit("Edit Narration", |pr, _| {
|
||||
let seq = pr.sequence_mut(seq_id).ok_or(EngineError::NoSequence)?;
|
||||
let (_, it) = seq.find_item_mut(clip).ok_or_else(|| bad(cmd, "the clip was deleted"))?;
|
||||
it.item = item;
|
||||
it.name = name;
|
||||
it.source_in = Tick::ZERO;
|
||||
seq.check().map_err(EngineError::Other)?;
|
||||
pr.narrations.insert(item, narration);
|
||||
Ok(())
|
||||
})?;
|
||||
Ok(json!({
|
||||
"clip": clip.0,
|
||||
"item": item.0,
|
||||
"previousItem": old_item.0,
|
||||
"duration": clip_dur.0,
|
||||
"speechDuration": n.speech_duration.0,
|
||||
"fileDuration": Tick::from_units(file_len as i64, i64::from(rate)).0,
|
||||
"path": path,
|
||||
"sampleRate": rate,
|
||||
}))
|
||||
})();
|
||||
crate::clip_ops::collapse_history(s, n0, "Edit Narration");
|
||||
if result.is_err() && s.history.undo.len() > n0 {
|
||||
let _ = s.undo();
|
||||
s.history.redo.clear();
|
||||
}
|
||||
result
|
||||
}
|
||||
|
||||
fn inspect(s: &mut Session, p: &Value) -> Result<Value> {
|
||||
let cmd = "tts.inspect";
|
||||
let (item, clip) = match u64_p(p, "item") {
|
||||
Some(i) => (ItemId(i), None),
|
||||
None => {
|
||||
let c = target_clip(s, p).ok_or_else(|| bad(cmd, "pass `item` or `clip`, or select a narration clip"))?;
|
||||
let seq = s.active_sequence().ok_or(EngineError::NoSequence)?;
|
||||
(seq.find_item(c).map(|(_, it)| it.item).ok_or_else(|| bad(cmd, "no such clip"))?, Some(c.0))
|
||||
}
|
||||
};
|
||||
let n = s.project.narrations.get(&item).ok_or_else(|| bad(cmd, "not a narration"))?;
|
||||
Ok(json!({"item": item.0, "clip": clip, "narration": n}))
|
||||
}
|
||||
|
||||
fn spec(
|
||||
id: &'static str,
|
||||
label: &'static str,
|
||||
params: &'static str,
|
||||
enabled: fn(&Session) -> std::result::Result<(), String>,
|
||||
run: fn(&mut Session, &Value) -> Result<Value>,
|
||||
journal: bool,
|
||||
) -> CommandSpec {
|
||||
CommandSpec { id, label, menu: &[], shortcut: None, params, enabled, run, journal }
|
||||
}
|
||||
|
||||
pub fn commands() -> Vec<CommandSpec> {
|
||||
vec![
|
||||
spec("tts.voices", "List Narration Voices", "{}", always, voices, false),
|
||||
spec(
|
||||
"tts.preview",
|
||||
"Preview Narration",
|
||||
r#"{"text":str?,"voice":str?,"pitch":"default"?,"pace":f64?,"sample":bool?,"clip":id?,"path":str?}"#,
|
||||
always,
|
||||
preview,
|
||||
false,
|
||||
),
|
||||
spec(
|
||||
"tts.create",
|
||||
"New Narration",
|
||||
r#"{"text":str,"voice":str?,"pitch":"extraLow|low|default|high|extraHigh"?,"pace":0.5..2?,"time":ticks?,"track":"A1"|id?,"dir":str?}"#,
|
||||
has_seq,
|
||||
create,
|
||||
true,
|
||||
),
|
||||
spec("tts.edit", "Edit Narration", r#"{"clip":id?,"text":str?,"voice":str?,"pitch":str?,"pace":f64?,"dir":str?}"#, has_narration_clip, edit, true),
|
||||
spec("tts.inspect", "Inspect Narration", r#"{"clip":id?,"item":id?}"#, always, inspect, false),
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,349 @@
|
|||
//! Tests of [`crate::narration`]: Text to Speech narrations are placed without overwriting, keep
|
||||
//! their clip length when edited, and are one exact undo step.
|
||||
|
||||
use super::*;
|
||||
use filmcraft_project::{ClipId, ItemId};
|
||||
use serde_json::json;
|
||||
|
||||
const RATE: i64 = filmcraft_tts::SAMPLE_RATE as i64;
|
||||
|
||||
fn tmp(name: &str) -> String {
|
||||
let d = std::env::temp_dir().join(format!("filmcraft-tts-{}-{name}", std::process::id()));
|
||||
let _ = std::fs::remove_dir_all(&d);
|
||||
d.to_string_lossy().into_owned()
|
||||
}
|
||||
|
||||
fn demo() -> Session {
|
||||
let mut s = Session::default();
|
||||
s.execute("file.openDemoProject", json!({})).unwrap();
|
||||
s
|
||||
}
|
||||
|
||||
/// The 32-bit float samples of a WAV written by `write_wav_f32`.
|
||||
fn wav_samples(path: &str) -> Vec<f32> {
|
||||
let b = std::fs::read(path).unwrap();
|
||||
b[44..].as_chunks::<4>().0.iter().map(|c| f32::from_le_bytes(*c)).collect()
|
||||
}
|
||||
|
||||
fn clip(s: &Session, id: u64) -> filmcraft_project::TrackItem {
|
||||
s.active_sequence().unwrap().find_item(ClipId(id)).unwrap().1.clone()
|
||||
}
|
||||
|
||||
fn all_clips(s: &Session) -> Vec<filmcraft_project::TrackItem> {
|
||||
s.active_sequence().unwrap().all_tracks().flat_map(|t| t.items.clone()).collect()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn voices_lists_the_built_in_voices() {
|
||||
let mut s = Session::default();
|
||||
let r = s.execute("tts.voices", json!({})).unwrap();
|
||||
let ids: Vec<&str> = r["voices"].as_array().unwrap().iter().map(|v| v["id"].as_str().unwrap()).collect();
|
||||
assert!(ids.contains(&"basic-female") && ids.contains(&"basic-male"), "{ids:?}");
|
||||
assert_eq!(r["languages"][0]["id"], "en-US");
|
||||
assert_eq!(r["pitches"].as_array().unwrap().len(), 5);
|
||||
assert_eq!(r["pauseMarker"], "[pause 1s]");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn create_places_the_narration_at_the_playhead_as_one_undo_step() {
|
||||
let mut s = demo();
|
||||
let dir = tmp("create");
|
||||
s.execute("playhead.set", json!({"seconds": 2.0})).unwrap();
|
||||
let t0 = s.playhead();
|
||||
assert!(t0 > Tick::ZERO);
|
||||
let before = all_clips(&s);
|
||||
let undo0 = s.history.undo.len();
|
||||
let r = s.execute("tts.create", json!({"text": "Welcome to FilmCraft.", "voice": "basic-male", "pitch": "low", "pace": 1.2, "dir": dir})).unwrap();
|
||||
let c = clip(&s, r["clip"].as_u64().unwrap());
|
||||
assert_eq!(c.start, t0);
|
||||
// the clip is exactly as long as the speech (sample-exact) and plays it from the start
|
||||
let samples = wav_samples(r["path"].as_str().unwrap());
|
||||
assert_eq!(c.duration, Tick::from_units(samples.len() as i64, RATE));
|
||||
assert_eq!(c.source_in, Tick::ZERO);
|
||||
assert_eq!(r["speechDuration"], c.duration.0);
|
||||
let n = s.project.narrations.get(&ItemId(r["item"].as_u64().unwrap())).unwrap();
|
||||
assert_eq!((n.text.as_str(), n.voice.as_str(), n.pitch, n.pace), ("Welcome to FilmCraft.", "basic-male", VocalPitch::Low, 1.2));
|
||||
assert_eq!(n.speech_duration, c.duration);
|
||||
assert_eq!(s.state.selection, vec![c.id]);
|
||||
// nothing that was there before moved or changed
|
||||
let after = all_clips(&s);
|
||||
for b in &before {
|
||||
assert!(after.contains(b), "clip {:?} changed", b.id);
|
||||
}
|
||||
assert_eq!(after.len(), before.len() + 1);
|
||||
// one undo step that removes the clip and the narration; redo brings both back
|
||||
assert_eq!(s.history.undo.len(), undo0 + 1);
|
||||
assert_eq!(s.history.undo.last().unwrap().0, "New Narration");
|
||||
s.execute("edit.undo", json!({})).unwrap();
|
||||
assert_eq!(all_clips(&s), before);
|
||||
assert!(s.project.narrations.is_empty());
|
||||
s.execute("edit.redo", json!({})).unwrap();
|
||||
assert_eq!(clip(&s, c.id.0), c);
|
||||
assert_eq!(s.project.narrations.len(), 1);
|
||||
let _ = std::fs::remove_dir_all(&dir);
|
||||
}
|
||||
|
||||
use filmcraft_project::VocalPitch;
|
||||
use filmcraft_time::Tick;
|
||||
|
||||
#[test]
|
||||
fn create_never_overwrites_and_adds_a_track_when_every_track_is_busy() {
|
||||
let mut s = demo();
|
||||
let dir = tmp("busy");
|
||||
// fill every audio track over 0–60 s with a narration-length clip so none is free at 0
|
||||
let tracks = s.active_sequence().unwrap().audio_tracks.len();
|
||||
let first = s.execute("tts.create", json!({"text": "One two three.", "time": 0, "dir": dir})).unwrap();
|
||||
let busy_len = Tick(first["duration"].as_i64().unwrap());
|
||||
for _ in 1..tracks + 2 {
|
||||
s.execute("tts.create", json!({"text": "One two three.", "time": 0, "dir": dir})).unwrap();
|
||||
}
|
||||
// every create landed on its own track at 0: no two narrations overlap on one track
|
||||
let seq = s.active_sequence().unwrap();
|
||||
for t in &seq.audio_tracks {
|
||||
let mut items: Vec<_> = t.items.iter().map(|i| (i.start, i.end())).collect();
|
||||
items.sort();
|
||||
for w in items.windows(2) {
|
||||
assert!(w[0].1 <= w[1].0, "overlap on {:?}", t.name);
|
||||
}
|
||||
}
|
||||
assert!(seq.audio_tracks.len() > tracks, "a new audio track was added");
|
||||
assert!(busy_len > Tick::ZERO);
|
||||
let _ = std::fs::remove_dir_all(&dir);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn create_on_an_occupied_or_unknown_track_is_refused_and_changes_nothing() {
|
||||
let mut s = demo();
|
||||
let dir = tmp("occupied");
|
||||
let r = s.execute("tts.create", json!({"text": "Hello.", "time": 0, "track": "A1", "dir": dir}));
|
||||
if s.active_sequence().unwrap().audio_tracks[0].items.iter().any(|i| i.start <= Tick::ZERO && Tick::ZERO < i.end()) {
|
||||
assert!(r.is_err(), "A1 is busy at 0");
|
||||
}
|
||||
let undo0 = s.history.undo.len();
|
||||
let items0 = s.project.items.len();
|
||||
assert!(s.execute("tts.create", json!({"text": "Hello.", "track": "A99", "dir": dir})).is_err());
|
||||
s.execute("timeline.setTrack", json!({"track": "A2", "locked": true})).unwrap();
|
||||
let undo1 = s.history.undo.len();
|
||||
assert!(s.execute("tts.create", json!({"text": "Hello.", "track": "A2", "time": 0, "dir": dir})).is_err());
|
||||
assert_eq!(s.history.undo.len(), undo1);
|
||||
assert!(undo1 >= undo0);
|
||||
assert_eq!(s.project.items.len(), items0, "no file was imported");
|
||||
let _ = std::fs::remove_dir_all(&dir);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn editing_to_longer_speech_keeps_the_clip_length_and_the_whole_speech() {
|
||||
let mut s = demo();
|
||||
let dir = tmp("longer");
|
||||
let r = s.execute("tts.create", json!({"text": "Hi.", "time": 0, "dir": dir})).unwrap();
|
||||
let c0 = clip(&s, r["clip"].as_u64().unwrap());
|
||||
let undo0 = s.history.undo.len();
|
||||
let e = s.execute("tts.edit", json!({"clip": c0.id.0, "text": "Hi. This narration is now a great deal longer than it was before."})).unwrap();
|
||||
let c1 = clip(&s, c0.id.0);
|
||||
assert_eq!((c1.id, c1.start, c1.duration, c1.speed), (c0.id, c0.start, c0.duration, c0.speed), "position and length kept");
|
||||
assert_ne!(c1.item, c0.item, "a new file");
|
||||
assert_eq!(c1.source_in, Tick::ZERO);
|
||||
let n = s.project.narrations.get(&c1.item).unwrap();
|
||||
assert!(n.speech_duration > c1.duration, "the speech is longer than the clip");
|
||||
// the file holds the whole speech, nothing padded
|
||||
let samples = wav_samples(e["path"].as_str().unwrap());
|
||||
assert_eq!(Tick::from_units(samples.len() as i64, RATE), n.speech_duration);
|
||||
assert_eq!(e["fileDuration"], n.speech_duration.0);
|
||||
// the old item and its narration are untouched (other clips of it, undo)
|
||||
assert_eq!(s.project.narrations.get(&c0.item).unwrap().text, "Hi.");
|
||||
assert_eq!(s.history.undo.len(), undo0 + 1);
|
||||
assert_eq!(s.history.undo.last().unwrap().0, "Edit Narration");
|
||||
// the edited file sits next to the first one
|
||||
assert!(e["path"].as_str().unwrap().starts_with(&dir), "{}", e["path"]);
|
||||
let _ = std::fs::remove_dir_all(&dir);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn editing_to_shorter_speech_pads_the_file_with_exact_silence() {
|
||||
let mut s = demo();
|
||||
let dir = tmp("shorter");
|
||||
let r = s.execute("tts.create", json!({"text": "This first version of the narration is fairly long, with several words.", "time": 0, "dir": dir})).unwrap();
|
||||
let id = r["clip"].as_u64().unwrap();
|
||||
let c0 = clip(&s, id);
|
||||
let e = s.execute("tts.edit", json!({"clip": id, "text": "Short."})).unwrap();
|
||||
let c1 = clip(&s, id);
|
||||
assert_eq!((c1.start, c1.duration), (c0.start, c0.duration));
|
||||
let n = s.project.narrations.get(&c1.item).unwrap().clone();
|
||||
assert!(n.speech_duration < c1.duration);
|
||||
let samples = wav_samples(e["path"].as_str().unwrap());
|
||||
let speech = n.speech_duration.to_units_floor(RATE) as usize;
|
||||
// the file covers the clip, and everything after the speech is exactly zero
|
||||
assert!(Tick::from_units(samples.len() as i64, RATE) >= c1.duration);
|
||||
assert!(samples.len() > speech);
|
||||
assert!(samples[speech..].iter().all(|&x| x == 0.0));
|
||||
assert!(samples[..speech].iter().any(|&x| x != 0.0));
|
||||
let _ = std::fs::remove_dir_all(&dir);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn undo_and_redo_of_an_edit_are_exact() {
|
||||
let mut s = demo();
|
||||
let dir = tmp("undo");
|
||||
let r = s.execute("tts.create", json!({"text": "Before.", "time": 0, "dir": dir})).unwrap();
|
||||
let id = r["clip"].as_u64().unwrap();
|
||||
let before = (*s.project).clone();
|
||||
s.execute("tts.edit", json!({"clip": id, "text": "After.", "voice": "basic-male", "pitch": "extraHigh", "pace": 0.7})).unwrap();
|
||||
let after = (*s.project).clone();
|
||||
assert_ne!(before, after);
|
||||
s.execute("edit.undo", json!({})).unwrap();
|
||||
assert_eq!(*s.project, before);
|
||||
let n = s.project.narrations.get(&clip(&s, id).item).unwrap();
|
||||
assert_eq!((n.text.as_str(), n.voice.as_str()), ("Before.", "basic-female"));
|
||||
s.execute("edit.redo", json!({})).unwrap();
|
||||
assert_eq!(*s.project, after);
|
||||
let n = s.project.narrations.get(&clip(&s, id).item).unwrap();
|
||||
assert_eq!((n.text.as_str(), n.pitch, n.pace), ("After.", VocalPitch::ExtraHigh, 0.7));
|
||||
let _ = std::fs::remove_dir_all(&dir);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn edit_without_changes_keeps_settings_and_settings_inherit() {
|
||||
let mut s = demo();
|
||||
let dir = tmp("inherit");
|
||||
let r = s.execute("tts.create", json!({"text": "Keep me.", "voice": "basic-male", "pitch": "high", "pace": 1.5, "time": 0, "dir": dir})).unwrap();
|
||||
let id = r["clip"].as_u64().unwrap();
|
||||
s.execute("tts.edit", json!({"clip": id, "pace": 0.8})).unwrap();
|
||||
let n = s.project.narrations.get(&clip(&s, id).item).unwrap();
|
||||
assert_eq!((n.text.as_str(), n.voice.as_str(), n.pitch, n.pace), ("Keep me.", "basic-male", VocalPitch::High, 0.8));
|
||||
let i = s.execute("tts.inspect", json!({"clip": id})).unwrap();
|
||||
assert_eq!(i["narration"]["pace"], 0.8);
|
||||
assert_eq!(i["narration"]["pitch"], "high");
|
||||
let _ = std::fs::remove_dir_all(&dir);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn edit_is_disabled_without_a_narration_clip() {
|
||||
let mut s = demo();
|
||||
let dir = tmp("enabled");
|
||||
s.state.selection.clear();
|
||||
assert!(!s.is_enabled("tts.edit"));
|
||||
assert!(s.execute("tts.edit", json!({"text": "x"})).is_err());
|
||||
// a non-narration clip is refused
|
||||
let other = s.active_sequence().unwrap().all_tracks().find_map(|t| t.items.first().map(|i| i.id)).unwrap();
|
||||
assert!(s.execute("tts.edit", json!({"clip": other.0, "text": "x"})).is_err());
|
||||
let r = s.execute("tts.create", json!({"text": "Enable me.", "time": 0, "dir": dir})).unwrap();
|
||||
assert!(s.is_enabled("tts.edit"), "the new narration is selected");
|
||||
s.state.selection.clear();
|
||||
assert!(s.execute("tts.edit", json!({"clip": r["clip"], "text": "By id."})).is_ok());
|
||||
let _ = std::fs::remove_dir_all(&dir);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn edit_on_a_locked_track_is_refused() {
|
||||
let mut s = demo();
|
||||
let dir = tmp("locked");
|
||||
let r = s.execute("tts.create", json!({"text": "Locked.", "time": 0, "dir": dir})).unwrap();
|
||||
let track = r["track"].as_str().unwrap().to_string();
|
||||
s.execute("timeline.setTrack", json!({"track": track, "locked": true})).unwrap();
|
||||
let snap = (*s.project).clone();
|
||||
assert!(s.execute("tts.edit", json!({"clip": r["clip"], "text": "Changed."})).is_err());
|
||||
assert_eq!(*s.project, snap);
|
||||
let _ = std::fs::remove_dir_all(&dir);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn hostile_parameters_are_refused_without_changes() {
|
||||
let mut s = demo();
|
||||
let dir = tmp("hostile");
|
||||
let snap = (*s.project).clone();
|
||||
let undo0 = s.history.undo.len();
|
||||
let big = "word ".repeat(300_000);
|
||||
let cases = [
|
||||
json!({}),
|
||||
json!({"text": ""}),
|
||||
json!({"text": " [pause 2s] "}),
|
||||
json!({"text": "👍"}),
|
||||
json!({"text": 42}),
|
||||
json!({"text": big}),
|
||||
json!({"text": "hi", "voice": "nope"}),
|
||||
json!({"text": "hi", "voice": 7}),
|
||||
json!({"text": "hi", "pitch": "loud"}),
|
||||
json!({"text": "hi", "pace": "fast"}),
|
||||
json!({"text": "hi", "pace": 5.0}),
|
||||
json!({"text": "hi", "pace": -1}),
|
||||
json!({"text": "hi", "pace": 1e308}),
|
||||
json!({"text": "hi", "time": -5}),
|
||||
json!({"text": "hi", "track": {"x": 1}}),
|
||||
json!({"text": "hi", "track": "V1"}),
|
||||
];
|
||||
for p in cases {
|
||||
let mut q = p.clone();
|
||||
q["dir"] = json!(dir);
|
||||
let r = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| s.execute("tts.create", q.clone())));
|
||||
assert!(r.is_ok(), "panicked on {p}");
|
||||
assert!(r.unwrap().is_err(), "accepted {p}");
|
||||
assert_eq!(*s.project, snap, "changed by {p}");
|
||||
assert_eq!(s.history.undo.len(), undo0);
|
||||
}
|
||||
for p in [json!({"clip": 999_999_999}), json!({"clip": "x"}), json!({"clip": -1})] {
|
||||
assert!(s.execute("tts.edit", p.clone()).is_err(), "{p}");
|
||||
assert!(s.execute("tts.inspect", p.clone()).is_err(), "{p}");
|
||||
}
|
||||
assert!(s.execute("tts.preview", json!({"text": "hi", "pace": 9})).is_err());
|
||||
// no sequence open
|
||||
let mut empty = Session::default();
|
||||
assert!(empty.execute("tts.create", json!({"text": "hi", "dir": dir})).is_err());
|
||||
let _ = std::fs::remove_dir_all(&dir);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn preview_changes_nothing_and_hands_the_audio_to_the_host() {
|
||||
let mut s = demo();
|
||||
let dir = tmp("preview");
|
||||
std::fs::create_dir_all(&dir).unwrap();
|
||||
let snap = (*s.project).clone();
|
||||
let undo0 = s.history.undo.len();
|
||||
let path = format!("{dir}/p.wav");
|
||||
let r = s.execute("tts.preview", json!({"text": "Preview me.", "path": path})).unwrap();
|
||||
assert_eq!(*s.project, snap);
|
||||
assert_eq!(s.history.undo.len(), undo0);
|
||||
let a = s.tts_preview.clone().unwrap();
|
||||
assert_eq!(r["samples"], a.samples.len());
|
||||
assert_eq!(wav_samples(&path), a.samples);
|
||||
// the sample sentence for "Hear this voice"
|
||||
let r = s.execute("tts.preview", json!({"sample": true, "voice": "basic-male"})).unwrap();
|
||||
assert_eq!(r["voice"], "basic-male");
|
||||
assert!(r["seconds"].as_f64().unwrap() > 1.0);
|
||||
let _ = std::fs::remove_dir_all(&dir);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn narrations_survive_save_and_reload() {
|
||||
let mut s = demo();
|
||||
let dir = tmp("save");
|
||||
let r = s.execute("tts.create", json!({"text": "Saved [pause 500ms] and loaded.", "pitch": "extraLow", "time": 0, "dir": dir})).unwrap();
|
||||
let bytes = filmcraft_format::encode(&s.project, false);
|
||||
let loaded = filmcraft_format::decode(&bytes).unwrap();
|
||||
assert_eq!(loaded.project.narrations, s.project.narrations);
|
||||
let n = loaded.project.narrations.get(&ItemId(r["item"].as_u64().unwrap())).unwrap();
|
||||
assert_eq!((n.text.as_str(), n.pitch), ("Saved [pause 500ms] and loaded.", VocalPitch::ExtraLow));
|
||||
// a project without narrations writes no `narrations` key (older files are unchanged)
|
||||
let plain = String::from_utf8(filmcraft_format::encode(&demo().project, false)).unwrap();
|
||||
assert!(!plain.contains("narrations"));
|
||||
let _ = std::fs::remove_dir_all(&dir);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_narration_is_heard_in_the_mix() {
|
||||
let mut s = demo();
|
||||
let dir = tmp("mix");
|
||||
let r = s.execute("tts.create", json!({"text": "Can you hear me now?", "time": 0, "dir": dir})).unwrap();
|
||||
let track = r["track"].as_str().unwrap().to_string();
|
||||
let n = s.active_sequence().unwrap().audio_tracks.len();
|
||||
for i in 1..=n {
|
||||
if format!("A{i}") != track {
|
||||
s.execute("mixer.setStrip", json!({"strip": format!("A{i}"), "muted": true})).unwrap();
|
||||
}
|
||||
}
|
||||
let provider = s.media.provider(s.project.clone(), s.services.clone());
|
||||
let sr = s.active_sequence().unwrap().settings.sample_rate as usize;
|
||||
let mix = filmcraft_render::audio::mix_sequence(&s.project, s.active_sequence().unwrap(), 0, sr, &provider).channels;
|
||||
let rms = (mix[0].iter().map(|x| f64::from(x * x)).sum::<f64>() / sr as f64).sqrt();
|
||||
assert!(rms > 0.01, "narration audible, rms {rms}");
|
||||
let _ = std::fs::remove_dir_all(&dir);
|
||||
}
|
||||
|
|
@ -386,6 +386,13 @@ fn sync(s: &mut Session, p: &Value) -> Result<Value> {
|
|||
|
||||
/// Directory recordings are written to.
|
||||
fn record_dir(s: &Session, p: &Value) -> String {
|
||||
media_dir(s, p, "Voice-over Recordings")
|
||||
}
|
||||
|
||||
/// Directory generated media (recordings, narrations) is written to: `dir`, else Project
|
||||
/// Settings ▸ Scratch Disks ▸ Captured, else next to the project, else `fallback` in the data or
|
||||
/// temporary directory.
|
||||
pub(crate) fn media_dir(s: &Session, p: &Value, fallback: &str) -> String {
|
||||
if let Some(d) = str_p(p, "dir") {
|
||||
return d.to_string();
|
||||
}
|
||||
|
|
@ -396,7 +403,21 @@ fn record_dir(s: &Session, p: &Value) -> String {
|
|||
return d.to_string_lossy().into_owned();
|
||||
}
|
||||
let base = s.prefs_path.as_ref().and_then(|p| p.parent()).map(|d| d.to_path_buf()).unwrap_or_else(|| crate::temp_dir().join("FilmCraft"));
|
||||
base.join("Voice-over Recordings").to_string_lossy().into_owned()
|
||||
base.join(fallback).to_string_lossy().into_owned()
|
||||
}
|
||||
|
||||
/// `<dir>/<base> <n>.wav` with the first `n` used by neither a project item nor an existing file.
|
||||
pub(crate) fn unique_wav_path(s: &Session, dir: &str, base: &str) -> String {
|
||||
let names: std::collections::HashSet<String> = s.project.items.values().map(|i| i.name.clone()).collect();
|
||||
let mut k = 1u64;
|
||||
loop {
|
||||
let file = format!("{base} {k}.wav");
|
||||
let path = std::path::Path::new(dir).join(&file).to_string_lossy().into_owned();
|
||||
if !names.contains(&file) && s.services.file_size(&path).is_err() {
|
||||
return path;
|
||||
}
|
||||
k += 1;
|
||||
}
|
||||
}
|
||||
|
||||
fn stop(s: &mut Session, p: &Value) -> Result<Value> {
|
||||
|
|
@ -449,16 +470,7 @@ fn stop(s: &mut Session, p: &Value) -> Result<Value> {
|
|||
// write the file
|
||||
let dir = record_dir(s, p);
|
||||
let base = s.prefs.voice_over.name.clone();
|
||||
let names: std::collections::HashSet<String> = s.project.items.values().map(|i| i.name.clone()).collect();
|
||||
let mut k = 1;
|
||||
let path = loop {
|
||||
let file = format!("{base} {k}.wav");
|
||||
let path = std::path::Path::new(&dir).join(&file).to_string_lossy().into_owned();
|
||||
if !names.contains(&file) && s.services.file_size(&path).is_err() {
|
||||
break path;
|
||||
}
|
||||
k += 1;
|
||||
};
|
||||
let path = unique_wav_path(s, &dir, &base);
|
||||
if !cfg!(target_arch = "wasm32") {
|
||||
std::fs::create_dir_all(&dir).map_err(|e| EngineError::Other(format!("{dir}: {e}")))?;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -33,6 +33,8 @@
|
|||
//! - **v12** (M10.7): graphics templates, rolls / crawls, responsive design (pins, intro / outro),
|
||||
//! per-character text styles (`TrackItem::graphic`, `EffectInstance::layer`) and source graphics
|
||||
//! (`Project::source_graphics`). No-op step; older builds would drop these fields when saving.
|
||||
//! - **v13** (M7.9): narrations (`Project::narrations`, Text to Speech). No-op step; older builds
|
||||
//! would drop them when saving.
|
||||
|
||||
#![cfg_attr(not(test), deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable))]
|
||||
|
||||
|
|
@ -53,7 +55,8 @@ pub type Migration = fn(Value) -> Result<Value, String>;
|
|||
|
||||
/// `MIGRATIONS[i]` upgrades schema `i + 1` to `i + 2`. Append one function per schema bump; never
|
||||
/// edit a shipped one.
|
||||
pub const MIGRATIONS: &[Migration] = &[v1_to_v2, v2_to_v3, v3_to_v4, v4_to_v5, v5_to_v6, v6_to_v7, v7_to_v8, v8_to_v9, v9_to_v10, v10_to_v11, v11_to_v12];
|
||||
pub const MIGRATIONS: &[Migration] =
|
||||
&[v1_to_v2, v2_to_v3, v3_to_v4, v4_to_v5, v5_to_v6, v6_to_v7, v7_to_v8, v8_to_v9, v9_to_v10, v10_to_v11, v11_to_v12, v12_to_v13];
|
||||
|
||||
/// The schema version this build writes (and the newest it reads).
|
||||
pub const SCHEMA_VERSION: u32 = MIGRATIONS.len() as u32 + 1;
|
||||
|
|
@ -315,13 +318,18 @@ fn v11_to_v12(doc: Value) -> Result<Value, String> {
|
|||
Ok(doc)
|
||||
}
|
||||
|
||||
/// v12 → v13: narrations (Text to Speech). Existing data needs no change.
|
||||
fn v12_to_v13(doc: Value) -> Result<Value, String> {
|
||||
Ok(doc)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn schema_version_matches_table() {
|
||||
assert_eq!(SCHEMA_VERSION, 12);
|
||||
assert_eq!(SCHEMA_VERSION, 13);
|
||||
}
|
||||
|
||||
#[test]
|
||||
|
|
|
|||
|
|
@ -20,6 +20,7 @@ pub mod keyframe;
|
|||
pub mod mask;
|
||||
pub mod mixer;
|
||||
pub mod multicam;
|
||||
pub mod narration;
|
||||
pub mod transcript;
|
||||
pub mod vtransition;
|
||||
|
||||
|
|
@ -39,6 +40,7 @@ pub use keyframe::{Interpolation, Keyframe, Param, ParamValue};
|
|||
pub use mask::{Mask, MaskMode, MaskPath, MaskVertex, TrackMethod};
|
||||
pub use mixer::{AutomationMode, InputMap, MixerStrip, TrackSend};
|
||||
pub use multicam::{Camera, MergedClip, MulticamAudio, MulticamSel, MulticamSource};
|
||||
pub use narration::{Narration, VocalPitch};
|
||||
pub use transcript::{Speaker, Transcript, Word};
|
||||
|
||||
macro_rules! id_type {
|
||||
|
|
@ -1187,6 +1189,10 @@ pub struct Project {
|
|||
/// project item. Schema v12.
|
||||
#[serde(default, skip_serializing_if = "BTreeMap::is_empty")]
|
||||
pub source_graphics: BTreeMap<ItemId, SourceGraphic>,
|
||||
/// Narrations (Text to Speech): the script and voice settings each generated WAV item was
|
||||
/// made from, keyed by that item. Schema v13.
|
||||
#[serde(default, skip_serializing_if = "BTreeMap::is_empty")]
|
||||
pub narrations: BTreeMap<ItemId, Narration>,
|
||||
}
|
||||
|
||||
/// How a sequence is shown in the Timeline panel: its zoom, scroll position and track heights.
|
||||
|
|
@ -1270,6 +1276,7 @@ impl Project {
|
|||
transcripts: BTreeMap::new(),
|
||||
search_bins: Vec::new(),
|
||||
source_graphics: BTreeMap::new(),
|
||||
narrations: BTreeMap::new(),
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,152 @@
|
|||
//! Narrations: audio generated from text (the Text to Speech panel).
|
||||
//!
|
||||
//! A [`Narration`] belongs to the project item of the WAV file that was generated for it
|
||||
//! (`Project::narrations`, keyed by the item id), so every clip of that item can be clicked to
|
||||
//! reopen the script and voice settings. Editing a narration writes a new WAV (a new item) and
|
||||
//! swaps the clip over to it; the old item and its narration stay, so undo restores them exactly.
|
||||
|
||||
use filmcraft_time::Tick;
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
/// Longest script accepted, in bytes.
|
||||
pub const MAX_TEXT_BYTES: usize = 64 * 1024;
|
||||
/// Pace limits (speaking rate multiplier).
|
||||
pub const MIN_PACE: f64 = 0.5;
|
||||
pub const MAX_PACE: f64 = 2.0;
|
||||
|
||||
/// Text to Speech ▸ Advanced ▸ Vocal pitch.
|
||||
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub enum VocalPitch {
|
||||
ExtraLow,
|
||||
Low,
|
||||
#[default]
|
||||
Default,
|
||||
High,
|
||||
ExtraHigh,
|
||||
}
|
||||
|
||||
impl VocalPitch {
|
||||
pub const ALL: [VocalPitch; 5] = [VocalPitch::ExtraLow, VocalPitch::Low, VocalPitch::Default, VocalPitch::High, VocalPitch::ExtraHigh];
|
||||
|
||||
pub fn label(self) -> &'static str {
|
||||
match self {
|
||||
VocalPitch::ExtraLow => "Extra low",
|
||||
VocalPitch::Low => "Low",
|
||||
VocalPitch::Default => "Default",
|
||||
VocalPitch::High => "High",
|
||||
VocalPitch::ExtraHigh => "Extra high",
|
||||
}
|
||||
}
|
||||
|
||||
/// The command / serde id (`extraLow`, `low`, `default`, `high`, `extraHigh`).
|
||||
pub fn id(self) -> &'static str {
|
||||
match self {
|
||||
VocalPitch::ExtraLow => "extraLow",
|
||||
VocalPitch::Low => "low",
|
||||
VocalPitch::Default => "default",
|
||||
VocalPitch::High => "high",
|
||||
VocalPitch::ExtraHigh => "extraHigh",
|
||||
}
|
||||
}
|
||||
|
||||
pub fn from_id(s: &str) -> Option<VocalPitch> {
|
||||
let n = s.to_ascii_lowercase().replace([' ', '_', '-'], "");
|
||||
Self::ALL.into_iter().find(|p| p.id().to_ascii_lowercase() == n)
|
||||
}
|
||||
|
||||
/// Pitch shift in semitones.
|
||||
pub fn semitones(self) -> f64 {
|
||||
match self {
|
||||
VocalPitch::ExtraLow => -6.0,
|
||||
VocalPitch::Low => -3.0,
|
||||
VocalPitch::Default => 0.0,
|
||||
VocalPitch::High => 3.0,
|
||||
VocalPitch::ExtraHigh => 6.0,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The script and voice settings a narration was generated from.
|
||||
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct Narration {
|
||||
/// The script, with pause markers (`[pause 1s]`) as typed.
|
||||
pub text: String,
|
||||
/// BCP 47 language tag (`en-US`).
|
||||
pub language: String,
|
||||
/// Voice id (`tts.voices`).
|
||||
pub voice: String,
|
||||
#[serde(default)]
|
||||
pub pitch: VocalPitch,
|
||||
/// Speaking rate, [`MIN_PACE`]–[`MAX_PACE`].
|
||||
pub pace: f64,
|
||||
/// Length of the speech in the generated file. The file can be longer: when an edit made the
|
||||
/// speech shorter than its clip, the file is padded with silence to the clip's length.
|
||||
pub speech_duration: Tick,
|
||||
}
|
||||
|
||||
impl Narration {
|
||||
/// Check values read from a project file or a command.
|
||||
pub fn check(&self) -> Result<(), String> {
|
||||
if self.text.len() > MAX_TEXT_BYTES {
|
||||
return Err(format!("the script is longer than {} KB", MAX_TEXT_BYTES / 1024));
|
||||
}
|
||||
if !(MIN_PACE..=MAX_PACE).contains(&self.pace) {
|
||||
return Err(format!("pace must be between {MIN_PACE} and {MAX_PACE}"));
|
||||
}
|
||||
if self.speech_duration < Tick::ZERO {
|
||||
return Err("negative speech duration".into());
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn sample() -> Narration {
|
||||
Narration {
|
||||
text: "Hello there. [pause 1s] Welcome.".into(),
|
||||
language: "en-US".into(),
|
||||
voice: "basic-female".into(),
|
||||
pitch: VocalPitch::High,
|
||||
pace: 1.25,
|
||||
speech_duration: Tick::from_seconds_f64(2.5),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn round_trips_through_json() {
|
||||
let n = sample();
|
||||
let s = serde_json::to_string(&n).unwrap();
|
||||
assert!(s.contains("\"pitch\":\"high\""), "{s}");
|
||||
assert_eq!(serde_json::from_str::<Narration>(&s).unwrap(), n);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn pitch_ids_round_trip() {
|
||||
for p in VocalPitch::ALL {
|
||||
assert_eq!(VocalPitch::from_id(p.id()), Some(p));
|
||||
}
|
||||
assert_eq!(VocalPitch::from_id("Extra High"), Some(VocalPitch::ExtraHigh));
|
||||
assert_eq!(VocalPitch::from_id("loud"), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn check_rejects_hostile_values() {
|
||||
let mut n = sample();
|
||||
n.pace = f64::NAN;
|
||||
assert!(n.check().is_err());
|
||||
n.pace = 3.0;
|
||||
assert!(n.check().is_err());
|
||||
let mut n = sample();
|
||||
n.text = "a".repeat(MAX_TEXT_BYTES + 1);
|
||||
assert!(n.check().is_err());
|
||||
let mut n = sample();
|
||||
n.speech_duration = Tick(-1);
|
||||
assert!(n.check().is_err());
|
||||
assert!(sample().check().is_ok());
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,14 @@
|
|||
[package]
|
||||
name = "filmcraft-tts"
|
||||
description = "FilmCraft text to speech: the Voice trait, the voice catalogue, script parsing (pause markers) and the built-in formant voice"
|
||||
version.workspace = true
|
||||
edition.workspace = true
|
||||
license.workspace = true
|
||||
rust-version.workspace = true
|
||||
|
||||
[lints]
|
||||
workspace = true
|
||||
|
||||
[dependencies]
|
||||
serde = { workspace = true }
|
||||
thiserror = { workspace = true }
|
||||
|
|
@ -0,0 +1,39 @@
|
|||
# filmcraft-tts
|
||||
|
||||
Text to speech for FilmCraft's Text to Speech panel (narrations, L2). Engine commands (`tts.*`) and
|
||||
clip behaviour: `crates/engine/src/narration.rs`.
|
||||
|
||||
- `Voice` trait (script in, mono 24 kHz `f32` out), the voice catalogue (`voices`, `voice`), and
|
||||
the limits (64 KB script, 30-minute narration, pace 0.5–2×, pitch ±12 semitones).
|
||||
- `script`: pause markers `[pause 1s]` / `[pause 500ms]` (case-insensitive, capped at 10 s;
|
||||
malformed markers are read as text).
|
||||
- `formant`: the built-in voices **Basic Female** and **Basic Male**, an original source–filter
|
||||
formant synthesizer: spelling rules → phone targets → glottal pulse train through three two-pole
|
||||
resonators, plus filtered noise for fricatives and bursts. No download, works in the web build.
|
||||
|
||||
Everything is deterministic. No dependencies beyond `serde` and `thiserror`.
|
||||
|
||||
## Status and limitations (M7.9)
|
||||
|
||||
- The built-in voices are **robotic**, and their spelling rules are crude: English words with
|
||||
irregular spelling are mispronounced, and numbers are read digit by digit. They are the fallback
|
||||
and test voice. Natural voices (Kokoro-82M, Apache-2.0, downloaded on first use) come in M7.12,
|
||||
with the clean-room text normalizer (`crates/tts-text`, M7.10) and English pronunciation from
|
||||
CMUdict (M7.11).
|
||||
- English (United States) only.
|
||||
- Vocal pitch shifts the voice's base pitch (and its formants slightly); pace scales every duration.
|
||||
|
||||
## Tests
|
||||
|
||||
`cargo test -p filmcraft-tts`: determinism, valid samples (finite, peak ≤ 0.8, audible RMS), pause
|
||||
markers produce exactly the requested number of zero samples, pace 2× / 0.5× gives 0.5× / 2× the
|
||||
length (±5 % / ±10 %), measured pitch (autocorrelation) of the female voice is ≥ 1.5× the male, and
|
||||
+6 semitones raises it by √2 (±0.12); empty, emoji-only, non-Latin, pause-only and over-long scripts
|
||||
are refused before rendering; hostile scripts never panic.
|
||||
|
||||
## References
|
||||
|
||||
- G. E. Peterson and H. L. Barney, "Control Methods Used in a Study of the Vowels", JASA 24 (1952):
|
||||
the published average formant frequencies used for the vowels.
|
||||
- D. H. Klatt, "Software for a cascade/parallel formant synthesizer", JASA 67 (1980): the standard
|
||||
two-pole digital resonator form. No code was read or copied; everything else is original.
|
||||
|
|
@ -0,0 +1,736 @@
|
|||
//! The built-in voices: an original source–filter formant synthesizer.
|
||||
//!
|
||||
//! Text is turned into phones by simple English spelling rules ([`phones`]): digraphs (`th`, `sh`,
|
||||
//! `ee`, `oo`, `igh`…), a silent final `e` that lengthens the vowel before it (`make`, `time`),
|
||||
//! and digits read as words. Each phone is a target (three formant frequencies and bandwidths,
|
||||
//! voicing and noise amplitudes, a duration). The renderer moves between targets with short
|
||||
//! transitions, drives a cascade of three two-pole resonators with a glottal pulse train for voiced
|
||||
//! sounds, and a resonator at the phone's noise frequency with white noise for fricatives, bursts
|
||||
//! and aspiration. Pitch follows a falling line across each sentence and rises at a question mark.
|
||||
//!
|
||||
//! Vowel formants are the published adult averages of Peterson & Barney, "Control Methods Used in
|
||||
//! a Study of the Vowels" (JASA 24, 1952). The digital resonator is the standard two-pole form
|
||||
//! described in D. H. Klatt, "Software for a cascade/parallel formant synthesizer" (JASA 67, 1980).
|
||||
//! Everything else is original. The result is intelligible-ish and clearly robotic: it is the
|
||||
//! no-download fallback, not the main voice.
|
||||
|
||||
use crate::script::{self, Segment};
|
||||
use crate::{Audio, Gender, MAX_SECONDS, Params, SAMPLE_RATE, TtsError, Voice, VoiceInfo, check_text};
|
||||
|
||||
/// A built-in voice: catalogue entry plus its speaking parameters.
|
||||
#[derive(Clone, Copy, Debug, PartialEq)]
|
||||
pub struct VoiceSpec {
|
||||
pub info: VoiceInfo,
|
||||
/// Base pitch in Hz.
|
||||
pub f0: f64,
|
||||
/// Formant scale (vocal tract length): 1 = adult male averages.
|
||||
pub formant_scale: f64,
|
||||
/// Breath noise mixed into voicing, 0–1.
|
||||
pub breath: f32,
|
||||
}
|
||||
|
||||
const LICENSE: &str = "Built into FilmCraft (project licence)";
|
||||
|
||||
pub static VOICES: &[VoiceSpec] = &[
|
||||
VoiceSpec {
|
||||
info: VoiceInfo {
|
||||
id: "basic-female",
|
||||
name: "Basic Female",
|
||||
language: "en-US",
|
||||
gender: Gender::Female,
|
||||
engine: "basic",
|
||||
description: "Built-in voice. No download; robotic.",
|
||||
license: LICENSE,
|
||||
installed: true,
|
||||
},
|
||||
f0: 205.0,
|
||||
formant_scale: 1.17,
|
||||
breath: 0.06,
|
||||
},
|
||||
VoiceSpec {
|
||||
info: VoiceInfo {
|
||||
id: "basic-male",
|
||||
name: "Basic Male",
|
||||
language: "en-US",
|
||||
gender: Gender::Male,
|
||||
engine: "basic",
|
||||
description: "Built-in voice. No download; robotic.",
|
||||
license: LICENSE,
|
||||
installed: true,
|
||||
},
|
||||
f0: 112.0,
|
||||
formant_scale: 1.0,
|
||||
breath: 0.03,
|
||||
},
|
||||
];
|
||||
|
||||
/// A built-in voice ready to speak.
|
||||
pub struct FormantVoice {
|
||||
spec: VoiceSpec,
|
||||
}
|
||||
|
||||
impl FormantVoice {
|
||||
pub fn new(spec: VoiceSpec) -> Self {
|
||||
FormantVoice { spec }
|
||||
}
|
||||
}
|
||||
|
||||
impl Voice for FormantVoice {
|
||||
fn info(&self) -> VoiceInfo {
|
||||
self.spec.info
|
||||
}
|
||||
|
||||
fn synthesize(&self, text: &str, params: &Params) -> Result<Audio, TtsError> {
|
||||
check_text(text)?;
|
||||
params.check()?;
|
||||
let segments = script::parse(text);
|
||||
// phones per speech segment, and the length check before anything is rendered
|
||||
let mut plan: Vec<Piece> = Vec::with_capacity(segments.len());
|
||||
let mut total_ms = 0.0f64;
|
||||
let mut spoken = false;
|
||||
for seg in &segments {
|
||||
match seg {
|
||||
Segment::Pause { millis } => {
|
||||
total_ms += f64::from(*millis);
|
||||
plan.push(Piece::Pause(*millis));
|
||||
}
|
||||
Segment::Speech(s) => {
|
||||
let ph = phones(s);
|
||||
spoken |= ph.iter().any(|p| p.kind != Kind::Silence);
|
||||
total_ms += ph.iter().map(|p| f64::from(p.dur_ms)).sum::<f64>() / params.pace;
|
||||
plan.push(Piece::Speech(ph));
|
||||
}
|
||||
}
|
||||
if total_ms > MAX_SECONDS * 1000.0 {
|
||||
return Err(TtsError::AudioTooLong);
|
||||
}
|
||||
}
|
||||
if !spoken {
|
||||
return Err(TtsError::Empty);
|
||||
}
|
||||
let estimate = (total_ms / 1000.0 * f64::from(SAMPLE_RATE)).ceil() as usize;
|
||||
let mut out: Vec<f32> = Vec::new();
|
||||
out.try_reserve(estimate.saturating_add(SAMPLE_RATE as usize)).map_err(|e| TtsError::Invalid(format!("unable to allocate the narration: {e}")))?;
|
||||
for piece in &plan {
|
||||
match piece {
|
||||
Piece::Pause(ms) => {
|
||||
let n = (f64::from(*ms) / 1000.0 * f64::from(SAMPLE_RATE)).round() as usize;
|
||||
out.resize(out.len().saturating_add(n), 0.0);
|
||||
}
|
||||
Piece::Speech(ph) => render(&self.spec, ph, params, &mut out),
|
||||
}
|
||||
}
|
||||
// one gain for the whole narration so loudness doesn't jump between pieces
|
||||
let peak = out.iter().fold(0.0f32, |m, s| m.max(s.abs()));
|
||||
if peak > 0.0 && peak.is_finite() {
|
||||
let g = 0.8 / peak;
|
||||
for s in &mut out {
|
||||
*s *= g;
|
||||
}
|
||||
} else if !peak.is_finite() {
|
||||
return Err(TtsError::Invalid("synthesis produced invalid samples".into()));
|
||||
}
|
||||
Ok(Audio { samples: out, sample_rate: SAMPLE_RATE })
|
||||
}
|
||||
}
|
||||
|
||||
enum Piece {
|
||||
Pause(u32),
|
||||
Speech(Vec<Phone>),
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------------------------------- phones
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
enum Kind {
|
||||
Vowel,
|
||||
Glide,
|
||||
Nasal,
|
||||
Fricative,
|
||||
Closure,
|
||||
Burst,
|
||||
Silence,
|
||||
}
|
||||
|
||||
/// One synthesis target.
|
||||
#[derive(Clone, Copy, Debug, PartialEq)]
|
||||
struct Phone {
|
||||
kind: Kind,
|
||||
/// Formant frequencies (adult male, scaled per voice) and bandwidths, Hz.
|
||||
f: [f32; 3],
|
||||
bw: [f32; 3],
|
||||
/// Voicing amplitude 0–1.
|
||||
voice: f32,
|
||||
/// Noise amplitude 0–1, through a resonator at `noise_f` / `noise_bw`.
|
||||
noise: f32,
|
||||
noise_f: f32,
|
||||
noise_bw: f32,
|
||||
dur_ms: f32,
|
||||
/// Pitch multiplier at this phone (stress, sentence end).
|
||||
pitch: f32,
|
||||
/// Ends a sentence (pitch reset after it).
|
||||
sentence_end: bool,
|
||||
}
|
||||
|
||||
const BW: [f32; 3] = [80.0, 100.0, 140.0];
|
||||
/// Neutral tract: what silences and closures move towards.
|
||||
const NEUTRAL: [f32; 3] = [500.0, 1500.0, 2500.0];
|
||||
|
||||
fn base(kind: Kind, f: [f32; 3], dur_ms: f32) -> Phone {
|
||||
Phone { kind, f, bw: BW, voice: 0.0, noise: 0.0, noise_f: 3000.0, noise_bw: 2000.0, dur_ms, pitch: 1.0, sentence_end: false }
|
||||
}
|
||||
|
||||
fn vowel(f: [f32; 3], dur_ms: f32) -> Phone {
|
||||
Phone { voice: 1.0, ..base(Kind::Vowel, f, dur_ms) }
|
||||
}
|
||||
|
||||
fn glide(f: [f32; 3]) -> Phone {
|
||||
Phone { voice: 0.8, ..base(Kind::Glide, f, 65.0) }
|
||||
}
|
||||
|
||||
fn nasal(f: [f32; 3]) -> Phone {
|
||||
Phone { voice: 0.45, bw: [60.0, 200.0, 300.0], ..base(Kind::Nasal, f, 75.0) }
|
||||
}
|
||||
|
||||
fn fricative(noise_f: f32, noise_bw: f32, noise: f32, voiced: bool, dur_ms: f32) -> Phone {
|
||||
Phone { voice: if voiced { 0.35 } else { 0.0 }, noise, noise_f, noise_bw, ..base(Kind::Fricative, NEUTRAL, dur_ms) }
|
||||
}
|
||||
|
||||
fn silence(dur_ms: f32) -> Phone {
|
||||
base(Kind::Silence, NEUTRAL, dur_ms)
|
||||
}
|
||||
|
||||
/// A stop: closure then burst.
|
||||
fn stop(out: &mut Vec<Phone>, burst_f: f32, voiced: bool) {
|
||||
out.push(Phone { voice: if voiced { 0.12 } else { 0.0 }, ..base(Kind::Closure, NEUTRAL, if voiced { 45.0 } else { 55.0 }) });
|
||||
out.push(Phone { noise: 0.9, noise_f: burst_f, noise_bw: 1800.0, ..base(Kind::Burst, NEUTRAL, if voiced { 15.0 } else { 30.0 }) });
|
||||
}
|
||||
|
||||
// Peterson & Barney (1952) adult male averages: F1, F2, F3.
|
||||
const AA: [f32; 3] = [730.0, 1090.0, 2440.0];
|
||||
const AE: [f32; 3] = [660.0, 1720.0, 2410.0];
|
||||
const AH: [f32; 3] = [640.0, 1190.0, 2390.0];
|
||||
const AO: [f32; 3] = [570.0, 840.0, 2410.0];
|
||||
const EH: [f32; 3] = [530.0, 1840.0, 2480.0];
|
||||
const ER: [f32; 3] = [490.0, 1350.0, 1690.0];
|
||||
const IH: [f32; 3] = [390.0, 1990.0, 2550.0];
|
||||
const IY: [f32; 3] = [270.0, 2290.0, 3010.0];
|
||||
const UH: [f32; 3] = [440.0, 1020.0, 2240.0];
|
||||
const UW: [f32; 3] = [300.0, 870.0, 2240.0];
|
||||
|
||||
fn diphthong(out: &mut Vec<Phone>, a: [f32; 3], b: [f32; 3], stressed: bool) {
|
||||
let d = if stressed { 95.0 } else { 70.0 };
|
||||
out.push(vowel(a, d));
|
||||
out.push(vowel(b, d * 0.8));
|
||||
}
|
||||
|
||||
const DIGITS: [&str; 10] = ["zero", "one", "two", "three", "four", "five", "six", "seven", "eight", "nine"];
|
||||
|
||||
fn is_vowel(c: char) -> bool {
|
||||
matches!(c, 'a' | 'e' | 'i' | 'o' | 'u')
|
||||
}
|
||||
|
||||
/// Phones for a piece of speech (no pause markers).
|
||||
fn phones(text: &str) -> Vec<Phone> {
|
||||
let mut out = Vec::new();
|
||||
let mut word = String::new();
|
||||
let mut question_from = 0usize;
|
||||
for c in text.chars().chain(std::iter::once(' ')) {
|
||||
let c = c.to_ascii_lowercase();
|
||||
if c.is_ascii_alphabetic() || c == '\'' {
|
||||
if c != '\'' {
|
||||
word.push(c);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if !word.is_empty() {
|
||||
speak_word(&word, &mut out);
|
||||
word.clear();
|
||||
}
|
||||
match c {
|
||||
'0'..='9' => {
|
||||
let d = DIGITS.get(c as usize - '0' as usize).copied().unwrap_or("");
|
||||
speak_word(d, &mut out);
|
||||
out.push(silence(30.0));
|
||||
}
|
||||
',' | ';' | ':' | '-' | '(' | ')' => out.push(silence(170.0)),
|
||||
'.' | '!' | '?' => {
|
||||
if c == '?' {
|
||||
// rise over the last word of the question
|
||||
let n = out.len();
|
||||
let from =
|
||||
out.get(question_from..).map_or(n, |s| s.iter().rposition(|p| p.kind == Kind::Silence).map_or(question_from, |i| question_from + i));
|
||||
for p in out.get_mut(from..).into_iter().flatten() {
|
||||
p.pitch *= 1.25;
|
||||
}
|
||||
}
|
||||
if let Some(last) = out.last_mut() {
|
||||
last.sentence_end = true;
|
||||
}
|
||||
out.push(silence(330.0));
|
||||
question_from = out.len();
|
||||
}
|
||||
_ => {
|
||||
if out.last().is_some_and(|p| p.kind != Kind::Silence) {
|
||||
out.push(silence(25.0));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// Spelling rules for one lowercase ASCII word.
|
||||
fn speak_word(w: &str, out: &mut Vec<Phone>) {
|
||||
let c: Vec<char> = w.chars().collect();
|
||||
let n = c.len();
|
||||
let at = |i: usize| c.get(i).copied().unwrap_or(' ');
|
||||
// a final silent e makes the vowel before the last consonant long: make, time, home, cute
|
||||
let magic_e = n >= 3 && at(n - 1) == 'e' && !is_vowel(at(n - 2)) && is_vowel(at(n - 3));
|
||||
let mut stressed = true;
|
||||
let mut i = 0;
|
||||
while i < n {
|
||||
let (a, b, d) = (at(i), at(i + 1), at(i + 2));
|
||||
let long = magic_e && i == n - 3;
|
||||
let vd = if stressed { 115.0 } else { 80.0 };
|
||||
let before = out.len();
|
||||
let step = match a {
|
||||
'a' if b == 'i' || b == 'y' => {
|
||||
diphthong(out, EH, IY, stressed);
|
||||
2
|
||||
}
|
||||
'a' if b == 'u' || b == 'w' => {
|
||||
out.push(vowel(AO, vd));
|
||||
2
|
||||
}
|
||||
'a' if b == 'r' => {
|
||||
out.push(vowel(AA, vd));
|
||||
out.push(glide([420.0, 1300.0, 1600.0]));
|
||||
2
|
||||
}
|
||||
'a' if long => {
|
||||
diphthong(out, EH, IY, stressed);
|
||||
1
|
||||
}
|
||||
'a' => {
|
||||
out.push(vowel(if stressed { AE } else { AH }, vd));
|
||||
1
|
||||
}
|
||||
'e' if b == 'e' || b == 'a' => {
|
||||
out.push(vowel(IY, vd + 20.0));
|
||||
2
|
||||
}
|
||||
'e' if b == 'r' => {
|
||||
out.push(vowel(ER, vd));
|
||||
2
|
||||
}
|
||||
'e' if b == 'y' && i + 2 == n => {
|
||||
out.push(vowel(IY, vd));
|
||||
2
|
||||
}
|
||||
'e' if i == n - 1 && n > 2 => 1, // silent final e
|
||||
'e' => {
|
||||
out.push(vowel(if i == n - 1 { IY } else { EH }, vd));
|
||||
1
|
||||
}
|
||||
'i' if b == 'g' && d == 'h' => {
|
||||
diphthong(out, AA, IY, stressed);
|
||||
3
|
||||
}
|
||||
'i' if long || (i == n - 1 && n <= 2) => {
|
||||
diphthong(out, AA, IY, stressed);
|
||||
1
|
||||
}
|
||||
'i' if b == 'r' => {
|
||||
out.push(vowel(ER, vd));
|
||||
2
|
||||
}
|
||||
'i' => {
|
||||
out.push(vowel(IH, vd));
|
||||
1
|
||||
}
|
||||
'o' if b == 'o' => {
|
||||
out.push(vowel(if d == 'k' { UH } else { UW }, vd + 15.0));
|
||||
2
|
||||
}
|
||||
'o' if b == 'u' || b == 'w' => {
|
||||
diphthong(out, AA, UW, stressed);
|
||||
2
|
||||
}
|
||||
'o' if b == 'i' || b == 'y' => {
|
||||
diphthong(out, AO, IY, stressed);
|
||||
2
|
||||
}
|
||||
'o' if b == 'a' || long || i == n - 1 => {
|
||||
diphthong(out, AO, UW, stressed);
|
||||
if b == 'a' { 2 } else { 1 }
|
||||
}
|
||||
'o' if b == 'r' => {
|
||||
out.push(vowel(AO, vd));
|
||||
out.push(glide([420.0, 1300.0, 1600.0]));
|
||||
2
|
||||
}
|
||||
'o' => {
|
||||
out.push(vowel(AA, vd));
|
||||
1
|
||||
}
|
||||
'u' if long => {
|
||||
out.push(glide([280.0, 2200.0, 3000.0]));
|
||||
out.push(vowel(UW, vd));
|
||||
1
|
||||
}
|
||||
'u' if b == 'r' => {
|
||||
out.push(vowel(ER, vd));
|
||||
2
|
||||
}
|
||||
'u' => {
|
||||
out.push(vowel(AH, vd));
|
||||
1
|
||||
}
|
||||
'y' if i == 0 => {
|
||||
out.push(glide([280.0, 2200.0, 3000.0]));
|
||||
1
|
||||
}
|
||||
'y' => {
|
||||
if i == n - 1 && n <= 3 {
|
||||
diphthong(out, AA, IY, stressed);
|
||||
} else {
|
||||
out.push(vowel(if i == n - 1 { IY } else { IH }, vd));
|
||||
}
|
||||
1
|
||||
}
|
||||
't' if b == 'h' => {
|
||||
out.push(fricative(6000.0, 4000.0, 0.25, false, 80.0));
|
||||
2
|
||||
}
|
||||
's' if b == 'h' => {
|
||||
out.push(fricative(2600.0, 1400.0, 0.7, false, 110.0));
|
||||
2
|
||||
}
|
||||
'c' if b == 'h' => {
|
||||
stop(out, 4000.0, false);
|
||||
out.push(fricative(2600.0, 1400.0, 0.7, false, 70.0));
|
||||
2
|
||||
}
|
||||
'p' if b == 'h' => {
|
||||
out.push(fricative(5000.0, 4000.0, 0.3, false, 90.0));
|
||||
2
|
||||
}
|
||||
'w' if b == 'h' => {
|
||||
out.push(glide([300.0, 610.0, 2200.0]));
|
||||
2
|
||||
}
|
||||
'n' if b == 'g' => {
|
||||
out.push(nasal([250.0, 2300.0, 2700.0]));
|
||||
2
|
||||
}
|
||||
'c' if b == 'k' => {
|
||||
stop(out, 2000.0, false);
|
||||
2
|
||||
}
|
||||
'q' if b == 'u' => {
|
||||
stop(out, 2000.0, false);
|
||||
out.push(glide([300.0, 610.0, 2200.0]));
|
||||
2
|
||||
}
|
||||
'c' if matches!(b, 'e' | 'i' | 'y') => {
|
||||
out.push(fricative(6000.0, 2000.0, 0.6, false, 100.0));
|
||||
1
|
||||
}
|
||||
'c' | 'k' | 'q' => {
|
||||
stop(out, 2000.0, false);
|
||||
1
|
||||
}
|
||||
'g' if matches!(b, 'e' | 'i') && i > 0 => {
|
||||
stop(out, 3500.0, true);
|
||||
out.push(fricative(2600.0, 1400.0, 0.5, true, 60.0));
|
||||
1
|
||||
}
|
||||
'g' => {
|
||||
stop(out, 2000.0, true);
|
||||
1
|
||||
}
|
||||
'j' => {
|
||||
stop(out, 3500.0, true);
|
||||
out.push(fricative(2600.0, 1400.0, 0.5, true, 60.0));
|
||||
1
|
||||
}
|
||||
'b' => {
|
||||
stop(out, 800.0, true);
|
||||
1
|
||||
}
|
||||
'd' => {
|
||||
stop(out, 3500.0, true);
|
||||
1
|
||||
}
|
||||
'p' => {
|
||||
stop(out, 800.0, false);
|
||||
1
|
||||
}
|
||||
't' => {
|
||||
stop(out, 4000.0, false);
|
||||
1
|
||||
}
|
||||
'x' => {
|
||||
stop(out, 2000.0, false);
|
||||
out.push(fricative(6000.0, 2000.0, 0.6, false, 90.0));
|
||||
1
|
||||
}
|
||||
'f' => {
|
||||
out.push(fricative(5000.0, 4000.0, 0.3, false, 90.0));
|
||||
1
|
||||
}
|
||||
'v' => {
|
||||
out.push(fricative(5000.0, 4000.0, 0.25, true, 70.0));
|
||||
1
|
||||
}
|
||||
's' => {
|
||||
out.push(fricative(6000.0, 2000.0, 0.6, false, 100.0));
|
||||
1
|
||||
}
|
||||
'z' => {
|
||||
out.push(fricative(6000.0, 2000.0, 0.45, true, 85.0));
|
||||
1
|
||||
}
|
||||
'h' => {
|
||||
out.push(fricative(1500.0, 2500.0, 0.3, false, 55.0));
|
||||
1
|
||||
}
|
||||
'l' => {
|
||||
out.push(glide([360.0, 1300.0, 2700.0]));
|
||||
1
|
||||
}
|
||||
'r' => {
|
||||
out.push(glide([420.0, 1300.0, 1600.0]));
|
||||
1
|
||||
}
|
||||
'w' => {
|
||||
out.push(glide([300.0, 610.0, 2200.0]));
|
||||
1
|
||||
}
|
||||
'm' => {
|
||||
out.push(nasal([250.0, 1100.0, 2400.0]));
|
||||
1
|
||||
}
|
||||
'n' => {
|
||||
out.push(nasal([250.0, 1700.0, 2600.0]));
|
||||
1
|
||||
}
|
||||
_ => 1,
|
||||
};
|
||||
let added_vowel = out.get(before..).is_some_and(|s| s.iter().any(|p| p.kind == Kind::Vowel));
|
||||
if added_vowel && stressed {
|
||||
for p in out.get_mut(before..).into_iter().flatten() {
|
||||
p.pitch *= 1.08;
|
||||
}
|
||||
stressed = false;
|
||||
}
|
||||
// doubled consonants are one sound: "ll", "ss", "tt"
|
||||
let mut skip = step;
|
||||
while !is_vowel(a) && at(i + skip) == a {
|
||||
skip += 1;
|
||||
}
|
||||
i += skip;
|
||||
}
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------------------------------- render
|
||||
|
||||
/// Two-pole resonator (unity gain at DC).
|
||||
#[derive(Clone, Copy, Default)]
|
||||
struct Resonator {
|
||||
a: f32,
|
||||
b: f32,
|
||||
c: f32,
|
||||
y1: f32,
|
||||
y2: f32,
|
||||
}
|
||||
|
||||
impl Resonator {
|
||||
fn set(&mut self, f: f32, bw: f32) {
|
||||
let t = 1.0 / SAMPLE_RATE as f32;
|
||||
let f = f.clamp(50.0, SAMPLE_RATE as f32 * 0.45);
|
||||
let bw = bw.clamp(20.0, 8000.0);
|
||||
self.c = -(-2.0 * std::f32::consts::PI * bw * t).exp();
|
||||
self.b = 2.0 * (-std::f32::consts::PI * bw * t).exp() * (2.0 * std::f32::consts::PI * f * t).cos();
|
||||
self.a = 1.0 - self.b - self.c;
|
||||
}
|
||||
fn tick(&mut self, x: f32) -> f32 {
|
||||
let y = self.a * x + self.b * self.y1 + self.c * self.y2;
|
||||
self.y2 = self.y1;
|
||||
self.y1 = y;
|
||||
y
|
||||
}
|
||||
}
|
||||
|
||||
/// Deterministic white noise (xorshift32), −1..1.
|
||||
struct Noise(u32);
|
||||
|
||||
impl Noise {
|
||||
fn next(&mut self) -> f32 {
|
||||
let mut x = self.0;
|
||||
x ^= x << 13;
|
||||
x ^= x >> 17;
|
||||
x ^= x << 5;
|
||||
self.0 = x;
|
||||
(x as f32 / u32::MAX as f32) * 2.0 - 1.0
|
||||
}
|
||||
}
|
||||
|
||||
/// Parameters are updated every `BLOCK` samples.
|
||||
const BLOCK: usize = 48;
|
||||
/// Transition into each phone, ms.
|
||||
const TRANSITION_MS: f32 = 30.0;
|
||||
|
||||
fn lerp(a: f32, b: f32, t: f32) -> f32 {
|
||||
a + (b - a) * t
|
||||
}
|
||||
|
||||
fn render(spec: &VoiceSpec, ph: &[Phone], params: &Params, out: &mut Vec<f32>) {
|
||||
let sr = SAMPLE_RATE as f32;
|
||||
let shift = 2f64.powf(params.semitones / 12.0);
|
||||
let f0_base = (spec.f0 * shift) as f32;
|
||||
let fscale = (spec.formant_scale * 2f64.powf(params.semitones / 12.0 * 0.15)) as f32;
|
||||
let pace = params.pace as f32;
|
||||
let mut res = [Resonator::default(); 3];
|
||||
let mut nres = Resonator::default();
|
||||
let mut noise = Noise(0x9E37_79B9);
|
||||
let mut phase = 0.0f32;
|
||||
let mut prev_glottal = 0.0f32;
|
||||
let mut dc = (0.0f32, 0.0f32);
|
||||
// sentence pitch line: from 1.1 down to 0.85 of the base over each sentence
|
||||
let sentence_len = |from: usize| -> f32 { ph.get(from..).map_or(0, |s| s.iter().position(|p| p.sentence_end).map_or(s.len(), |e| e + 1)) as f32 };
|
||||
let mut sentence_start = 0usize;
|
||||
let mut sentence_n = sentence_len(0).max(1.0);
|
||||
let mut prev = silence(0.0);
|
||||
for (k, p) in ph.iter().enumerate() {
|
||||
let n = ((p.dur_ms / pace) / 1000.0 * sr).round().max(1.0) as usize;
|
||||
let trans = ((TRANSITION_MS / pace) / 1000.0 * sr).min(n as f32 * 0.5).max(1.0);
|
||||
let line = 1.1 - 0.25 * ((k - sentence_start) as f32 / sentence_n);
|
||||
let mut i = 0;
|
||||
while i < n {
|
||||
let t = ((i as f32) / trans).min(1.0);
|
||||
let f = [0, 1, 2].map(|j| lerp(prev.f[j], p.f[j], t) * fscale);
|
||||
for (j, r) in res.iter_mut().enumerate() {
|
||||
r.set(f[j], p.bw[j] * fscale.sqrt());
|
||||
}
|
||||
nres.set(lerp(prev.noise_f, p.noise_f, t), lerp(prev.noise_bw, p.noise_bw, t));
|
||||
let voice = lerp(prev.voice, p.voice, t);
|
||||
let namp = lerp(prev.noise, p.noise, t);
|
||||
let f0 = (f0_base * line * lerp(prev.pitch, p.pitch, t)).clamp(40.0, 600.0);
|
||||
let end = (i + BLOCK).min(n);
|
||||
for _ in i..end {
|
||||
phase += f0 / sr;
|
||||
if phase >= 1.0 {
|
||||
phase -= 1.0;
|
||||
}
|
||||
// glottal flow pulse (open 60 % of the period), differentiated for lip radiation
|
||||
let g = if phase < 0.6 { (std::f32::consts::PI * phase / 0.6).sin().powi(2) } else { 0.0 };
|
||||
let dg = g - prev_glottal;
|
||||
prev_glottal = g;
|
||||
let breath = noise.next() * spec.breath * g;
|
||||
let mut v = (dg * 6.0 + breath) * voice;
|
||||
for r in &mut res {
|
||||
v = r.tick(v);
|
||||
}
|
||||
let nz = nres.tick(noise.next()) * namp * 0.6;
|
||||
// DC blocker
|
||||
let x = v + nz;
|
||||
let y = x - dc.0 + 0.995 * dc.1;
|
||||
dc = (x, y);
|
||||
out.push(y);
|
||||
}
|
||||
i = end;
|
||||
}
|
||||
prev = *p;
|
||||
if p.sentence_end {
|
||||
sentence_start = k + 1;
|
||||
sentence_n = sentence_len(k + 1).max(1.0);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::voice;
|
||||
|
||||
fn say(id: &str, text: &str, p: Params) -> Audio {
|
||||
voice(id).unwrap().synthesize(text, &p).unwrap()
|
||||
}
|
||||
|
||||
/// Dominant pitch of the middle of `s` by autocorrelation, 60–500 Hz.
|
||||
fn pitch(s: &[f32]) -> f64 {
|
||||
let mid = &s[s.len() / 3..s.len() * 2 / 3];
|
||||
let sr = f64::from(SAMPLE_RATE);
|
||||
let (lo, hi) = ((sr / 500.0) as usize, (sr / 60.0) as usize);
|
||||
let best = (lo..hi).max_by(|&a, &b| {
|
||||
let ca: f64 = mid.iter().zip(&mid[a..]).map(|(x, y)| f64::from(x * y)).sum();
|
||||
let cb: f64 = mid.iter().zip(&mid[b..]).map(|(x, y)| f64::from(x * y)).sum();
|
||||
ca.total_cmp(&cb)
|
||||
});
|
||||
sr / best.unwrap() as f64
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn speaks_deterministically_with_valid_samples() {
|
||||
let a = say("basic-female", "Hello world, this is FilmCraft. Ready?", Params::default());
|
||||
let b = say("basic-female", "Hello world, this is FilmCraft. Ready?", Params::default());
|
||||
assert_eq!(a, b);
|
||||
assert_eq!(a.sample_rate, SAMPLE_RATE);
|
||||
assert!(a.seconds() > 1.0 && a.seconds() < 8.0, "{}", a.seconds());
|
||||
assert!(a.samples.iter().all(|s| s.is_finite() && s.abs() <= 0.8001));
|
||||
let rms = (a.samples.iter().map(|s| f64::from(s * s)).sum::<f64>() / a.samples.len() as f64).sqrt();
|
||||
assert!(rms > 0.03, "too quiet: {rms}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn pause_markers_are_exact_silence() {
|
||||
let plain = say("basic-male", "One. Two.", Params::default());
|
||||
let paused = say("basic-male", "One. [pause 1s] Two.", Params::default());
|
||||
assert_eq!(paused.samples.len(), plain.samples.len() + SAMPLE_RATE as usize);
|
||||
let one = say("basic-male", "One. ", Params::default()).samples.len();
|
||||
assert!(paused.samples[one..one + SAMPLE_RATE as usize].iter().all(|&s| s == 0.0));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn pace_scales_duration() {
|
||||
let text = "The quick brown fox jumps over the lazy dog.";
|
||||
let n1 = say("basic-female", text, Params::default()).samples.len() as f64;
|
||||
let n2 = say("basic-female", text, Params { pace: 2.0, ..Default::default() }).samples.len() as f64;
|
||||
let nh = say("basic-female", text, Params { pace: 0.5, ..Default::default() }).samples.len() as f64;
|
||||
assert!((n2 / n1 - 0.5).abs() < 0.05, "{}", n2 / n1);
|
||||
assert!((nh / n1 - 2.0).abs() < 0.1, "{}", nh / n1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn pitch_follows_voice_and_setting() {
|
||||
let text = "aaaaaaaaaaaaaaaaaaaa";
|
||||
let female = pitch(&say("basic-female", text, Params::default()).samples);
|
||||
let male = pitch(&say("basic-male", text, Params::default()).samples);
|
||||
let high = pitch(&say("basic-male", text, Params { semitones: 6.0, ..Default::default() }).samples);
|
||||
assert!(female > male * 1.5, "female {female} male {male}");
|
||||
assert!((high / male - 2f64.powf(0.5)).abs() < 0.12, "high {high} male {male}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn refuses_what_it_cannot_say() {
|
||||
let v = voice("basic-female").unwrap();
|
||||
assert_eq!(v.synthesize("", &Params::default()), Err(TtsError::Empty));
|
||||
assert_eq!(v.synthesize("👍 🎬", &Params::default()), Err(TtsError::Empty));
|
||||
assert_eq!(v.synthesize("日本語", &Params::default()), Err(TtsError::Empty));
|
||||
assert_eq!(v.synthesize("[pause 2s]", &Params::default()), Err(TtsError::Empty));
|
||||
assert!(matches!(v.synthesize("hi", &Params { pace: f64::NAN, ..Default::default() }), Err(TtsError::Invalid(_))));
|
||||
// 64 KB of letters would be over the 30-minute limit: refused before rendering
|
||||
assert_eq!(v.synthesize(&"a ".repeat(crate::MAX_TEXT_BYTES / 2), &Params { pace: 0.5, ..Default::default() }), Err(TtsError::AudioTooLong));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn hostile_scripts_never_panic() {
|
||||
let v = voice("basic-male").unwrap();
|
||||
for s in ["a\0b", "x\u{301}y", "'''", "9999999999999999999999", "[pause 1s]a[pause", "Ünïcödé façade", "?!?!?!", "a?b?c?", "zzzzzzzz", "e"] {
|
||||
let r = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| v.synthesize(s, &Params::default())));
|
||||
assert!(r.is_ok(), "panicked on {s:?}");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,187 @@
|
|||
//! Text to speech for FilmCraft's Text to Speech panel (narrations).
|
||||
//!
|
||||
//! - [`Voice`]: the trait every voice implements: a script in, mono samples out.
|
||||
//! - [`voices`] / [`voice`]: the voice catalogue (id, name, language, gender, engine, licence).
|
||||
//! - [`script`]: splits a script into speech and pause markers (`[pause 1s]`, `[pause 500ms]`).
|
||||
//! - [`formant`]: the built-in voices, an original source–filter formant synthesizer. They need
|
||||
//! no download and work everywhere (also in the web build); they sound robotic. The neural voices
|
||||
//! (Kokoro-82M, downloaded on first use) are added behind a feature in a later milestone.
|
||||
//!
|
||||
//! Everything is deterministic: the same script and settings give the same samples.
|
||||
|
||||
pub mod formant;
|
||||
pub mod script;
|
||||
|
||||
use serde::Serialize;
|
||||
|
||||
/// Sample rate of every voice's output.
|
||||
pub const SAMPLE_RATE: u32 = 24_000;
|
||||
/// Longest narration a voice produces, in seconds.
|
||||
pub const MAX_SECONDS: f64 = 30.0 * 60.0;
|
||||
/// Longest script accepted, in bytes (same as `filmcraft_project::narration::MAX_TEXT_BYTES`).
|
||||
pub const MAX_TEXT_BYTES: usize = 64 * 1024;
|
||||
/// Pace limits.
|
||||
pub const MIN_PACE: f64 = 0.5;
|
||||
pub const MAX_PACE: f64 = 2.0;
|
||||
/// Pitch shift limit in semitones (either way).
|
||||
pub const MAX_SEMITONES: f64 = 12.0;
|
||||
|
||||
#[derive(Debug, thiserror::Error, PartialEq)]
|
||||
pub enum TtsError {
|
||||
#[error("unknown voice `{0}`")]
|
||||
UnknownVoice(String),
|
||||
#[error("the script has nothing to say")]
|
||||
Empty,
|
||||
#[error("the script is longer than {} KB", MAX_TEXT_BYTES / 1024)]
|
||||
TextTooLong,
|
||||
#[error("the narration would be longer than {} minutes", MAX_SECONDS / 60.0)]
|
||||
AudioTooLong,
|
||||
#[error("{0}")]
|
||||
Invalid(String),
|
||||
}
|
||||
|
||||
/// How a voice speaks.
|
||||
#[derive(Clone, Copy, Debug, PartialEq)]
|
||||
pub struct Params {
|
||||
/// Pitch shift in semitones, ±[`MAX_SEMITONES`].
|
||||
pub semitones: f64,
|
||||
/// Speaking rate, [`MIN_PACE`]–[`MAX_PACE`] (1 = normal).
|
||||
pub pace: f64,
|
||||
}
|
||||
|
||||
impl Default for Params {
|
||||
fn default() -> Self {
|
||||
Params { semitones: 0.0, pace: 1.0 }
|
||||
}
|
||||
}
|
||||
|
||||
impl Params {
|
||||
pub fn check(&self) -> Result<(), TtsError> {
|
||||
if !(MIN_PACE..=MAX_PACE).contains(&self.pace) {
|
||||
return Err(TtsError::Invalid(format!("pace must be between {MIN_PACE} and {MAX_PACE}")));
|
||||
}
|
||||
if !(-MAX_SEMITONES..=MAX_SEMITONES).contains(&self.semitones) {
|
||||
return Err(TtsError::Invalid(format!("pitch must be within ±{MAX_SEMITONES} semitones")));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
/// Mono audio at [`SAMPLE_RATE`].
|
||||
#[derive(Clone, Debug, Default, PartialEq)]
|
||||
pub struct Audio {
|
||||
pub samples: Vec<f32>,
|
||||
pub sample_rate: u32,
|
||||
}
|
||||
|
||||
impl Audio {
|
||||
pub fn seconds(&self) -> f64 {
|
||||
self.samples.len() as f64 / f64::from(self.sample_rate.max(1))
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub enum Gender {
|
||||
Female,
|
||||
Male,
|
||||
}
|
||||
|
||||
/// A voice in the catalogue.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Serialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct VoiceInfo {
|
||||
pub id: &'static str,
|
||||
pub name: &'static str,
|
||||
/// BCP 47 tag.
|
||||
pub language: &'static str,
|
||||
pub gender: Gender,
|
||||
/// `basic` (built in) or `neural` (downloaded).
|
||||
pub engine: &'static str,
|
||||
pub description: &'static str,
|
||||
pub license: &'static str,
|
||||
/// Usable now (built in, or downloaded).
|
||||
pub installed: bool,
|
||||
}
|
||||
|
||||
/// A voice that turns a script into audio.
|
||||
pub trait Voice: Send + Sync {
|
||||
fn info(&self) -> VoiceInfo;
|
||||
/// Speak `text` (pause markers allowed). Fails on an empty or over-long script, or bad params.
|
||||
fn synthesize(&self, text: &str, params: &Params) -> Result<Audio, TtsError>;
|
||||
}
|
||||
|
||||
/// Languages offered (English only for now).
|
||||
pub const LANGUAGES: &[(&str, &str)] = &[("en-US", "English (United States)")];
|
||||
|
||||
/// Every voice FilmCraft knows.
|
||||
pub fn voices() -> Vec<VoiceInfo> {
|
||||
formant::VOICES.iter().map(|v| v.info).collect()
|
||||
}
|
||||
|
||||
/// The voice with this id.
|
||||
pub fn voice(id: &str) -> Result<Box<dyn Voice>, TtsError> {
|
||||
formant::VOICES
|
||||
.iter()
|
||||
.find(|v| v.info.id == id)
|
||||
.map(|v| Box::new(formant::FormantVoice::new(*v)) as Box<dyn Voice>)
|
||||
.ok_or_else(|| TtsError::UnknownVoice(id.to_string()))
|
||||
}
|
||||
|
||||
/// The voice used when none is chosen.
|
||||
pub fn default_voice_id() -> &'static str {
|
||||
formant::VOICES.first().map(|v| v.info.id).unwrap_or("basic-female")
|
||||
}
|
||||
|
||||
/// The sentence "Hear this voice" plays.
|
||||
pub const SAMPLE_SENTENCE: &str = "Hello! This is how I sound when I read your script.";
|
||||
|
||||
/// Validate a script before synthesis: length, and that it says something.
|
||||
pub fn check_text(text: &str) -> Result<(), TtsError> {
|
||||
if text.len() > MAX_TEXT_BYTES {
|
||||
return Err(TtsError::TextTooLong);
|
||||
}
|
||||
let says_something = script::parse(text).iter().any(|s| matches!(s, script::Segment::Speech(t) if t.chars().any(char::is_alphanumeric)));
|
||||
if !says_something {
|
||||
return Err(TtsError::Empty);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn catalogue_ids_are_unique_and_resolvable() {
|
||||
let v = voices();
|
||||
assert!(v.len() >= 2);
|
||||
for (i, a) in v.iter().enumerate() {
|
||||
assert!(v.iter().skip(i + 1).all(|b| b.id != a.id), "duplicate {}", a.id);
|
||||
assert_eq!(voice(a.id).unwrap().info().id, a.id);
|
||||
}
|
||||
assert_eq!(voice("nope").err(), Some(TtsError::UnknownVoice("nope".into())));
|
||||
assert!(voice(default_voice_id()).is_ok());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn params_are_checked() {
|
||||
assert!(Params::default().check().is_ok());
|
||||
for p in [
|
||||
Params { pace: f64::NAN, ..Default::default() },
|
||||
Params { pace: 0.1, ..Default::default() },
|
||||
Params { semitones: 40.0, ..Default::default() },
|
||||
Params { semitones: f64::INFINITY, ..Default::default() },
|
||||
] {
|
||||
assert!(p.check().is_err(), "{p:?}");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn empty_scripts_are_refused() {
|
||||
assert_eq!(check_text(""), Err(TtsError::Empty));
|
||||
assert_eq!(check_text(" [pause 1s] ... "), Err(TtsError::Empty));
|
||||
assert_eq!(check_text(&"a".repeat(MAX_TEXT_BYTES + 1)), Err(TtsError::TextTooLong));
|
||||
assert!(check_text("Hi").is_ok());
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,105 @@
|
|||
//! Script parsing: speech text and pause markers.
|
||||
//!
|
||||
//! A pause marker is `[pause <n>s]` or `[pause <n>ms]` (case-insensitive, `n` a decimal number,
|
||||
//! spaces optional: `[pause 1.5s]`, `[Pause 500 ms]`). Pauses are capped at [`MAX_PAUSE_MS`]. A
|
||||
//! bracket that is not a well-formed marker is ordinary text. The Text to Speech panel's
|
||||
//! *Add pause* button inserts `[pause 1s]`.
|
||||
|
||||
/// Longest single pause.
|
||||
pub const MAX_PAUSE_MS: u32 = 10_000;
|
||||
|
||||
/// A piece of a script.
|
||||
#[derive(Clone, Debug, PartialEq)]
|
||||
pub enum Segment {
|
||||
Speech(String),
|
||||
Pause { millis: u32 },
|
||||
}
|
||||
|
||||
/// The marker *Add pause* inserts.
|
||||
pub const PAUSE_MARKER: &str = "[pause 1s]";
|
||||
|
||||
/// Split `text` into speech and pauses, in order. Empty speech pieces are dropped.
|
||||
pub fn parse(text: &str) -> Vec<Segment> {
|
||||
let mut out = Vec::new();
|
||||
let mut speech = String::new();
|
||||
let mut rest = text;
|
||||
while let Some(open) = rest.find('[') {
|
||||
let (before, from) = rest.split_at(open);
|
||||
speech.push_str(before);
|
||||
let close = from.find(']');
|
||||
match (close, close.and_then(|c| from.get(..=c)).and_then(pause_millis)) {
|
||||
(Some(close), Some(millis)) => {
|
||||
flush(&mut out, &mut speech);
|
||||
out.push(Segment::Pause { millis });
|
||||
rest = from.get(close + 1..).unwrap_or("");
|
||||
}
|
||||
_ => {
|
||||
speech.push('[');
|
||||
rest = from.get(1..).unwrap_or("");
|
||||
}
|
||||
}
|
||||
}
|
||||
speech.push_str(rest);
|
||||
flush(&mut out, &mut speech);
|
||||
out
|
||||
}
|
||||
|
||||
fn flush(out: &mut Vec<Segment>, speech: &mut String) {
|
||||
if !speech.trim().is_empty() {
|
||||
out.push(Segment::Speech(std::mem::take(speech)));
|
||||
}
|
||||
speech.clear();
|
||||
}
|
||||
|
||||
/// `[pause 1.5s]` → 1500. None if `marker` is not a pause marker.
|
||||
fn pause_millis(marker: &str) -> Option<u32> {
|
||||
let inner = marker.strip_prefix('[')?.strip_suffix(']')?.trim();
|
||||
let lower = inner.to_ascii_lowercase();
|
||||
let arg = lower.strip_prefix("pause")?.trim();
|
||||
let (num, scale) = if let Some(n) = arg.strip_suffix("ms") { (n.trim(), 1.0) } else { (arg.strip_suffix('s')?.trim(), 1000.0) };
|
||||
if num.is_empty() || num.len() > 12 || !num.chars().all(|c| c.is_ascii_digit() || c == '.') {
|
||||
return None;
|
||||
}
|
||||
let v: f64 = num.parse().ok()?;
|
||||
let ms = (v * scale).round();
|
||||
if !ms.is_finite() || ms < 0.0 {
|
||||
return None;
|
||||
}
|
||||
Some(ms.min(f64::from(MAX_PAUSE_MS)) as u32)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn sp(s: &str) -> Segment {
|
||||
Segment::Speech(s.into())
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn splits_speech_and_pauses() {
|
||||
assert_eq!(parse("Hello. [pause 1s] World"), vec![sp("Hello. "), Segment::Pause { millis: 1000 }, sp(" World")]);
|
||||
assert_eq!(parse("[Pause 500 ms]Go"), vec![Segment::Pause { millis: 500 }, sp("Go")]);
|
||||
assert_eq!(parse("a[pause 1.5s][pause 2s]b"), vec![sp("a"), Segment::Pause { millis: 1500 }, Segment::Pause { millis: 2000 }, sp("b")]);
|
||||
assert_eq!(parse(PAUSE_MARKER), vec![Segment::Pause { millis: 1000 }]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn malformed_markers_are_text() {
|
||||
assert_eq!(parse("[pause]"), vec![sp("[pause]")]);
|
||||
assert_eq!(parse("[pause 1x] a"), vec![sp("[pause 1x] a")]);
|
||||
assert_eq!(parse("[pause 1s"), vec![sp("[pause 1s")]);
|
||||
assert_eq!(parse("[note] [pause -1s]"), vec![sp("[note] [pause -1s]")]);
|
||||
assert_eq!(parse("[[pause 1s]]"), vec![sp("["), Segment::Pause { millis: 1000 }, sp("]")]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn hostile_inputs_never_panic_and_pauses_are_capped() {
|
||||
assert_eq!(parse("[pause 999999999999s]"), vec![Segment::Pause { millis: MAX_PAUSE_MS }]);
|
||||
assert_eq!(parse("[pause 1e9s]"), vec![sp("[pause 1e9s]")]);
|
||||
for s in ["", "[", "]", "[[[[", "日本語[pause 1s]👍", "\u{301}[", "[pause 1.2.3s]", "\0[pause 0s]"] {
|
||||
let _ = parse(s);
|
||||
}
|
||||
assert_eq!(parse(" "), vec![]);
|
||||
}
|
||||
}
|
||||
|
|
@ -16,6 +16,7 @@ filmcraft-color = { workspace = true }
|
|||
filmcraft-frame = { workspace = true }
|
||||
filmcraft-media = { workspace = true }
|
||||
filmcraft-project = { workspace = true }
|
||||
filmcraft-tts = { workspace = true }
|
||||
filmcraft-edit = { workspace = true }
|
||||
filmcraft-render = { workspace = true }
|
||||
filmcraft-text = { workspace = true }
|
||||
|
|
|
|||
|
|
@ -30,6 +30,7 @@ pub enum PanelKind {
|
|||
LumetriScopes,
|
||||
EssentialGraphics,
|
||||
EssentialSound,
|
||||
TextToSpeech,
|
||||
Properties,
|
||||
Text,
|
||||
Events,
|
||||
|
|
@ -39,7 +40,7 @@ pub enum PanelKind {
|
|||
}
|
||||
|
||||
impl PanelKind {
|
||||
pub const ALL: [PanelKind; 26] = [
|
||||
pub const ALL: [PanelKind; 27] = [
|
||||
PanelKind::Project,
|
||||
PanelKind::MediaBrowser,
|
||||
PanelKind::Libraries,
|
||||
|
|
@ -60,6 +61,7 @@ impl PanelKind {
|
|||
PanelKind::LumetriScopes,
|
||||
PanelKind::EssentialGraphics,
|
||||
PanelKind::EssentialSound,
|
||||
PanelKind::TextToSpeech,
|
||||
PanelKind::Properties,
|
||||
PanelKind::Text,
|
||||
PanelKind::Events,
|
||||
|
|
@ -89,6 +91,7 @@ impl PanelKind {
|
|||
PanelKind::LumetriScopes => "Lumetri Scopes",
|
||||
PanelKind::EssentialGraphics => "Essential Graphics",
|
||||
PanelKind::EssentialSound => "Essential Sound",
|
||||
PanelKind::TextToSpeech => "Text to Speech",
|
||||
PanelKind::Properties => "Properties",
|
||||
PanelKind::Text => "Text",
|
||||
PanelKind::Events => "Events",
|
||||
|
|
@ -268,7 +271,7 @@ pub fn workspace(name: &str) -> DockNode {
|
|||
hsplit(Ratio(0.5), tabs(&[AudioTrackMixer, Source, AudioClipMixer, EffectControls], 0), tabs(&[Program], 0)),
|
||||
hsplit(Ratio(0.3), tabs(&[Project, MediaBrowser, Effects, Markers, History], 0), bottom_editing()),
|
||||
),
|
||||
tabs(&[EssentialSound], 0),
|
||||
tabs(&[EssentialSound, TextToSpeech], 0),
|
||||
),
|
||||
"Captions and Graphics" => hsplit(
|
||||
FixedB(330.0),
|
||||
|
|
@ -284,7 +287,7 @@ pub fn workspace(name: &str) -> DockNode {
|
|||
hsplit(
|
||||
Ratio(0.5),
|
||||
tabs(&[Source, EffectControls, AudioClipMixer, Metadata, LumetriScopes, Text], 0),
|
||||
hsplit(Ratio(0.6), tabs(&[Program], 0), tabs(&[LumetriColor, EssentialGraphics, EssentialSound, Properties, AudioTrackMixer], 0)),
|
||||
hsplit(Ratio(0.6), tabs(&[Program], 0), tabs(&[LumetriColor, EssentialGraphics, EssentialSound, TextToSpeech, Properties, AudioTrackMixer], 0)),
|
||||
),
|
||||
hsplit(Ratio(0.26), tabs(&[Project, MediaBrowser, Libraries, Info, Effects, Markers, History], 0), bottom_editing()),
|
||||
),
|
||||
|
|
|
|||
|
|
@ -592,6 +592,11 @@ impl FilmcraftApp {
|
|||
}
|
||||
if !self.ui.dock.contains(p) {
|
||||
let near = match p {
|
||||
// a side panel beside Properties (or Essential Sound), never over the Program monitor
|
||||
PanelKind::TextToSpeech => [PanelKind::Properties, PanelKind::EssentialSound, PanelKind::EssentialGraphics]
|
||||
.into_iter()
|
||||
.find(|k| self.ui.dock.contains(*k))
|
||||
.unwrap_or(PanelKind::Program),
|
||||
PanelKind::LumetriColor | PanelKind::EssentialGraphics | PanelKind::EssentialSound | PanelKind::Properties => PanelKind::Program,
|
||||
PanelKind::Source
|
||||
| PanelKind::EffectControls
|
||||
|
|
@ -710,6 +715,34 @@ impl FilmcraftApp {
|
|||
panels::voiceover::on_play(self);
|
||||
}
|
||||
|
||||
/// Play the last Text to Speech preview (`Session::tts_preview`) on the audio output, stopping
|
||||
/// timeline playback first. Starting playback again takes the output back.
|
||||
pub fn play_tts_preview(&mut self) -> Result<f64, String> {
|
||||
let audio = self.session.tts_preview.clone().ok_or("nothing to preview")?;
|
||||
if self.playback.playing {
|
||||
self.stop();
|
||||
}
|
||||
let a = self.audio.as_mut().ok_or("no audio output (check Settings ▸ Audio Hardware)")?;
|
||||
let sr = a.sample_rate().max(1);
|
||||
a.stop();
|
||||
let step = f64::from(audio.sample_rate.max(1)) / f64::from(sr);
|
||||
let seconds = audio.seconds();
|
||||
let mut pos = 0.0f64;
|
||||
let fill = Box::new(move |buf: &mut [f32], ch: usize| {
|
||||
for frame in buf.chunks_mut(ch.max(1)) {
|
||||
let i = pos as usize;
|
||||
let f = (pos - i as f64) as f32;
|
||||
let x0 = audio.samples.get(i).copied().unwrap_or(0.0);
|
||||
let x1 = audio.samples.get(i + 1).copied().unwrap_or(0.0);
|
||||
frame.fill(x0 + (x1 - x0) * f);
|
||||
pos += step;
|
||||
}
|
||||
});
|
||||
a.start(fill)?;
|
||||
self.playback.audio_clock = false;
|
||||
Ok(seconds)
|
||||
}
|
||||
|
||||
pub fn stop(&mut self) {
|
||||
self.playback.playing = false;
|
||||
self.playback.stop_at = None;
|
||||
|
|
@ -1232,6 +1265,8 @@ impl FilmcraftApp {
|
|||
}
|
||||
self.handle_shortcuts(&ctx);
|
||||
self.advance_playback(&ctx);
|
||||
// Text to Speech: a selected narration clip loads into the panel
|
||||
panels::tts::follow_selection(self);
|
||||
let t = self.tokens;
|
||||
let full = ui.max_rect();
|
||||
ui.painter().rect_filled(full, 0.0, t.app_bg);
|
||||
|
|
|
|||
|
|
@ -45,6 +45,7 @@ pub mod timeline_automation;
|
|||
pub mod timeline_captions;
|
||||
pub mod tools;
|
||||
pub mod trim_monitor;
|
||||
pub mod tts;
|
||||
pub mod voiceover;
|
||||
pub mod workspaces;
|
||||
|
||||
|
|
@ -94,6 +95,7 @@ pub fn show(app: &mut FilmcraftApp, ui: &mut egui::Ui, p: PanelKind, rect: Rect)
|
|||
PanelKind::ReferenceMonitor => reference::show(app, ui, rect),
|
||||
PanelKind::Text => text::show(app, ui, rect),
|
||||
PanelKind::EssentialSound => essential_sound::show(app, ui, rect),
|
||||
PanelKind::TextToSpeech => tts::show(app, ui, rect),
|
||||
other => crate::dock::placeholder(ui, rect, &app.tokens, &format!("{} — coming in a later milestone", other.title())),
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -0,0 +1,288 @@
|
|||
//! Text to Speech panel: write what the narrator says, pick a language, voice, vocal pitch and pace,
|
||||
//! hear it, and save it as a narration clip. Selecting a narration clip on the timeline loads its
|
||||
//! script and settings here (and brings the panel forward), and Save regenerates it in place.
|
||||
//! Every action is a `tts.*` command, so the panel, MCP and the control channel share one path.
|
||||
//!
|
||||
//! Automation ids: `tts.language`, `tts.voice`, `tts.hearVoice`, `tts.advanced` (twirl),
|
||||
//! `tts.pitch`, `tts.pace`, `tts.text`, `tts.addPause`, `tts.preview`, `tts.save`, `tts.new`.
|
||||
|
||||
use egui::{Align2, Rect, Sense, Stroke, StrokeKind, pos2, vec2};
|
||||
use filmcraft_project::{ClipId, VocalPitch};
|
||||
use serde::{Deserialize, Serialize};
|
||||
use serde_json::json;
|
||||
|
||||
use crate::FilmcraftApp;
|
||||
use crate::icons::{self, Icon};
|
||||
use crate::theme::Tokens;
|
||||
|
||||
const PAD: f32 = 14.0;
|
||||
const FOOTER_H: f32 = 48.0;
|
||||
|
||||
/// What the panel is writing (UI state, serde so agents can read it).
|
||||
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
|
||||
#[serde(default, rename_all = "camelCase")]
|
||||
pub struct TtsDraft {
|
||||
pub text: String,
|
||||
pub language: String,
|
||||
pub voice: String,
|
||||
pub pitch: VocalPitch,
|
||||
pub pace: f64,
|
||||
pub advanced_open: bool,
|
||||
/// The narration clip and item the draft was loaded from (None: a new narration).
|
||||
pub loaded_from: Option<(u64, u64)>,
|
||||
}
|
||||
|
||||
impl Default for TtsDraft {
|
||||
fn default() -> Self {
|
||||
TtsDraft {
|
||||
text: String::new(),
|
||||
language: "en-US".into(),
|
||||
voice: filmcraft_tts::default_voice_id().into(),
|
||||
pitch: VocalPitch::Default,
|
||||
pace: 1.0,
|
||||
advanced_open: true,
|
||||
loaded_from: None,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl TtsDraft {
|
||||
fn params(&self) -> serde_json::Value {
|
||||
json!({"text": self.text, "voice": self.voice, "pitch": self.pitch.id(), "pace": self.pace})
|
||||
}
|
||||
}
|
||||
|
||||
/// The selected narration clip and its item, if any.
|
||||
fn selected_narration(app: &FilmcraftApp) -> Option<(ClipId, u64)> {
|
||||
let clip = filmcraft_engine::narration::target_clip(&app.session, &serde_json::Value::Null)?;
|
||||
let item = app.session.active_sequence()?.find_item(clip)?.1.item;
|
||||
Some((clip, item.0))
|
||||
}
|
||||
|
||||
/// Load the selected narration clip into the draft when the selection moves to a different one,
|
||||
/// and bring the panel forward. Runs every frame, also when the panel is hidden.
|
||||
pub fn follow_selection(app: &mut FilmcraftApp) {
|
||||
let sel = selected_narration(app);
|
||||
match sel {
|
||||
Some((clip, item)) if app.ui.tts.loaded_from != Some((clip.0, item)) => {
|
||||
let first_time = app.ui.tts.loaded_from.map(|(c, _)| c) != Some(clip.0);
|
||||
if let Some(n) = app.session.project.narrations.get(&filmcraft_project::ItemId(item)).cloned() {
|
||||
let d = &mut app.ui.tts;
|
||||
d.text = n.text;
|
||||
d.language = n.language;
|
||||
d.voice = n.voice;
|
||||
d.pitch = n.pitch;
|
||||
d.pace = n.pace;
|
||||
d.loaded_from = Some((clip.0, item));
|
||||
if first_time {
|
||||
app.show_panel(crate::dock::PanelKind::TextToSpeech);
|
||||
}
|
||||
}
|
||||
}
|
||||
None if app.ui.tts.loaded_from.is_some() => {
|
||||
// deselected: keep the text so it can be saved as a new narration
|
||||
app.ui.tts.loaded_from = None;
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
|
||||
enum Action {
|
||||
Run(&'static str, serde_json::Value, bool),
|
||||
New,
|
||||
}
|
||||
|
||||
pub fn show(app: &mut FilmcraftApp, ui: &mut egui::Ui, rect: Rect) {
|
||||
let t = app.tokens;
|
||||
let painter = ui.painter().clone();
|
||||
painter.rect_filled(rect, 0.0, t.panel_bg);
|
||||
let voices = filmcraft_tts::voices();
|
||||
let editing = app.ui.tts.loaded_from.map(|(c, _)| c);
|
||||
let mut actions: Vec<Action> = Vec::new();
|
||||
let auto = &mut app.auto;
|
||||
let d = &mut app.ui.tts;
|
||||
|
||||
let body = Rect::from_min_max(pos2(rect.min.x + PAD, rect.min.y + 8.0), pos2(rect.max.x - PAD, rect.max.y - FOOTER_H));
|
||||
let mut child = ui.new_child(egui::UiBuilder::new().max_rect(body).id_salt("tts-body"));
|
||||
egui::ScrollArea::vertical().auto_shrink([false, false]).show(&mut child, |ui| {
|
||||
ui.set_max_width(ui.available_width() - 6.0);
|
||||
let w = ui.available_width();
|
||||
// mode line: editing a clip, or writing a new narration
|
||||
ui.horizontal(|ui| {
|
||||
let mode = if editing.is_some() { "Editing the selected narration" } else { "New narration" };
|
||||
ui.label(egui::RichText::new(mode).color(t.text_dim).size(11.5));
|
||||
if editing.is_some() {
|
||||
ui.with_layout(egui::Layout::right_to_left(egui::Align::Center), |ui| {
|
||||
let b = ui.small_button("New");
|
||||
auto.add("tts.new", visible(ui, b.rect), "New narration");
|
||||
if b.clicked() {
|
||||
actions.push(Action::New);
|
||||
}
|
||||
});
|
||||
}
|
||||
});
|
||||
ui.add_space(6.0);
|
||||
label(ui, &t, "Language");
|
||||
let lang_name = filmcraft_tts::LANGUAGES.iter().find(|(id, _)| *id == d.language).map_or(d.language.as_str(), |(_, n)| n);
|
||||
let cb = egui::ComboBox::from_id_salt("tts-language").selected_text(lang_name).width(w).show_ui(ui, |ui| {
|
||||
for (id, name) in filmcraft_tts::LANGUAGES {
|
||||
if ui.selectable_label(d.language == *id, *name).clicked() {
|
||||
d.language = id.to_string();
|
||||
}
|
||||
}
|
||||
});
|
||||
auto.add("tts.language", visible(ui, cb.response.rect), "Language");
|
||||
ui.add_space(10.0);
|
||||
label(ui, &t, "Voice");
|
||||
let voice_name = voices.iter().find(|v| v.id == d.voice).map_or(d.voice.as_str(), |v| v.name);
|
||||
let cb = egui::ComboBox::from_id_salt("tts-voice").selected_text(voice_name).width(w).show_ui(ui, |ui| {
|
||||
for v in voices.iter().filter(|v| v.language == d.language) {
|
||||
let r = ui.selectable_label(d.voice == v.id, v.name).on_hover_text(v.description);
|
||||
if r.clicked() {
|
||||
d.voice = v.id.to_string();
|
||||
}
|
||||
}
|
||||
});
|
||||
auto.add("tts.voice", visible(ui, cb.response.rect), "Voice");
|
||||
ui.add_space(4.0);
|
||||
let (r, resp) = ui.allocate_exact_size(vec2(130.0, 22.0), Sense::click());
|
||||
let col = if resp.hovered() { t.accent_hover } else { t.accent };
|
||||
icons::paint(ui.painter(), Rect::from_center_size(pos2(r.min.x + 8.0, r.center().y), vec2(14.0, 14.0)), Icon::Play, col);
|
||||
ui.painter().text(pos2(r.min.x + 22.0, r.center().y), Align2::LEFT_CENTER, "Hear this voice", Tokens::semibold(12.0), col);
|
||||
auto.add("tts.hearVoice", visible(ui, r), "Hear this voice");
|
||||
if resp.clicked() {
|
||||
actions.push(Action::Run("tts.preview", json!({"sample": true, "voice": d.voice, "pitch": d.pitch.id(), "pace": d.pace}), true));
|
||||
}
|
||||
ui.add_space(8.0);
|
||||
let (hr, open) = crate::widgets::section_header(ui, egui::Id::new("tts-advanced"), "Advanced", d.advanced_open, &t, true);
|
||||
auto.add("tts.advanced", visible(ui, hr.rect), "Advanced");
|
||||
d.advanced_open = open;
|
||||
if open {
|
||||
ui.add_space(4.0);
|
||||
label(ui, &t, "Vocal pitch");
|
||||
let cb = egui::ComboBox::from_id_salt("tts-pitch").selected_text(d.pitch.label()).width(w).show_ui(ui, |ui| {
|
||||
for p in VocalPitch::ALL {
|
||||
if ui.selectable_label(d.pitch == p, p.label()).clicked() {
|
||||
d.pitch = p;
|
||||
}
|
||||
}
|
||||
});
|
||||
auto.add("tts.pitch", visible(ui, cb.response.rect), "Vocal pitch");
|
||||
ui.add_space(10.0);
|
||||
label(ui, &t, "Pace");
|
||||
ui.spacing_mut().slider_width = w - 56.0;
|
||||
let s = ui.add(egui::Slider::new(&mut d.pace, filmcraft_tts::MIN_PACE..=filmcraft_tts::MAX_PACE).step_by(0.05).fixed_decimals(2).suffix("×"));
|
||||
auto.add("tts.pace", visible(ui, s.rect), "Pace");
|
||||
let (r, _) = ui.allocate_exact_size(vec2(w - 56.0, 14.0), Sense::hover());
|
||||
for (x, l) in [(0.0, "0.5×"), (1.0 / 3.0, "1×"), (1.0, "2×")] {
|
||||
let px = r.min.x + 6.0 + (r.width() - 12.0) * x;
|
||||
ui.painter().text(pos2(px, r.center().y), Align2::CENTER_CENTER, l, Tokens::ui(10.5), t.text_faint);
|
||||
}
|
||||
}
|
||||
ui.add_space(12.0);
|
||||
ui.horizontal(|ui| {
|
||||
ui.label(egui::RichText::new("Text").color(t.text).size(12.5).strong());
|
||||
ui.with_layout(egui::Layout::right_to_left(egui::Align::Center), |ui| {
|
||||
let (ir, i) = ui.allocate_exact_size(vec2(14.0, 14.0), Sense::hover());
|
||||
icons::paint(ui.painter(), ir, Icon::Info, t.text_faint);
|
||||
i.on_hover_text("Add pause inserts [pause 1s]. Change the number for a longer or shorter pause, e.g. [pause 500ms] or [pause 2s].");
|
||||
let b = ui.small_button("Add pause");
|
||||
auto.add("tts.addPause", visible(ui, b.rect), "Add pause");
|
||||
if b.clicked() {
|
||||
if !d.text.is_empty() && !d.text.ends_with(' ') {
|
||||
d.text.push(' ');
|
||||
}
|
||||
d.text.push_str(filmcraft_tts::script::PAUSE_MARKER);
|
||||
d.text.push(' ');
|
||||
}
|
||||
});
|
||||
});
|
||||
ui.add_space(4.0);
|
||||
let te = egui::TextEdit::multiline(&mut d.text)
|
||||
.hint_text("Type what you want the voice to say")
|
||||
.desired_width(w)
|
||||
.desired_rows(8)
|
||||
.char_limit(filmcraft_tts::MAX_TEXT_BYTES);
|
||||
let r = ui.add(te);
|
||||
auto.add("tts.text", visible(ui, r.rect), "Text");
|
||||
});
|
||||
|
||||
// footer: Preview | Save
|
||||
let fy = rect.max.y - FOOTER_H + 10.0;
|
||||
let half = (rect.width() - PAD * 2.0 - 10.0) / 2.0;
|
||||
let can_say = filmcraft_tts::check_text(&d.text).is_ok();
|
||||
let pr = Rect::from_min_size(pos2(rect.min.x + PAD, fy), vec2(half, 28.0));
|
||||
if button(ui, &t, pr, Icon::Play, "Preview", false, can_say, "tts-preview").clicked() && can_say {
|
||||
actions.push(Action::Run("tts.preview", d.params(), true));
|
||||
}
|
||||
auto.add("tts.preview", pr, "Preview");
|
||||
let sr = Rect::from_min_size(pos2(pr.max.x + 10.0, fy), vec2(half, 28.0));
|
||||
let can_save = can_say && app.session.active_sequence().is_some();
|
||||
if button(ui, &t, sr, Icon::Plus, if editing.is_some() { "Save" } else { "Add to timeline" }, true, can_save, "tts-save").clicked() && can_save {
|
||||
let mut p = d.params();
|
||||
match editing {
|
||||
Some(c) => {
|
||||
p["clip"] = json!(c);
|
||||
actions.push(Action::Run("tts.edit", p, false));
|
||||
}
|
||||
None => actions.push(Action::Run("tts.create", p, false)),
|
||||
}
|
||||
}
|
||||
auto.add("tts.save", sr, "Save");
|
||||
|
||||
for a in actions {
|
||||
match a {
|
||||
Action::New => {
|
||||
let keep = (app.ui.tts.language.clone(), app.ui.tts.voice.clone(), app.ui.tts.pitch, app.ui.tts.pace);
|
||||
app.session.state.selection.clear();
|
||||
app.ui.tts = TtsDraft { language: keep.0, voice: keep.1, pitch: keep.2, pace: keep.3, ..Default::default() };
|
||||
}
|
||||
Action::Run(cmd, p, play) => match app.session.execute(cmd, p) {
|
||||
Ok(r) => {
|
||||
if play {
|
||||
if let Err(e) = app.play_tts_preview() {
|
||||
app.ui.status = e;
|
||||
}
|
||||
} else {
|
||||
app.ui.status = match cmd {
|
||||
"tts.create" => format!("Narration added on {}", r["track"].as_str().unwrap_or("an audio track")),
|
||||
_ => "Narration updated".into(),
|
||||
};
|
||||
}
|
||||
}
|
||||
Err(e) => app.ui.status = e.to_string(),
|
||||
},
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The part of `r` the scroll area shows (what a click can reach).
|
||||
fn visible(ui: &egui::Ui, r: Rect) -> Rect {
|
||||
r.intersect(ui.clip_rect())
|
||||
}
|
||||
|
||||
fn label(ui: &mut egui::Ui, t: &Tokens, text: &str) {
|
||||
ui.label(egui::RichText::new(text).color(t.text).size(12.5).strong());
|
||||
ui.add_space(2.0);
|
||||
}
|
||||
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
fn button(ui: &mut egui::Ui, t: &Tokens, r: Rect, icon: Icon, text: &str, primary: bool, active: bool, id: &str) -> egui::Response {
|
||||
let resp = ui.interact(r, egui::Id::new(id), Sense::click());
|
||||
let p = ui.painter();
|
||||
let (bg, fg) = match (primary, active) {
|
||||
(true, true) => (if resp.hovered() { t.accent_hover } else { t.accent }, egui::Color32::WHITE),
|
||||
(_, false) => (t.field_bg, t.text_faint),
|
||||
(false, true) => (if resp.hovered() { t.hover } else { t.field_bg }, t.text),
|
||||
};
|
||||
p.rect_filled(r, 4.0, bg);
|
||||
if !primary {
|
||||
p.rect_stroke(r, 4.0, Stroke::new(1.0, t.field_border), StrokeKind::Inside);
|
||||
}
|
||||
let g = p.layout_no_wrap(text.to_string(), Tokens::semibold(12.5), fg);
|
||||
let total = 18.0 + g.size().x;
|
||||
let x0 = r.center().x - total / 2.0;
|
||||
icons::paint(p, Rect::from_center_size(pos2(x0 + 7.0, r.center().y), vec2(14.0, 14.0)), icon, fg);
|
||||
p.galley(pos2(x0 + 18.0, r.center().y - g.size().y / 2.0), g, fg);
|
||||
resp
|
||||
}
|
||||
|
|
@ -392,6 +392,9 @@ pub struct UiState {
|
|||
/// Essential Sound sub-tab: "Edit" or "Browse".
|
||||
#[serde(default)]
|
||||
pub essential_sound_tab: String,
|
||||
/// Text to Speech panel: the script and voice settings being written or edited.
|
||||
#[serde(default)]
|
||||
pub tts: crate::panels::tts::TtsDraft,
|
||||
/// Export mode: settings, preset, destination, range, the Preset Manager and Quick Export.
|
||||
#[serde(default)]
|
||||
pub export: crate::panels::export_mode::ExportUi,
|
||||
|
|
@ -807,6 +810,7 @@ impl Default for UiState {
|
|||
show_scopes: false,
|
||||
status: String::new(),
|
||||
essential_sound_tab: "Edit".into(),
|
||||
tts: Default::default(),
|
||||
export: Default::default(),
|
||||
text_tab: captions_tab(),
|
||||
caption_search: String::new(),
|
||||
|
|
|
|||
|
|
@ -0,0 +1,190 @@
|
|||
//! Headless UI tests of the Text to Speech panel: typing a script, Add pause, adding the narration
|
||||
//! to the timeline, clicking the narration clip to load it back, and saving an edit in place.
|
||||
|
||||
use std::sync::mpsc::{Sender, channel};
|
||||
|
||||
use egui_kittest::Harness;
|
||||
use filmcraft_engine::Session;
|
||||
use filmcraft_ui_egui::FilmcraftApp;
|
||||
use filmcraft_ui_egui::control::ControlRequest;
|
||||
use serde_json::{Value, json};
|
||||
|
||||
struct Driver {
|
||||
harness: Harness<'static, FilmcraftApp>,
|
||||
tx: Sender<ControlRequest>,
|
||||
}
|
||||
|
||||
impl Driver {
|
||||
fn demo(scratch: &str) -> Self {
|
||||
let mut session = Session::default();
|
||||
session.execute("file.openDemoProject", json!({})).expect("demo project");
|
||||
session.execute("file.projectSettings.scratchDisks", json!({"captured": scratch})).expect("scratch disk");
|
||||
let (tx, rx) = channel();
|
||||
let app = FilmcraftApp::new(session).with_control(rx);
|
||||
let harness = Harness::builder().with_size(egui::vec2(1600.0, 980.0)).with_max_steps(10_000).build_eframe(move |_cc| app);
|
||||
let mut d = Driver { harness, tx };
|
||||
d.frames(4);
|
||||
d
|
||||
}
|
||||
|
||||
fn frames(&mut self, n: usize) {
|
||||
for _ in 0..n {
|
||||
let ctx = self.harness.ctx.clone();
|
||||
let mut raw = std::mem::take(self.harness.input_mut());
|
||||
eframe::App::raw_input_hook(self.harness.state_mut(), &ctx, &mut raw);
|
||||
*self.harness.input_mut() = raw;
|
||||
self.harness.step();
|
||||
}
|
||||
}
|
||||
|
||||
fn call(&mut self, method: &str, params: Value) -> Value {
|
||||
let (req, reply) = ControlRequest::new(method, params.clone());
|
||||
self.tx.send(req).unwrap();
|
||||
for _ in 0..600 {
|
||||
self.frames(1);
|
||||
if let Ok(v) = reply.try_recv() {
|
||||
return v;
|
||||
}
|
||||
}
|
||||
panic!("no reply to {method} {params}");
|
||||
}
|
||||
|
||||
fn ok(&mut self, method: &str, params: Value) -> Value {
|
||||
let v = self.call(method, params.clone());
|
||||
assert_eq!(v["ok"], json!(true), "{method} {params} failed: {v}");
|
||||
v["result"].clone()
|
||||
}
|
||||
|
||||
fn exec(&mut self, command: &str, params: Value) -> Value {
|
||||
self.ok("engine.execute", json!({"command": command, "params": params}))
|
||||
}
|
||||
|
||||
fn click(&mut self, id: &str) {
|
||||
self.ok("ui.click", json!({"id": id}));
|
||||
self.frames(3);
|
||||
}
|
||||
|
||||
fn type_text(&mut self, text: &str) {
|
||||
self.ok("ui.type", json!({"text": text}));
|
||||
self.frames(3);
|
||||
}
|
||||
|
||||
fn ui(&mut self) -> Value {
|
||||
self.ok("ui.inspect", json!({}))
|
||||
}
|
||||
|
||||
fn draft(&mut self) -> Value {
|
||||
self.ui()["ui"]["tts"].clone()
|
||||
}
|
||||
|
||||
fn ids(&mut self, prefix: &str) -> Vec<String> {
|
||||
let v = self.ok("ui.elements", json!({"prefix": prefix}));
|
||||
v.as_array().unwrap().iter().filter_map(|e| e["id"].as_str().map(str::to_string)).collect()
|
||||
}
|
||||
|
||||
fn narration_clips(&mut self) -> Vec<u64> {
|
||||
let seq = self.exec("sequence.inspect", json!({}));
|
||||
let mut out = Vec::new();
|
||||
for t in seq["audio"].as_array().into_iter().flatten() {
|
||||
for it in t["items"].as_array().into_iter().flatten() {
|
||||
let id = it["clip"].as_u64().unwrap();
|
||||
if self.call("engine.execute", json!({"command": "tts.inspect", "params": {"clip": id}}))["ok"] == json!(true) {
|
||||
out.push(id);
|
||||
}
|
||||
}
|
||||
}
|
||||
out
|
||||
}
|
||||
}
|
||||
|
||||
fn scratch(name: &str) -> String {
|
||||
let d = std::env::temp_dir().join(format!("filmcraft-tts-ui-{}-{name}", std::process::id()));
|
||||
let _ = std::fs::remove_dir_all(&d);
|
||||
std::fs::create_dir_all(&d).unwrap();
|
||||
d.to_string_lossy().into_owned()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn write_add_click_to_edit_and_save_in_place() {
|
||||
let dir = scratch("flow");
|
||||
let mut d = Driver::demo(&dir);
|
||||
d.exec("window.panel.TextToSpeech", json!({}));
|
||||
d.frames(3);
|
||||
// the panel opens beside Properties, leaving the Program monitor visible
|
||||
assert!(!d.ids("program.").is_empty(), "Program monitor still shown");
|
||||
let ids = d.ids("tts.");
|
||||
for id in ["tts.language", "tts.voice", "tts.hearVoice", "tts.advanced", "tts.pitch", "tts.pace", "tts.text", "tts.addPause", "tts.preview", "tts.save"] {
|
||||
assert!(ids.iter().any(|i| i == id), "missing {id} in {ids:?}");
|
||||
}
|
||||
// type a script and add a pause
|
||||
d.click("tts.text");
|
||||
d.type_text("Hello from the panel.");
|
||||
d.click("tts.addPause");
|
||||
d.click("tts.text");
|
||||
d.ok("ui.key", json!({"key": "End"}));
|
||||
d.type_text("Goodbye.");
|
||||
let text = d.draft()["text"].as_str().unwrap().to_string();
|
||||
assert!(text.starts_with("Hello from the panel. [pause 1s]") && text.ends_with("Goodbye."), "{text:?}");
|
||||
// add it to the timeline
|
||||
d.exec("playhead.set", json!({"seconds": 1.0}));
|
||||
d.click("tts.save");
|
||||
let clips = d.narration_clips();
|
||||
let status = d.ui()["ui"]["status"].clone();
|
||||
assert_eq!(clips.len(), 1, "one narration clip; status {status}");
|
||||
let clip = clips[0];
|
||||
let n = d.exec("tts.inspect", json!({"clip": clip}));
|
||||
assert_eq!(n["narration"]["text"].as_str().unwrap(), text);
|
||||
assert_eq!(d.ui()["selection"], json!([clip]), "the new clip is selected");
|
||||
assert_eq!(d.draft()["loadedFrom"][0], json!(clip), "the panel now edits it");
|
||||
// deselect and start fresh, then click the clip on the timeline: the panel loads it back
|
||||
d.exec("edit.deselectAll", json!({}));
|
||||
d.frames(3);
|
||||
assert!(d.draft()["loadedFrom"].is_null());
|
||||
d.click("tts.text");
|
||||
d.ok("ui.key", json!({"key": "Cmd+A"}));
|
||||
d.type_text("Something else");
|
||||
let at = d.ok("ui.timeline.locate", json!({"clip": clip}));
|
||||
d.ok("ui.click", json!({"x": at["x"], "y": at["y"]}));
|
||||
d.frames(4);
|
||||
assert_eq!(d.ui()["selection"], json!([clip]));
|
||||
assert_eq!(d.draft()["text"].as_str().unwrap(), text, "clicking the clip loads its script");
|
||||
// change the script and save: same clip, new audio, one undo step
|
||||
let before = d.exec("tts.inspect", json!({"clip": clip}))["item"].clone();
|
||||
d.click("tts.text");
|
||||
d.ok("ui.key", json!({"key": "Cmd+A"}));
|
||||
d.type_text("A new script for the same clip.");
|
||||
d.click("tts.save");
|
||||
let after = d.exec("tts.inspect", json!({"clip": clip}));
|
||||
assert_eq!(after["narration"]["text"], "A new script for the same clip.");
|
||||
assert_ne!(after["item"], before);
|
||||
assert_eq!(d.narration_clips(), vec![clip]);
|
||||
let hist = d.exec("history.list", json!({}));
|
||||
assert_eq!(hist["undo"].as_array().unwrap().last().unwrap(), "Edit Narration");
|
||||
d.exec("edit.undo", json!({}));
|
||||
d.frames(3);
|
||||
assert_eq!(d.exec("tts.inspect", json!({"clip": clip}))["narration"]["text"].as_str().unwrap(), text);
|
||||
let _ = std::fs::remove_dir_all(&dir);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn settings_controls_and_hear_this_voice() {
|
||||
let dir = scratch("controls");
|
||||
let mut d = Driver::demo(&dir);
|
||||
d.exec("window.panel.TextToSpeech", json!({}));
|
||||
d.frames(3);
|
||||
// Save is disabled with an empty script
|
||||
d.click("tts.save");
|
||||
assert!(d.narration_clips().is_empty());
|
||||
// Hear this voice runs a preview (the headless harness has no audio output: a status line)
|
||||
d.click("tts.hearVoice");
|
||||
let status = d.ui()["ui"]["status"].as_str().unwrap_or_default().to_string();
|
||||
assert!(status.contains("audio output") || status.is_empty(), "{status}");
|
||||
// Advanced folds away and back
|
||||
assert_eq!(d.draft()["advancedOpen"], true);
|
||||
d.click("tts.advanced");
|
||||
assert_eq!(d.draft()["advancedOpen"], false);
|
||||
assert!(!d.ids("tts.").iter().any(|i| i == "tts.pitch"));
|
||||
d.click("tts.advanced");
|
||||
assert!(d.ids("tts.").iter().any(|i| i == "tts.pitch"));
|
||||
let _ = std::fs::remove_dir_all(&dir);
|
||||
}
|
||||
|
|
@ -435,6 +435,10 @@ stereo / mono track ──5.1 panner───┼─► 5.1 submix / 5.1 Mix ─
|
|||
|
||||
**Voice-over recording** (`engine::voiceover`, `audio.voiceover.*`). The record point R is the playhead, or the In point when In/Out are set (punch-in; punch-out at Out). Playback starts the pre-roll before it (C = max(0, R − pre-roll)) and the input is captured from C. When the UI's audio clock really starts, `audio.voiceover.sync` restarts the capture there. `audio.voiceover.stop` keeps the audio from R to min(stop, Out), writes a mono 32-bit float WAV `<Name> <n>.wav` (Scratch Disks ▸ Captured, else next to the project, else the data or temporary directory), imports it and overwrites it onto the record track at R as one undo step. The record track is the given one, else the record-armed one, else the first targeted one. Input goes through the `AudioInput` trait: cpal in the desktop app (`apps/filmcraft/src/audio_in.rs`, on its own thread), and `SyntheticInput` headless, which produces exactly the samples the timeline asks for so recordings land sample-accurately. Voice-Over Record Settings (Source, Input channel, Name, Countdown Sound Cues, Pre-/Post-roll) are preferences (`voiceOver`). The track header's microphone records or stops, and a right-click opens the settings dialog. Countdown beeps (1 kHz, 100 ms, each whole second of pre-roll and at R) are mixed into playback, an overlay counts down, and playback stops at Out + post-roll.
|
||||
|
||||
### 5.1.2a Text to Speech (narrations)
|
||||
|
||||
**Narrations** (`engine::narration`, `tts.*`; voices in `filmcraft-tts`). The Text to Speech panel (Window ▸ Text to Speech) writes a script with pause markers (`[pause 1s]`), a voice, a vocal pitch and a pace; `tts.create` synthesizes it, writes `Narration <n>.wav` (mono 32-bit float, 24 kHz, where voice-over recordings go), imports it and places it at the playhead on the first free targeted audio track, else the first free audio track, else a new one, as one undo step. The settings are kept per generated item in `Project::narrations` (schema v13). Selecting a narration clip loads it into the panel; `tts.edit` writes a new file (beside the previous one) and swaps the clip to it, keeping the clip's start, duration and speed: longer speech is kept whole in the file but plays only to the clip's end; shorter speech is padded with silence to the clip's length. The old item and narration stay, so undo is exact. `tts.preview` synthesizes without changing the project and the UI plays `Session::tts_preview` on the audio output. Built-in voices are an original formant synthesizer (robotic, no download); neural voices come later.
|
||||
|
||||
### 5.1.3 Remix
|
||||
|
||||
```text
|
||||
|
|
|
|||
|
|
@ -30,6 +30,7 @@ FILMCRAFT_REQUIRE_ORACLES=1 cargo test --workspace # CI: missing ffmpeg fails
|
|||
| 5.1 / multichannel | `crates/audio-dsp/src/channels.rs`, `crates/render/src/mixer_tests.rs`, `crates/export/src/surround_tests.rs`, `crates/engine/src/mixer_tests.rs`, `crates/ui-egui/tests/audio51_ui.rs` | BS.775 downmix / upmix coefficients exact, mixdown types; 5.1 panner point gains equal-power (Σg² = 1 over a 21×21 grid × 3 centre settings), speaker positions exact, 5.1 through the default puck bit-exact identity; graph: stereo/mono tracks into a 5.1 Mix (L/R, C, rear right, room centre −6 dB each, LFE = mean), 5.1 track through a 5.1 Mix and folded into stereo, a 5.1 submix, `pan51` automation block-invariant, six-channel meters; export: 6-ch WAV (`WAVE_FORMAT_EXTENSIBLE`, mask 0x3F) and ProRes MOV (6 × 24-bit PCM) decoded by ffmpeg **sample-exact**, ffprobe `channels` 6 / `channel_layout` 5.1; AAC 5.1 in MP4: every decoded channel's own tone ≥ 30 dB above the others' (no swapped channels), stereo export of a 5.1 Mix = BS.775 within 2·10⁻⁶; `sequence.settings {mix}`, `file.newSequence {mix, trackType}`, panner lanes, Write pass records the puck; UI: puck drag (one undo step), Center / LFE, six meter bars |
|
||||
| Audio Clip Mixer automation, Track Mixer view | `crates/engine/src/mixer_tests.rs`, `crates/ui-egui/tests/audio51_ui.rs` | Touch: a 201-point fader ramp thins to ≤ 6 clip keyframes, values within 0.1 dB, live override while held, values outside the gesture unchanged, one undo step; Latch holds until stop; Write records volume and pan from the start; a gesture across two clips writes both; Read commits once without a pass; UI: Clip Mixer mode dropdown, a recorded fader drag writes clip keyframes; Show/Hide Tracks, Meter Input(s) Only, transport record button |
|
||||
| Voice-over | `crates/engine/src/voiceover_tests.rs`, `crates/ui-egui/tests/voiceover_ui.rs` | synthetic click train lands in `mix_sequence` exactly at R + k·period (pre-roll dropped, exact sample count), punch-in between In/Out (ramp source, every sample checked), one undo step + undo/redo, take numbering, record-arm/targeting track choice, sync/discard/zero-length/locked cases, cue times; UI: right-click Mic → Voice-Over Record Settings dialog (edits persist on OK, Cancel discards), the Mic button starts and stops a take on its track with pre-roll and overlay |
|
||||
| Text to Speech | `crates/tts/src/*.rs`, `crates/engine/src/narration_tests.rs`, `crates/ui-egui/tests/tts_ui.rs` | built-in voices: determinism, finite samples, exact-silence pause markers, pace scales length, measured pitch per voice and per semitone, hostile scripts; engine: narration placed at the playhead with clip length = speech (sample-exact), never overwrites (busy tracks → next free track → new track), occupied/locked/unknown track refused with no change, edit keeps start/length/speed (longer speech: whole speech in the file; shorter: file padded with exact zeros), edits write beside the previous file, undo/redo restore the project exactly, enabled only with a narration clip (or `clip`), 16 hostile parameter sets refused without change, preview changes nothing, save/reload keeps narrations, the narration is audible in `mix_sequence`; UI: type + Add pause + Add to timeline, the new clip is selected and loaded, clicking the clip on the timeline loads its script, Save edits it in place as one undo step, Advanced twirl, panel opens beside Properties |
|
||||
| Remix | `crates/audio-dsp/src/remix.rs`, `crates/render/src/remix.rs`, `crates/engine/src/remix_tests.rs`, `crates/ui-egui/tests/remix_ui.rs` | generated rhythmic music (Rust-synthesised drums, bass, chord sections at 90–128 BPM, 22.05/44.1/48 kHz): tempo within 0.3 BPM, every beat within 10 ms (measured ≤ 8.9 ms); remix targets from 0.4× to 2.2× the source within one beat (measured ≤ 12 ms at default sliders), every cut on a detected beat, intro and outro kept, pieces ≥ 4 beats, deterministic; silence, tone and noise refused; rendering independent of request cuts, source-exact outside the 20 ms equal-power crossfades, no clicks; the engine's mix equals the source pieces (< 1e-5); undo, redo and revert exact; overlaps refused; Remix Properties dialog and Remix tool drag driven by automation id |
|
||||
| Oracle | `crates/*/tests/*oracle*.rs`, `conformance.rs` | compare with ffmpeg/ffprobe (§2) |
|
||||
| Robustness / fuzz | `*/tests/robustness.rs`, `*/tests/fuzz.rs` | seeded mutation and truncation of real and synthetic files; nothing may panic |
|
||||
|
|
|
|||
|
|
@ -60,6 +60,7 @@ const LAYERS: &[(&str, u8)] = &[
|
|||
("audio", 2),
|
||||
("captions", 2),
|
||||
("speech", 2),
|
||||
("tts", 2),
|
||||
("interchange", 2),
|
||||
("render", 3),
|
||||
("gpu", 3),
|
||||
|
|
|
|||
Loading…
Reference in New Issue