shifathosn
/
filmcraft
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

M4.9: H.264 inverse transforms reconstruct across columns at once 

The column pass and the add/clip of idct4/idct8 work on whole rows so they
vectorise (same integer steps in the spec's row-then-column order; checked
against the per-column formulation on random blocks). Docs: draft decoding in
architecture.md and perf.stats fields in control-protocol.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Brandon Thomas 2026-10-02 23:46:52 -04:00
parent 1f4546d549
commit aa3ad1d950
3 changed files with 89 additions and 25 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

102
crates/h264/src/transform.rs
View File

@ -88,28 +88,34 @@ fn clip_u8(v: i32) -> u8 {
}
/// Inverse 4x4 transform of raster-order `d` and add to `dst` (8.5.12.2, 8.5.14).
///
/// Rows first, then columns (the spec's order: the `>> 1` steps make it matter). The column
/// pass and the reconstruction run across all columns at once so they vectorise.
pub fn idct4_add(d: &[i32; 16], dst: &mut [u8], stride: usize) {
let mut t = [0i32; 16];
for i in 0..4 {
let mut t = [[0i32; 4]; 4];
for (i, row) in t.iter_mut().enumerate() {
let r = &d[i * 4..i * 4 + 4];
let e = r[0] + r[2];
let f = r[0] - r[2];
let g = (r[1] >> 1) - r[3];
let h = r[1] + (r[3] >> 1);
t[i * 4] = e + h;
t[i * 4 + 1] = f + g;
t[i * 4 + 2] = f - g;
t[i * 4 + 3] = e - h;
*row = [e + h, f + g, f - g, e - h];
}
let mut o = [[0i32; 4]; 4];
for j in 0..4 {
let e = t[j] + t[8 + j];
let f = t[j] - t[8 + j];
let g = (t[4 + j] >> 1) - t[12 + j];
let h = t[4 + j] + (t[12 + j] >> 1);
let col = [e + h, f + g, f - g, e - h];
for i in 0..4 {
let p = &mut dst[i * stride + j];
*p = clip_u8(*p as i32 + ((col[i] + 32) >> 6));
let e = t[0][j] + t[2][j];
let f = t[0][j] - t[2][j];
let g = (t[1][j] >> 1) - t[3][j];
let h = t[1][j] + (t[3][j] >> 1);
o[0][j] = e + h;
o[1][j] = f + g;
o[2][j] = f - g;
o[3][j] = e - h;
}
for (i, row) in o.iter().enumerate() {
let line: &mut [u8; 4] = (&mut dst[i * stride..i * stride + 4]).try_into().unwrap();
for (p, &v) in line.iter_mut().zip(row) {
*p = clip_u8(*p as i32 + ((v + 32) >> 6));
}
}
}
@ -146,18 +152,24 @@ fn idct8_1d(d: [i32; 8]) -> [i32; 8] {
[b0 + b7, b2 + b5, b4 + b3, b6 + b1, b6 - b1, b4 - b3, b2 - b5, b0 - b7]
}
/// Inverse 8x8 transform of raster-order `d` and add to `dst` (8.5.13.2).
/// Inverse 8x8 transform of raster-order `d` and add to `dst` (8.5.13.2): rows, then columns,
/// the column pass and reconstruction across all columns at once (they vectorise).
pub fn idct8_add(d: &[i32; 64], dst: &mut [u8], stride: usize) {
let mut t = [0i32; 64];
for i in 0..8 {
let row: [i32; 8] = d[i * 8..i * 8 + 8].try_into().unwrap();
t[i * 8..i * 8 + 8].copy_from_slice(&idct8_1d(row));
let mut t = [[0i32; 8]; 8];
for (i, row) in t.iter_mut().enumerate() {
*row = idct8_1d(d[i * 8..i * 8 + 8].try_into().unwrap());
}
let mut o = [[0i32; 8]; 8];
for j in 0..8 {
let col = idct8_1d(std::array::from_fn(|i| t[i * 8 + j]));
let col = idct8_1d(std::array::from_fn(|i| t[i][j]));
for i in 0..8 {
let p = &mut dst[i * stride + j];
*p = clip_u8(*p as i32 + ((col[i] + 32) >> 6));
o[i][j] = col[i];
}
}
for (i, row) in o.iter().enumerate() {
let line: &mut [u8; 8] = (&mut dst[i * stride..i * stride + 8]).try_into().unwrap();
for (p, &v) in line.iter_mut().zip(row) {
*p = clip_u8(*p as i32 + ((v + 32) >> 6));
}
}
}
@ -166,6 +178,52 @@ pub fn idct8_add(d: &[i32; 64], dst: &mut [u8], stride: usize) {
mod tests {
use super::*;
/// The straightforward per-column formulation (8.5.12.2 / 8.5.13.2) the vectorised
/// transforms must equal.
fn reference_add(d: &[i32], n: usize, dst: &mut [u8], stride: usize) {
let one = |v: &[i32]| -> Vec<i32> {
if n == 4 {
let (e, f, g, h) = (v[0] + v[2], v[0] - v[2], (v[1] >> 1) - v[3], v[1] + (v[3] >> 1));
vec![e + h, f + g, f - g, e - h]
} else {
idct8_1d(v.try_into().unwrap()).to_vec()
}
};
let rows: Vec<Vec<i32>> = (0..n).map(|i| one(&d[i * n..i * n + n])).collect();
for j in 0..n {
let col = one(&(0..n).map(|i| rows[i][j]).collect::<Vec<_>>());
for i in 0..n {
let p = &mut dst[i * stride + j];
*p = (*p as i32 + ((col[i] + 32) >> 6)).clamp(0, 255) as u8;
}
}
}
#[test]
fn vectorised_transforms_match_reference() {
let mut rng = 0x1234_5678_9abc_def1u64;
let mut next = move || {
rng ^= rng << 13;
rng ^= rng >> 7;
rng ^= rng << 17;
rng
};
for iter in 0..5000 {
let range = [64u64, 1024, 8192, 1 << 15][iter % 4];
let d4: [i32; 16] = std::array::from_fn(|_| (next() % (2 * range + 1)) as i32 - range as i32);
let d8: [i32; 64] = std::array::from_fn(|_| (next() % (2 * range + 1)) as i32 - range as i32);
let base: Vec<u8> = (0..24 * 8).map(|_| next() as u8).collect();
let (mut a, mut b) = (base.clone(), base.clone());
idct4_add(&d4, &mut a[3..], 24);
reference_add(&d4, 4, &mut b[3..], 24);
assert_eq!(a, b, "4x4 iteration {iter}");
let (mut a, mut b) = (base.clone(), base);
idct8_add(&d8, &mut a[5..], 24);
reference_add(&d8, 8, &mut b[5..], 24);
assert_eq!(a, b, "8x8 iteration {iter}");
}
}
#[test]
fn idct4_dc_only_matches_shortcut() {
let mut d = [0i32; 16];

10
docs/architecture.md
View File

@ -242,8 +242,14 @@ file ──► codecs (MP4/MOV, MKV, audio) demux + decode, GOP-aware see
playhead's distance, two seconds for a scrubbed-to frame), and the GOP cache skips
non-reference pictures of late frames on the way (`VideoDecoder::is_disposable`, H.264
`nal_ref_idc` 0 and HEVC sub-layer non-reference pictures; the wanted frame decodes exactly as
before). `perf.stats` reports the counters. GPU plans carry their texels already
converted for upload (`filmcraft_gpu::prepare`), so the UI thread only copies them.
before). With Settings ▸ Playback ▸ Draft decoding (off by default), frames played at 1/2
resolution or lower are requested as draft frames (`FrameKey::draft`,
`filmcraft_media::cancel::with_draft`): the H.264 decoder skips deblocking of non-reference
pictures (`VideoDecoder::set_draft`), the GOP cache serves those draft frames to draft
requests only (a paused frame, render or export decodes them again exactly), and draft plans
hand the GPU box-decimated Y'CbCr planes at the drawn size. `perf.stats` reports the counters.
GPU plans carry their texels already converted for upload (`filmcraft_gpu::prepare`), so the UI
thread only copies them.
- **Audio clock.** The desktop app passes a cpal output (`apps/filmcraft/src/audio.rs`) to the UI as
`AudioOut`. While playing, the samples played by the sound card drive the playhead and video follows.
Without an audio device, playback falls back to the wall clock. `PlaybackMeter` counts timeline

2
docs/control-protocol.md
View File

@ -29,7 +29,7 @@ Methods (handlers in `crates/ui-egui/src/control.rs`):
| `ui.playback` | `{action: play|stop|toggle, speed?}` | |
| `ui.screenshot` | `{path?, panel?}` | PNG of the window or one panel; fails after 10 s when no frame is presented (window hidden, display asleep) |
| `ui.resize`, `ui.focus`, `app.quit` | | `ui.focus` is the only request that activates the app and takes keyboard focus |
| `perf.stats` | – | performance counters (also the command `perf.stats`, so `engine.execute` and MCP `command_run` reach it; headless sessions return the engine part). `decode`: GOP-cache requests / hits / `cacheHitRate`, decoder seeks, samples decoded and skipped while catching up, decoder ms (total and per sample); `playback`: playing, shown / dropped frames and `dropRate` of the current or last play, resolution; `frames`: frame-worker jobs, cancelled, `requestHitRate`, `decodeMs` / `renderMs` (mean, p50, p95 of the last 256 jobs), queue, render-cost estimate, cache use; `ui`: fps and frame ms; `process`: CPU seconds; `media`, `jobs`. Counters are cumulative: diff two readings to measure an interval ([performance.md](performance.md)) |
| `perf.stats` | – | performance counters (also the command `perf.stats`, so `engine.execute` and MCP `command_run` reach it; headless sessions return the engine part). `decode`: GOP-cache requests / hits / `cacheHitRate`, decoder seeks, samples decoded and skipped while catching up, `draftFrames` (decoded in draft mode), `h264Threads` (frame-threading workers per H.264 decoder), decoder ms (total and per sample); `playback`: playing, shown / dropped frames and `dropRate` of the current or last play, resolution, `draftDecode` (Settings ▸ Playback ▸ Draft decoding); `frames`: frame-worker jobs, cancelled, `requestHitRate`, `decodeMs` / `renderMs` (mean, p50, p95 of the last 256 jobs), queue, render-cost estimate, cache use; `ui`: fps and frame ms; `process`: CPU seconds; `media`, `jobs`. Counters are cumulative: diff two readings to measure an interval ([performance.md](performance.md)) |
**Focus.** Driving the app never steals the user's keyboard. Started with `--control`, the app opens
without activating itself. UI requests and screenshots that need a rendered frame bring the window