filmcraft/crates/aac/README.md

filmcraft-aac

A clean-room AAC-LC encoder and decoder (ISO/IEC 14496-3, MPEG-4 Audio, Low Complexity profile), written from the standard and the public psychoacoustics literature. Layer L0: it depends only on filmcraft-bitstream and thiserror (plus optional rayon), has no unsafe, and builds for wasm32-unknown-unknown.

No FFmpeg/FAAC/FDK-AAC/libfaad or other GPL/LGPL code was read or ported. ffmpeg is used only as an external test oracle (decoding our streams, producing reference streams).

API

use filmcraft_aac::{Decoder, Encoder, EncoderConfig, BitrateMode, AdtsHeader, split_adts};

// Encoder: planar f32 (±1.0) → raw access units
let mut enc = Encoder::new(EncoderConfig::cbr(48_000, 2, 128_000))?;   // or EncoderConfig::vbr(48_000, 2, 3.0)
let mut aus: Vec<Vec<u8>> = enc.encode(&[&left, &right]);              // 0..n AUs per call, any chunk size
aus.extend(enc.flush());                                               // zero-pads and drains the look-ahead
let asc = enc.audio_specific_config();                                 // bytes for the MP4 `esds`
let delay = enc.priming_samples();                                     // 1024 → edit list media_time
let stats = enc.stats();                                               // frames, bytes, bitrate(), short blocks, M/S, TNS

// Decoder: AudioSpecificConfig + raw AUs → planar f32, 1024 samples per channel per AU
let mut dec = Decoder::new(&asc)?;
let pcm: Vec<Vec<f32>> = dec.decode(&aus[0])?;

// ADTS
let frames = split_adts(&adts_bytes)?;               // [(AdtsHeader, raw payload)]
let mut dec = Decoder::from_adts(&frames[0].0)?;     // channel config 0 → layout from the in-band PCE

EncoderConfig fields: sample_rate (the 13 AAC rates, 7.35–96 kHz), channels (1–8), mode (BitrateMode::Cbr(bps) / BitrateMode::Vbr(quality 1.0..=5.0)), bandwidth (Hz, None = automatic from bitrate/quality), tns, ms_stereo, window_shape (sine/KBD), adts (prefix each AU with a 7-byte ADTS header).

Channel order is AAC element order, for both encoder input and decoder output:

channels	configuration	order
1	1	C
2	2	L R
3	3	C L R
4	4	C L R Cs
5	5	C L R Ls Rs
6	6	C L R Ls Rs LFE
7	0 + PCE	C L R Ls Rs Cs LFE (PCE in the ASC; repeated in every AU when ADTS-framed)
8	7	C Lc Rc L R Ls Rs LFE

The decoder also reads configurations 11 (6.1: C L R Ls Rs Cs LFE) and 12 (7.1: C L R Ls Rs Lsr Rsr LFE) from ISO/IEC 14496-3 Amd 4; ffmpeg 9 writes 7.1 AAC as configuration 12.

Decoder::layout() reports the element layout (it follows a PCE for configuration 0).

Encoder

Per 1024-sample frame and channel element:

1. Transient detection: energy of the first-difference (high-passed) signal in 128-sample sub-blocks against the mean of the previous eight; one frame of look-ahead selects ONLY_LONG / LONG_START / EIGHT_SHORT / LONG_STOP. Short windows are grouped by energy similarity (an attack starts a new group). LFE is always long.
2. MDCT (2048/256 points) via a DCT-IV on an N/4-point radix-2 complex FFT; sine or KBD (α = 4 long, 6 short) windows.
3. Psychoacoustic model in the MDCT domain: band energies, tonality from the spectral flatness of a ≥16-line neighbourhood, Schroeder's spreading function in the Bark domain, tonality-dependent masking offset (30 dB tonal … 6 dB noise), Terhardt threshold in quiet (capped at 40 dB SPL) as a floor, pre-echo control (threshold ≤ 2× previous frame), perceptual entropy.
4. TNS on long windows: Levinson–Durbin (order 8) over the spectrum above 1.4 kHz; applied when the prediction gain exceeds 1.4, with 4-bit reflection coefficients.
5. M/S per band (and group) for channel pairs (always common window): chosen when the estimated perceptual cost of M and S (with the stricter L/R threshold, halved) is below that of L and R.
6. Rate–distortion loop: one multiplier λ scales every band's allowed noise. Each band gets the coarsest scalefactor whose measured quantisation error meets its target (binary search, 0.4054 rounding), scalefactor differences are kept within ±60, codebooks and sections are chosen by dynamic programming over all 12 codebook states, and λ is bracketed and bisected until the frame fits its budget (CBR) or fixed from the quality (VBR).
7. Bit reservoir (CBR): frame budgets follow the perceptual entropy (short-window frames get extra), bounded by the 6144-bit-per-channel decoder buffer; overflow is padded with fill elements and the final frame drains the reservoir, so the stream is exactly frames × bitrate × 1024 / rate.

Bandwidth: automatic from bitrate per full-bandwidth channel (e.g. 64 kbps/ch → 16 kHz, 96 → 19 kHz, ≥144 → 22 kHz) or from quality in VBR.

Delay: 1024 priming samples (priming_samples()); after flush() the decoded stream minus the priming covers every input sample (ceil(n / 1024) + 1 access units).

Performance (Apple Silicon, release, single thread): 10 s of 44.1 kHz stereo encodes at ~85× realtime and decodes at ~1000× realtime. With the rayon feature channels are quantised in parallel inside the rate loop.

Decoder

Full AAC-LC syntax: SCE/CPE/LFE, DSE, FIL, PCE (reconfigures a configuration-0 stream), all window sequences and both window shapes (shape of the previous frame for the left half), section data, scalefactors, pulse data, TNS (all-pole synthesis, long and short), M/S (per band / all bands), intensity stereo (in/out of phase, M/S-inverted), PNS (with correlated noise for ms_used noise bands as specified), IMDCT and overlap-add. Output is planar f32 at ±1.0 full scale. All reads are bounds checked: malformed access units return Err, never panic.

Test results

All produced by cargo test -p filmcraft-aac -- --nocapture (ffmpeg 9.0 at /opt/homebrew/bin as oracle; tests skip when it is absent). Fixtures go to target/fixtures/aac/.

Encoder quality / bitrate (2 s synthetic signals, decoded by ffmpeg)

SNR is full-band; bwSNR is measured after low-passing both signals at the coded bandwidth; segSNR averages 1024-sample segments above −50 dBFS (clamped to −10…60 dB). "us-vs-ff" is the largest difference between our decoder and ffmpeg's on the same stream. Every stream decodes without ffmpeg errors, no output clips (peak < 1.0), and CBR is within ±0.1% of target (test limit ±5%).

signal	kbps	actual	bw Hz	SNR	bwSNR	segSNR	short frames	us-vs-ff
sweep 2ch 44.1k	64	64.00	11000	12.1	36.1	43.6	1	5e-7
sweep 2ch 44.1k	128	127.92	16000	26.1	27.3	54.2	1	3e-7
sweep 2ch 44.1k	320	319.90	21609	46.6	46.6	59.6	1	3e-7
white noise 2ch 44.1k	64	64.00	11000	1.3	3.3	1.3	1	5e-7
white noise 2ch 44.1k	320	319.91	21609	16.4	16.5	16.6	1	1.5e-6
speech-like 2ch 48k	64	64.00	11000	13.6	14.0	15.2	13	2e-7
speech-like 2ch 48k	128	127.98	16000	21.3	23.0	20.7	13	2e-7
speech-like 2ch 48k	320	319.96	22000	31.2	31.5	31.3	13	2e-7
chords 2ch 44.1k	64	64.01	11000	12.4	12.4	20.6	4	2e-7
chords 2ch 44.1k	128	128.00	16000	20.1	20.1	38.5	4	2e-7
chords 2ch 44.1k	320	319.99	21609	35.9	35.9	55.5	4	2e-7
clicks 2ch 48k	64	64.00	11000	10.4	11.2	48.6	11	9e-8
clicks 2ch 48k	128	128.00	16000	20.2	25.1	53.9	11	2e-7
clicks 2ch 48k	320	319.95	22000	39.6	39.6	58.0	11	1e-7
speech-like 1ch 44.1k	64	64.00	16000	18.4	19.2	18.0	15	2e-7
chords 5.1 48k	320	320.00	16000	27.2	27.2	37.9	9	2e-7
chords 3/4/5/7/8ch 48k	192–448	exact	16000	26–28	26–28	38	7–13	2e-7
chords 2ch 8 k … 96 k (10 rates)	24–256	exact	3.9–21 k	13–38		17–47		≤3e-7

Reference points from ffmpeg's own AAC encoder on the same signals (stereo 48 kHz, segSNR): sweep 128k 39.7, speech 128k 21.2, chords 128k 32.8, clicks 128k 43.3 (ours: 52/20/39/54). Pure SNR is not a perceptual measure: the model deliberately leaves noise under the masking threshold, and full-band white noise is the worst case by design.

VBR (Vbr(q), 44.1 kHz stereo): chords q1 62 kbps / q3 139 / q5 225 (segSNR 18 / 39 / 51); speech-like q1 42 / q3 166 / q5 272 kbps (segSNR 13 / 29 / 41). Bitrate and quality grow monotonically with q.

Decoder accuracy vs ffmpeg (streams from ffmpeg -c:a aac, PNS off)

stream	max abs error
stereo 44.1k 128k / 48k 256k / 96k 256k	1.2e-7 / 1.2e-7 / 1.3e-7
mono 48k 64k / 22.05k 32k / 16k 24k / 8k 16k	1.5e-7 / 1.2e-7 / 1.2e-7 / 8.9e-8
stereo 44.1k 96k with intensity stereo + M/S + TNS	1.2e-7
stereo 32k 48k with intensity stereo	1.2e-7
3.0, 5.1 (320k), 7.1 wide with PCE (448k)	1.2e-7, 1.5e-7, 1.3e-7
stereo 44.1k 160k -aac_coder fast	1.2e-7

Our own encoder's streams: ≤1.6e-6 against ffmpeg for every case above. PNS (ffmpeg's encoder does not emit it, so streams are hand-built): total energy within 0.15 dB of ffmpeg; per-frame energy within 0.8 dB (random vectors differ). For noise bands flagged ms_used we produce identical L/R noise as the standard specifies; ffmpeg produces independent noise there.

Robustness

12 000 mutated access units (bit flips, byte overwrites, truncation, extension, splices, random buffers) over mono/stereo/5.1/7-channel streams, 20 000 random AudioSpecificConfigs and 2 000 random ADTS streams: no panics. The encoder accepts NaN/±inf/over-range input, any chunk size, and missing channels.

Clean-room notes

• Syntax, tables (scalefactor bands, TNS limits), filterbank, TNS, PNS, M/S and intensity stereo are implemented from ISO/IEC 14496-3.
• The Huffman codebooks (src/huffman_tables.rs) are the normative values of the standard's Annex 4.A, recovered by black-box probing of the ffmpeg binary as an oracle: tools/ contains the Python (stdlib-only) scripts that craft single-frame streams repeating a candidate bit string, fit the decoded output to the IMDCT basis, and walk the code tree. Every table is verified to be a complete prefix code (Kraft sum exactly 1) covering each index once (unit test), and the decoder matches ffmpeg to ~1e-7 on real streams.
• Psychoacoustics follow textbook models (Bark scale, Schroeder spreading, Terhardt ATH, Johnston-style tonality/SFM offsets, perceptual entropy).

Gaps / not implemented

• HE-AAC v1/v2 (SBR/PS): explicit signalling is parsed and only the AAC-LC core is decoded (half-rate, no SBR); implicit SBR fill elements are skipped.
• AAC Main (prediction), SSR (gain control), LTP, ER/LD/ELD object types, coupling channel elements (CCE → Err(Unsupported)), 960-sample frames.
• Encoder: no PNS or intensity stereo (decoder supports both), no TNS on short windows, no pulse data; window shape is fixed per stream (sine or KBD). Short blocks at onsets cost segmental SNR on speech-like material relative to long windows + TNS; perceptually they avoid pre-echo.
• No error concealment: a corrupt AU returns Err; the caller may substitute silence.
• ADTS with CRC is parsed (header skipped, CRC not verified); multi-block ADTS frames are split per frame only.