diff --git a/ROADMAP.md b/ROADMAP.md index 85b2b5e1..bf14f137 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -11,7 +11,7 @@ projects (DrawCraft reached its first four milestones in ≈ 4½ h) and are revi | M0 | Skeleton + visual shell | workspace, xtask CI + layering, geom/color/raster, develop model, pipeline v0, catalog v0, engine commands, Lightroom-look UI (grid, loupe, filmstrip, Edit panel), control channel, MCP, web build | 3–5 | ✅ | | M1 | Library core | import (JPEG/PNG/TIFF/WebP), EXIF/XMP, persistent catalog (op log + snapshots), albums, ratings/flags/labels, filter/search/sort, thumbnail cache, 100k-photo grid | 6–10 | ✅ (100k-photo grid scale test pending) | | M2 | Pipeline v1 (quality) | WB temp/tint, profiles, local tone mapping (highlights/shadows), curves, HSL, point colour, colour grading, texture/clarity/dehaze, vignette, grain, B&W, auto tone/WB, histogram, before/after | 10–15 | ✅ (look tuning vs our references ongoing) | -| M3 | RAW I | TIFF/DNG (LJ92, deflate, tiles, opcodes), demosaic (AHD/PPG/bilinear), highlight recovery, DNG colour model, CR2, NEF, ARW, embedded previews | 10–15 | 🚧 (DNG, CR2, ARW, NEF uncompressed, embedded previews ✅; NEF Huffman ⬜) | +| M3 | RAW I | TIFF/DNG (LJ92, deflate, tiles, opcodes), demosaic (AHD/PPG/bilinear), highlight recovery, DNG colour model, CR2, NEF, ARW, embedded previews | 10–15 | 🚧 (DNG, CR2, ARW, NEF uncompressed + Huffman lossless/lossy, embedded previews ✅; NEF lossy-after-split ⬜) | | M4 | Crop, geometry, optics | crop tool + overlays, straighten, Upright (auto/level/vertical/full/guided), manual transforms, CA, defringe, manual lens corrections | 6–10 | ✅ | | M5 | Performance | source pyramids, wgpu compute pipeline (CPU oracle), draft/full renders, prefetch, budgets (16 ms slider updates on 24 MP) | 10–15 | 🚧 (stage cache, source pyramid, wgpu pipeline, prefetch, memory budget ✅; colour NR at half resolution, GPU histogram ⬜) | | M6 | Masking | brush, linear/radial gradients, colour/luminance/depth range, add/subtract/intersect/invert, all local adjustments, masks panel | 8–12 | 🚧 (brush/linear/radial/colour/luminance range, add/subtract/intersect, masks panel ✅; depth range, AI masks ⬜) | @@ -19,7 +19,7 @@ projects (DrawCraft reached its first four milestones in ≈ 4½ h) and are revi | M8 | Heal / Remove | content-aware remove (PatchMatch), heal, clone, brush spots, visualize spots, red/pet eye | 6–10 | 🚧 (heal, clone, auto source, visualize spots, red/pet eye ✅; PatchMatch remove ⬜) | | M9 | Presets, profiles, versions, sync | preset browser + amount, create/import presets, profile browser, versions, history, copy/paste/sync settings | 5–8 | ✅ | | M10 | Export & share | export dialog (JPEG/PNG/TIFF/DNG/AVIF/JXL/original), sizing, sharpening, metadata, watermark, naming, batch jobs, XMP sidecars, HDR export | 6–10 | 🚧 (all formats incl. DNG/original, sizing, presets, background jobs ✅; JXL encode, HDR export ⬜) | -| M11 | RAW II | CR3, RAF (X-Trans), ORF, RW2, PEF, SRW, 3FR, IIQ + long tail; camera calibration DB; HEIC/AVIF/JXL import | 20–35 | 🚧 (RAF uncompressed, RW2 packed, PEF, ORF uncompressed ✅; CR3, compressed NEF/ORF/RAF ⬜) | +| M11 | RAW II | CR3, RAF (X-Trans), ORF, RW2, PEF, SRW, 3FR, IIQ + long tail; camera calibration DB; HEIC/AVIF/JXL import | 20–35 | 🚧 (RAF uncompressed, RW2 packed, PEF, ORF uncompressed ✅; CR3, compressed ORF/RAF ⬜) | | M12 | AI & smart features | subject/sky/background/people/object masks, semantic search, faces/People (permissively licensed models, pure-Rust inference) | 20–40 | ⬜ | | M13 | Merge | HDR merge (deghost), panorama (projections, boundary warp, fill edges), HDR panorama | 10–15 | ✅ | | M14 | Video | import/playback/trim via FilmCraft crates, global edits + presets on video, video export | 6–10 | ⬜ | @@ -50,7 +50,7 @@ trip, slideshow, auto import…) in ≈ 12 h on 2026-10-02; four to six parallel | Work package | Tracker rows | Agent-hours | Risk | |---|---|---:|---| | Remaining P0/P1 UI and library features (folder rename/move, keyword painter, people view…) | ≈ 8 | 5–10 | low | -| Raw codecs: CR3 (CRX), compressed NEF / ORF / RAF, RW2 v4, HEIC/AVIF decode, JPEG XL DNG | LR-IMP-FORMATS | 40–80 | **high** — clean-room black-box analysis, no permissive specs | +| Raw codecs: CR3 (CRX), compressed ORF / RAF, NEF lossy-after-split, RW2 v4, HEIC/AVIF decode, JPEG XL DNG | LR-IMP-FORMATS | 40–80 | **high** — clean-room black-box analysis, no permissive specs | | Lens-profile database of our own (calibration targets, fitting, data) | LR-EDIT-OPTICS-PROFILE | 15–30 | data collection | | Video: playback, trim, edits, export (pure-Rust decode, ideally shared with FilmCraft) | R. Video | 20–40 | medium | | AI: subject / sky / background / people / object masks, object-aware remove, AI denoise, super resolution, lens blur, people & faces, natural-language search, culling | ≈ 30 | 80–150 | **high** — permissively licensed weights, pure-Rust inference, maybe training | @@ -94,16 +94,16 @@ The milestone estimates in the table above were made before work started and are ## Raw format coverage and known gaps Decoded (CC0 corpus from raw.pixls.us, `cargo xtask corpus --download`, `crates/raw/tests/corpus.rs`): DNG (uncompressed, -LJ92, lossy JPEG / Smart Previews, Deflate, float, linear), CR2, ARW (uncompressed, ARW2, LJ92), NEF/NRW uncompressed, RAF uncompressed (Bayer and +LJ92, lossy JPEG / Smart Previews, Deflate, float, linear), CR2, ARW (uncompressed, ARW2, LJ92), NEF/NRW uncompressed and Huffman-compressed (lossless, lossy type 1/2, 12/14-bit), RAF uncompressed (Bayer and X-Trans), RW2 packed 12/14-bit, PEF (uncompressed and Huffman), ORF uncompressed (16-bit and 12-bit packed). Every supported container also yields its embedded JPEG preview (CR3 too), and the engine shows that preview for raw variants it can't decode yet. Not decoded yet — preview only (no permissively licensed description; black-box analysis incomplete): -- **Nikon Huffman NEF** (lossless / lossy). Black-box findings so far: maker note `0x0096` holds per-parity predictor - seeds (2048 for 14-bit lossless); ≈ 9 bits per pixel; the category-0 code word is a 6-bit rotation of `011111`. The - Huffman tables are not stored in the files, and an automatic table search (beam search over prefix codes scored by - smoothness and bit rate, validated on Pentax files whose tables *are* stored) did not converge. +- **Nikon "lossy after split" NEF** (non-zero split row in maker note `0x0096`, e.g. some D3400/D5000/D5200/D5500/D5600 + files): rows above the split decode with the regular lossy table; from the split row on a different code is used + that our black-box analysis has not recovered yet (none of the four regular tables fits, also not with byte + alignment or reset predictors at the split row). The other Nikon Huffman variants are decoded (`crates/raw/src/vendor/nefc.rs` documents the analysis). - **Panasonic RW2 raw format 4** (quantised): the block layout is known (0x4000-byte chunks rotated by 0x1ff8; 128-bit blocks of two 12-bit seeds plus four groups of a 2-bit scale and three 8-bit codes; scales 0/1 are ×1/×2 differences from the same-colour pixel two to the left), the reconstruction rule for scales 2/3 is not established. diff --git a/crates/raw/src/lib.rs b/crates/raw/src/lib.rs index 554bc1f0..29e280b6 100644 --- a/crates/raw/src/lib.rs +++ b/crates/raw/src/lib.rs @@ -11,10 +11,10 @@ //! and produces camera → linear Rec.2020 D65 matrices. //! //! Formats: DNG (uncompressed, lossless JPEG, lossy JPEG (Smart Previews), Deflate incl. floating point, tiled/stripped, CFA and LinearRaw), -//! Canon CR2, Nikon NEF/NRW (uncompressed), Sony ARW (uncompressed, ARW2, lossless), Fujifilm RAF (uncompressed Bayer +//! Canon CR2, Nikon NEF/NRW (uncompressed, Huffman lossless / lossy compressed), Sony ARW (uncompressed, ARW2, lossless), Fujifilm RAF (uncompressed Bayer //! and X-Trans), Panasonic RW2 (packed 12/14-bit), Pentax PEF (uncompressed, Huffman), Olympus ORF (uncompressed). -//! [`embedded_preview`] covers all of them plus CR3. Variants we can't decode yet (Nikon Huffman NEF, Panasonic -//! quantised RW2, compressed ORF/RAF, CR3) return [`RawError::Unsupported`]; each vendor module documents its sources +//! [`embedded_preview`] covers all of them plus CR3. Variants we can't decode yet (Nikon "lossy after split" NEF, +//! Panasonic quantised RW2, compressed ORF/RAF, CR3) return [`RawError::Unsupported`]; each vendor module documents its sources //! (public specifications, tag-name documentation, black-box analysis of CC0 samples) and gaps. Non-DNG files carry no //! colour matrix: [`color`] falls back to a documented neutral model. The decoders never panic on malformed input. #![forbid(unsafe_code)] diff --git a/crates/raw/src/vendor/mod.rs b/crates/raw/src/vendor/mod.rs index 74a4461b..2de155ef 100644 --- a/crates/raw/src/vendor/mod.rs +++ b/crates/raw/src/vendor/mod.rs @@ -3,6 +3,7 @@ pub mod arw; pub mod cr2; pub mod nef; +pub mod nefc; pub mod orf; pub mod pef; pub mod raf; diff --git a/crates/raw/src/vendor/nef.rs b/crates/raw/src/vendor/nef.rs index fb4b644a..e81ada36 100644 --- a/crates/raw/src/vendor/nef.rs +++ b/crates/raw/src/vendor/nef.rs @@ -1,22 +1,22 @@ -//! Nikon NEF / NRW — uncompressed variants only. +//! Nikon NEF / NRW — uncompressed and Huffman-compressed (lossless, lossy) variants. //! //! Sources: TIFF 6.0 (the raw image is a standard CFA SubIFD), Laurent Clévy's NEF structure notes (prose: IFD //! layout, SubIFDs, maker note header) and the ExifTool Nikon tag-name documentation (`0x000c` WB_RBLevels, -//! `0x003d` BlackLevel). Nikon's Huffman-compressed variants (compression 34713: lossless, lossy type 1/2) are -//! **not** implemented: the only descriptions we found are derived from GPL code, which our clean-room rules -//! forbid. The coding is T.81-style (difference category + additional bits, as in Pentax PEF), but unlike PEF the -//! Huffman tables are not stored in the files and our black-box table search has not converged (see ROADMAP.md). -//! They are reported as [`RawError::Unsupported`]; their embedded previews still work. +//! `0x003d` BlackLevel, `0x0096` NEFLinearizationTable). The Huffman-compressed data (compression 34713) is decoded +//! by [`super::nefc`], which documents its clean-room sources; files it can't decode yet ("lossy after split") +//! are reported as [`RawError::Unsupported`] and their embedded previews still work. -use super::white_from_data; +use super::{nefc, white_from_data}; use crate::tiffraw::{Packing, read_image}; use crate::{BlackLevel, Cfa, ColorData, OpcodeLists, RawData, RawError, RawFormat, RawImage, Rect, Result}; use lightcraft_geom::Orientation; +use lightcraft_tiff::image::ImageInfo; use lightcraft_tiff::tags::{self as t, photometric}; -use lightcraft_tiff::{Ifd, Tiff, makernote}; +use lightcraft_tiff::{ByteOrder, Ifd, Tiff, makernote}; const WB_RB_LEVELS: u16 = 0x000c; const BLACK_LEVEL: u16 = 0x003d; +const LINEARIZATION_TABLE: u16 = 0x0096; fn raw_ifd(tiff: &Tiff) -> Option<&Ifd> { tiff.all_ifds() @@ -47,30 +47,15 @@ pub(crate) fn decode(bytes: &[u8]) -> Result { let ifd0 = &tiff.ifds[0]; let raw = raw_ifd(&tiff).ok_or_else(|| RawError::Unsupported("NEF without a CFA image IFD".into()))?; let info = raw.image()?; - if info.compression == t::compression::NIKON { - return Err(RawError::Unsupported("Nikon Huffman-compressed NEF (no clean-room description available)".into())); - } let (w, h) = (info.width as usize, info.height as usize); let bits = info.bits() as u32; - let row_samples = w * info.samples_per_pixel as usize; - let chunks = info.chunks(bytes.len() as u64); - let rows_in_first = chunks.first().map(|c| c.height as usize).unwrap_or(h).max(1); - let bytes_per_row = chunks.first().map(|c| c.len as usize / rows_in_first).unwrap_or(0); - let packing = if bytes_per_row >= row_samples * 2 { - Packing::Word16 - } else if bits == 12 && bytes_per_row * 8 >= row_samples * 12 && bytes_per_row * 8 < row_samples * 13 { - Packing::Msb - } else if info.compression == 1 && bits != 8 && bits != 16 { - return Err(RawError::Unsupported(format!("NEF uncompressed packing ({bytes_per_row} bytes per {row_samples}-sample row)"))); - } else { - Packing::Msb - }; - let data = read_image(bytes, &info, tiff.order, packing)?; - let RawData::U16(ref samples) = data else { return Err(RawError::Unsupported("float NEF".into())) }; - let make = ifd0.string(t::MAKE).unwrap_or_default(); let mn = tiff.exif().and_then(|e| e.get(t::MAKER_NOTE)).and_then(|e| makernote::parse_makernote(bytes, e.offset, e.count() as u64, tiff.order, &make)); + let data = + if info.compression == t::compression::NIKON { compressed(bytes, &info, mn.as_ref())? } else { uncompressed(bytes, &info, tiff.order)? }; + let RawData::U16(ref samples) = data else { return Err(RawError::Unsupported("float NEF".into())) }; + let black = match mn.as_ref().and_then(|m| m.ifd.f64s(BLACK_LEVEL)).as_deref() { Some([a, b, c, d]) if [a, b, c, d].iter().all(|v| **v < 16384.0) => { BlackLevel { repeat_rows: 2, repeat_cols: 2, values: vec![*a as f32, *b as f32, *c as f32, *d as f32], ..Default::default() } @@ -114,6 +99,46 @@ pub(crate) fn decode(bytes: &[u8]) -> Result { Ok(img) } +/// Uncompressed strips: 16-bit words or 12-bit MSB-packed, told apart by the strip size. +fn uncompressed(bytes: &[u8], info: &ImageInfo, order: ByteOrder) -> Result { + let (w, h) = (info.width as usize, info.height as usize); + let bits = info.bits() as u32; + let row_samples = w * info.samples_per_pixel as usize; + let chunks = info.chunks(bytes.len() as u64); + let rows_in_first = chunks.first().map(|c| c.height as usize).unwrap_or(h).max(1); + let bytes_per_row = chunks.first().map(|c| c.len as usize / rows_in_first).unwrap_or(0); + let packing = if bytes_per_row >= row_samples * 2 { + Packing::Word16 + } else if bits == 12 && bytes_per_row * 8 >= row_samples * 12 && bytes_per_row * 8 < row_samples * 13 { + Packing::Msb + } else if info.compression == 1 && bits != 8 && bits != 16 { + return Err(RawError::Unsupported(format!("NEF uncompressed packing ({bytes_per_row} bytes per {row_samples}-sample row)"))); + } else { + Packing::Msb + }; + read_image(bytes, info, order, packing) +} + +/// Nikon Huffman-compressed strip (see [`super::nefc`]); the decode table is maker note `0x0096`. +fn compressed(bytes: &[u8], info: &ImageInfo, mn: Option<&makernote::MakerNote>) -> Result { + // (some Z bodies label uncompressed, row-padded data 34713 without a table: not handled here yet) + let table = mn + .and_then(|m| Some((m.ifd.bytes(LINEARIZATION_TABLE)?, m.order))) + .ok_or_else(|| RawError::Unsupported("Nikon compressed NEF without a linearization table (maker note 0x96)".into()))?; + if info.samples_per_pixel != 1 { + return Err(RawError::Unsupported(format!("Nikon compressed NEF with {} samples per pixel", info.samples_per_pixel))); + } + let bits = info.bits() as u32; + let table = nefc::parse_table(table.0, table.1, bits)?; + // one strip in every sample; several would be contiguous parts of the same bit stream + let chunks = info.chunks(bytes.len() as u64); + let (Some(first), Some(last)) = (chunks.first(), chunks.last()) else { return Err(RawError::Corrupt("NEF: no image data".into())) }; + let end = last.offset.saturating_add(last.len).min(bytes.len() as u64); + let src = usize::try_from(first.offset).ok().zip(usize::try_from(end).ok()).and_then(|(a, b)| bytes.get(a..b)); + let src = src.ok_or_else(|| RawError::Corrupt("NEF: image data outside the file".into()))?; + Ok(RawData::U16(nefc::decode(src, info.width as usize, info.height as usize, bits, &table)?)) +} + #[cfg(test)] mod tests { use super::*; @@ -159,7 +184,7 @@ mod tests { } #[test] - fn compressed_is_unsupported() { + fn compressed_without_table_is_unsupported() { let bytes = nef(34713, 14, vec![vec![0; 64]], 8, 8, 8); assert!(matches!(crate::decode(&bytes), Err(RawError::Unsupported(_)))); } diff --git a/crates/raw/src/vendor/nefc.rs b/crates/raw/src/vendor/nefc.rs new file mode 100644 index 00000000..c0260d95 --- /dev/null +++ b/crates/raw/src/vendor/nefc.rs @@ -0,0 +1,481 @@ +//! Nikon Huffman-compressed NEF image data (TIFF compression 34713): "lossless compressed" and "lossy compressed" +//! (type 1 and type 2) at 12 and 14 bits. +//! +//! **Clean-room.** No raw-decoder source code (dcraw, LibRaw, rawspeed, rawloader/rawler, darktable, RawTherapee, +//! libopenraw, ExifTool's Perl code, …) was read or consulted. What we used: +//! +//! - Prose format descriptions: Laurent Clévy, "Nikon Electronic File (NEF) format" (), +//! the maker-note tag table and the "0x96 (linearization table) tag format" table (version bytes, four `u16` +//! predictor seeds at offset 2, curve size at 10, curve at 12, split value at 562 for lossy type 2, "lossy type +//! 2 reads an incomplete table and interpolation is required"); Bill Claff, "NEF Compression" +//! (): a lossy encoding curve followed by a +//! non-adaptive (fixed-table) Huffman stage, and the number of encoded values per camera (683, 769, 1025, 2753, +//! 3073, 4097 …), which confirmed our curve interpolation; the libopenraw format notes +//! (, prose only); ExifTool's Nikon tag-name documentation +//! (`0x0093` NEFCompression, `0x0096` NEFLinearizationTable); ITU-T T.81 Annex F/H for the "difference category +//! + additional bits" coding and canonical (BITS/HUFFVAL) Huffman codes. +//! - Our own black-box analysis of CC0 samples from raw.pixls.us. Nikon's Huffman tables are not stored in the +//! files and no prose source lists them, so they were recovered from the data: several cameras (D5100, D300, +//! D700) were shot with the same scene in compressed and uncompressed modes; a beam search assigned code words +//! to difference categories so that the decoded compressed image matched the uncompressed one (likelihood of +//! the decoded values given the reference, minus the bits consumed, plus a description-length penalty per new +//! code word). Each table was then verified on ~45 files from ~35 bodies (D40 … D850, Df, Z 6, Z 50, 1 J1): with +//! the right table the whole strip decodes and ends within the last byte of the data, with every value inside +//! the encoded range. The predictor and difference coding are the same as Pentax PEF (see `pef.rs`). +//! +//! Format, as established: +//! - One strip, MSB-first bit stream, no byte stuffing, continuous across rows. Per pixel a canonical Huffman code +//! gives the category `n`, followed by `n` additional bits (T.81: a leading 0 means a negative difference). +//! - Each pixel is predicted from the same-colour pixel two to the left; the first two pixels of a row from the +//! first two pixels of the previous row of the same parity, starting at the seeds of maker note `0x0096`. +//! - Lossless (`0x0096` version `0x46`): the decoded values are the samples. Lossy (version `0x44`): they index a +//! curve. Type 1 (`0x44 0x10`) stores the full curve; type 2 (`0x44 0x20`, `0x44 0x40`) stores 257 points, +//! one every `2^bits / 256` codes, linearly interpolated. +//! - Values in the maker note are in the maker note's byte order (newer bodies write little-endian notes). +//! +//! Not supported (returned as [`RawError::Unsupported`], the embedded preview is used instead): +//! - "Lossy after split" files (non-zero split row at `0x0096` offset 562): from that row on a different table +//! is used which our analysis has not recovered yet. +//! - Other `0x0096` versions, bit depths other than 12/14. +//! - Code words that never occurred in any sample (the lossy 12-bit 8-bit code word `11111110`, and code words +//! longer than 9 bits in the lossy tables) are rejected as corrupt; they cannot code a difference within the +//! lossy value range anyway. + +use super::pef::{Bits, Huffman, diff}; +use crate::{MAX_SAMPLES, RawError, Result}; +use lightcraft_tiff::ByteOrder; + +/// A table code word whose symbol was never observed (see the module docs). +const UNSEEN: u8 = u8::MAX; + +/// Canonical Huffman table: number of codes of length 1..=16 and the symbols (difference categories) in code order. +struct Table { + counts: &'static [u8], + symbols: &'static [u8], +} + +const LOSSLESS_12: Table = Table { counts: &[0, 1, 4, 2, 3, 1, 2], symbols: &[5, 4, 6, 3, 7, 2, 8, 1, 9, 0, 10, 11, 12] }; +const LOSSLESS_14: Table = Table { counts: &[0, 1, 4, 2, 2, 3, 1, 2], symbols: &[7, 6, 8, 5, 9, 4, 10, 3, 11, 12, 2, 0, 1, 13, 14] }; +const LOSSY_12: Table = Table { counts: &[0, 1, 5, 1, 1, 1, 1, 1, 1], symbols: &[5, 4, 3, 6, 2, 7, 1, 0, 8, 9, UNSEEN, 10] }; +const LOSSY_14: Table = Table { counts: &[0, 1, 4, 3, 1, 1, 1, 1, 1], symbols: &[5, 6, 4, 7, 8, 3, 9, 2, 1, 0, 10, 11, 12] }; + +impl Table { + /// `(left-aligned 12-bit code, length)` per symbol, for [`Huffman::new`]. + fn codes(&self) -> Vec<(u16, u8)> { + let mut out = vec![(0u16, 0u8); 17]; + let mut code = 0u32; + let mut symbols = self.symbols.iter(); + for (i, &n) in self.counts.iter().enumerate() { + let len = i as u32 + 1; + for _ in 0..n { + if let Some(&s) = symbols.next() + && let Some(slot) = out.get_mut(s as usize) + { + *slot = ((code << (12 - len)) as u16, len as u8); + } + code += 1; + } + code <<= 1; + } + out + } +} + +/// How the decoded values map to samples. +#[derive(Clone, Debug, PartialEq)] +pub(crate) enum Encoding { + /// The decoded values are the samples (`0..2^bits`). + Lossless, + /// The decoded values index this curve. + Lossy(Vec), +} + +/// Maker note `0x0096` (NEF linearization table), parsed. +#[derive(Clone, Debug, PartialEq)] +pub(crate) struct DecodeTable { + pub encoding: Encoding, + /// Predictor seeds `[row parity][column]`. + pub seeds: [[i32; 2]; 2], +} + +/// Parse maker note `0x0096` for a `bits`-bit image. `order` is the maker note's byte order. +pub(crate) fn parse_table(t: &[u8], order: ByteOrder, bits: u32) -> Result { + let u16_at = |o: usize| t.get(o..o + 2).and_then(|s| <[u8; 2]>::try_from(s).ok()).map(|b| order.u16(b)); + let short = || RawError::Corrupt(format!("NEF: linearization table too short ({} bytes)", t.len())); + if bits != 12 && bits != 14 { + return Err(RawError::Unsupported(format!("Nikon compressed NEF with {bits}-bit samples"))); + } + let (v0, v1) = (t.first().copied().ok_or_else(short)?, t.get(1).copied().ok_or_else(short)?); + let mut seeds = [[0i32; 2]; 2]; + for (i, s) in seeds.iter_mut().flatten().enumerate() { + *s = u16_at(2 + 2 * i).ok_or_else(short)? as i32; + } + let encoding = match (v0, v1) { + (0x46, _) => Encoding::Lossless, + (0x44, 0x10 | 0x20 | 0x40) => { + let n = u16_at(10).ok_or_else(short)? as usize; + let points: Vec = (0..n).map(|i| u16_at(12 + 2 * i)).collect::>().ok_or_else(short)?; + if n < 2 { + return Err(RawError::Corrupt(format!("NEF: linearization curve of {n} points"))); + } + if v1 == 0x10 { + Encoding::Lossy(points) + } else { + let split = u16_at(562).unwrap_or(0); + if split != 0 { + return Err(RawError::Unsupported(format!( + "Nikon \"lossy after split\" compressed NEF (split at row {split}) is not decoded yet" + ))); + } + // type 2: `n` points, one every 2^bits / (n - 1) codes + let range = 1usize << bits; + if n - 1 > range || !range.is_multiple_of(n - 1) { + return Err(RawError::Unsupported(format!("NEF: {n}-point lossy curve for {bits}-bit data"))); + } + let step = range / (n - 1); + let mut curve = Vec::with_capacity(range + 1); + for pair in points.windows(2) { + let &[a, b] = pair else { continue }; + let (a, b) = (a as u32, b as u32); + for k in 0..step as u32 { + let s = step as u32; + curve.push(((a * (s - k) + b * k) / s) as u16); + } + } + curve.extend(points.last().copied()); + Encoding::Lossy(curve) + } + } + _ => return Err(RawError::Unsupported(format!("Nikon compressed NEF version {v0:#04x} {v1:#04x}"))), + }; + Ok(DecodeTable { encoding, seeds }) +} + +/// Decode a `w × h` Nikon Huffman-compressed strip `src` with `bits`-bit samples. +pub(crate) fn decode(src: &[u8], w: usize, h: usize, bits: u32, table: &DecodeTable) -> Result> { + if bits != 12 && bits != 14 { + return Err(RawError::Unsupported(format!("Nikon compressed NEF with {bits}-bit samples"))); + } + let n = w.checked_mul(h).filter(|&n| n > 0 && n <= MAX_SAMPLES).ok_or(RawError::Limit("NEF image size"))?; + // every code word is at least 2 bits long: a strip that can't hold the image is truncated (also bounds the + // allocation by the file size) + if n / 4 > src.len() { + return Err(RawError::Corrupt(format!("NEF: {} bytes of compressed data for {w}x{h} pixels", src.len()))); + } + let huff = Huffman::new( + &match (&table.encoding, bits) { + (Encoding::Lossless, 12) => LOSSLESS_12, + (Encoding::Lossless, _) => LOSSLESS_14, + (Encoding::Lossy(_), 12) => LOSSY_12, + (Encoding::Lossy(_), _) => LOSSY_14, + } + .codes(), + )?; + let (lut, max): (&[u16], i32) = match &table.encoding { + Encoding::Lossless => (&[], (1i32 << bits) - 1), + Encoding::Lossy(curve) => (curve, curve.len() as i32 - 1), + }; + let mut out = vec![0u16; n]; + let mut stream = Bits::new(src); + let mut vpred = table.seeds; + let mut clipped = 0usize; + for (y, row) in out.chunks_exact_mut(w).enumerate() { + let seeds = vpred.get_mut(y & 1).ok_or(RawError::Limit("NEF row"))?; + let mut hpred = [0i32; 2]; + for (x, o) in row.iter_mut().enumerate() { + let d = diff(&mut stream, &huff).ok_or_else(|| RawError::Corrupt(format!("NEF: invalid Huffman code at row {y}")))?; + let p = if x < 2 { + let s = seeds.get_mut(x).ok_or(RawError::Limit("NEF column"))?; + *s += d; + hpred[x & 1] = *s; + *s + } else { + hpred[x & 1] += d; + hpred[x & 1] + }; + let v = p.clamp(0, max); + clipped += (v != p) as usize; + *o = if lut.is_empty() { v as u16 } else { lut.get(v as usize).copied().unwrap_or(0) }; + } + // the camera pads the strip; reading more than a few bytes past its end means truncated or corrupt data + if stream.consumed_bits() > src.len() * 8 + 64 { + return Err(RawError::Corrupt(format!("NEF: compressed data ends at row {y} of {h}"))); + } + } + // a wrong table / corrupt stream drifts out of range quickly; real files stay inside (a handful of edge pixels) + if clipped > n / 1000 + 16 { + return Err(RawError::Corrupt(format!("NEF: {clipped} decoded values out of range"))); + } + Ok(out) +} + +#[cfg(test)] +mod tests { + use super::*; + + /// MSB-first bit writer for the test encoder. + #[derive(Default)] + struct Writer { + out: Vec, + acc: u64, + n: u32, + } + + impl Writer { + fn put(&mut self, v: u32, len: u32) { + for i in (0..len).rev() { + self.acc = (self.acc << 1) | ((v >> i) & 1) as u64; + self.n += 1; + if self.n == 8 { + self.out.push(self.acc as u8); + self.acc = 0; + self.n = 0; + } + } + } + fn finish(mut self) -> Vec { + if self.n > 0 { + self.out.push((self.acc << (8 - self.n)) as u8); + } + self.out.extend_from_slice(&[0; 4]); + self.out + } + } + + fn category(d: i32) -> u32 { + 32 - d.unsigned_abs().leading_zeros() + } + + /// Encode `values` (decoded domain: samples or curve indices) the way the camera does. + fn encode(values: &[i32], w: usize, table: &Table, seeds: [[i32; 2]; 2]) -> Vec { + let codes = table.codes(); + let mut wr = Writer::default(); + let mut vpred = seeds; + for (y, row) in values.chunks(w).enumerate() { + for (x, &v) in row.iter().enumerate() { + let pred = if x < 2 { vpred[y & 1][x] } else { row[x - 2] }; + if x < 2 { + vpred[y & 1][x] = v; + } + let d = v - pred; + let k = category(d); + let (code, len) = codes[k as usize]; + assert!(len > 0, "no code for category {k}"); + wr.put(code as u32 >> (12 - len), len as u32); + let extra = if d < 0 { d + (1 << k) - 1 } else { d }; + wr.put(extra as u32, k); + } + } + wr.finish() + } + + /// Deterministic test image: smooth gradient + texture + hard edges, within `0..=max`. + fn image(w: usize, h: usize, max: i32, seed: u32) -> Vec { + let mut s = seed; + (0..w * h) + .map(|i| { + let (x, y) = ((i % w) as i32, (i / w) as i32); + s = s.wrapping_mul(1_664_525).wrapping_add(1_013_904_223); + let noise = (s >> 24) as i32 % 9 - 4; + let base = (x * 37 + y * 11) % (max / 2) + if (x / 7 + y / 5) % 3 == 0 { max / 3 } else { 0 }; + (base + noise).clamp(0, max) + }) + .collect() + } + + fn lossless(bits: u32) -> DecodeTable { + let s = 1 << (bits - 3); + DecodeTable { encoding: Encoding::Lossless, seeds: [[s, s], [s, s]] } + } + + #[test] + fn tables_are_complete_prefix_codes() { + for t in [LOSSLESS_12, LOSSLESS_14, LOSSY_12, LOSSY_14] { + assert_eq!(t.counts.iter().map(|&c| c as usize).sum::(), t.symbols.len()); + let kraft: f64 = t.counts.iter().enumerate().map(|(i, &c)| c as f64 / (1u64 << (i + 1)) as f64).sum(); + assert!(kraft <= 1.0); + assert!(Huffman::new(&t.codes()).is_ok()); + } + assert_eq!(LOSSLESS_14.codes()[7], (0b00 << 10, 2)); + assert_eq!(LOSSLESS_14.codes()[0], (0b111110 << 6, 6)); + assert_eq!(LOSSY_12.codes()[10], (0b111111110 << 3, 9)); + } + + #[test] + fn lossless_roundtrip_12_and_14() { + for (bits, table) in [(12, LOSSLESS_12), (14, LOSSLESS_14)] { + let (w, h) = (64, 9); + let max = (1 << bits) - 1; + let mut img = image(w, h, max, bits); + // extreme jumps exercise the longest categories + img[5] = 0; + img[7] = max; + img[w + 9] = max; + let t = lossless(bits); + let src = encode(&img, w, &table, t.seeds); + let out = decode(&src, w, h, bits, &t).unwrap(); + assert_eq!(out, img.iter().map(|&v| v as u16).collect::>(), "{bits}-bit"); + } + } + + fn lossy_table(bits: u32, order: ByteOrder) -> Vec { + // version 0x44 0x20, seeds, 257 points of a concave curve, padding up to the split value (0) + let mut t = vec![0x44, 0x20]; + let put = |t: &mut Vec, v: u16| t.extend_from_slice(&if order == ByteOrder::Big { v.to_be_bytes() } else { v.to_le_bytes() }); + for _ in 0..4 { + put(&mut t, 300); + } + put(&mut t, 257); + let max = ((1u32 << bits) - 1) as f64; + for i in 0..257u32 { + put(&mut t, (max * (i as f64 / 64.0).min(1.0).sqrt()) as u16); + } + t.resize(624, 0); + t + } + + #[test] + fn lossy_type2_curve_and_roundtrip() { + for (bits, table) in [(12u32, LOSSY_12), (14, LOSSY_14)] { + for order in [ByteOrder::Big, ByteOrder::Little] { + let t = parse_table(&lossy_table(bits, order), order, bits).unwrap(); + assert_eq!(t.seeds, [[300, 300], [300, 300]]); + let Encoding::Lossy(curve) = &t.encoding else { panic!() }; + let step = (1usize << bits) / 256; + assert_eq!(curve.len(), (1 << bits) + 1); + assert_eq!(curve[64 * step], (1 << bits) - 1); + assert!(curve.windows(2).all(|p| p[0] <= p[1])); + // the encoded range is the curve's rising part + let (w, h) = (48, 7); + let img = image(w, h, 64 * step as i32 - 1, 7); + let src = encode(&img, w, &table, t.seeds); + let out = decode(&src, w, h, bits, &t).unwrap(); + assert_eq!(out, img.iter().map(|&v| curve[v as usize]).collect::>()); + } + } + } + + #[test] + fn lossy_type1_full_curve() { + let mut t = vec![0x44, 0x10, 0, 10, 0, 10, 0, 10, 0, 10, 0, 5]; + for v in [0u16, 100, 400, 900, 1600] { + t.extend_from_slice(&v.to_be_bytes()); + } + let parsed = parse_table(&t, ByteOrder::Big, 12).unwrap(); + assert_eq!(parsed.encoding, Encoding::Lossy(vec![0, 100, 400, 900, 1600])); + let img = vec![0, 4, 1, 3, 2, 2, 4, 0]; + let src = encode(&img, 4, &LOSSY_12, parsed.seeds); + assert_eq!(decode(&src, 4, 2, 12, &parsed).unwrap(), vec![0, 1600, 100, 900, 400, 400, 1600, 0]); + } + + #[test] + fn rejects_unsupported_and_malformed_tables() { + let unsupported = |r: Result| matches!(r, Err(RawError::Unsupported(_))); + let corrupt = |r: Result| matches!(r, Err(RawError::Corrupt(_))); + let mut split = lossy_table(12, ByteOrder::Big); + split[562..564].copy_from_slice(&345u16.to_be_bytes()); + assert!(unsupported(parse_table(&split, ByteOrder::Big, 12))); + assert!(unsupported(parse_table(&[0x49, 0x30, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], ByteOrder::Big, 12))); + assert!(unsupported(parse_table(&lossy_table(12, ByteOrder::Big), ByteOrder::Big, 16))); + assert!(corrupt(parse_table(&[], ByteOrder::Big, 12))); + assert!(corrupt(parse_table(&[0x46, 0x30, 8], ByteOrder::Big, 14))); + assert!(corrupt(parse_table(&lossy_table(12, ByteOrder::Big)[..100], ByteOrder::Big, 12))); + // 300 points don't divide the 12-bit range + let mut odd = lossy_table(12, ByteOrder::Big); + odd[10..12].copy_from_slice(&300u16.to_be_bytes()); + assert!(parse_table(&odd, ByteOrder::Big, 12).is_err()); + assert!(parse_table(&[0x44, 0x20, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0], ByteOrder::Big, 12).is_err()); + } + + /// A minimal NEF: CFA SubIFD with one compressed strip, Exif maker note (`Nikon\0` v2 header + embedded TIFF in + /// `mn_order`) holding the linearization table. + fn nef_file(strip: Vec, w: u32, h: u32, bits: u16, table: Vec, mn_order: ByteOrder) -> Vec { + use lightcraft_tiff::tags::{self as t, photometric}; + use lightcraft_tiff::{IfdBuilder, ImageData, TiffWriter, Value}; + let mut mn = IfdBuilder::new(); + mn.set(0x0096, Value::Undefined(table)); + let mut note = b"Nikon\0\x02\x10\0\0".to_vec(); + note.extend(TiffWriter::new(mn_order, false).write(&[mn]).unwrap()); + let mut exif = IfdBuilder::new(); + exif.set(t::MAKER_NOTE, Value::Undefined(note)); + let mut raw = IfdBuilder::new(); + raw.set(t::NEW_SUBFILE_TYPE, Value::Long(vec![0])); + raw.set(t::IMAGE_WIDTH, Value::Long(vec![w])); + raw.set(t::IMAGE_LENGTH, Value::Long(vec![h])); + raw.set(t::BITS_PER_SAMPLE, Value::Short(vec![bits])); + raw.set(t::COMPRESSION, Value::Short(vec![t::compression::NIKON])); + raw.set(t::PHOTOMETRIC, Value::Short(vec![photometric::CFA])); + raw.set(t::CFA_REPEAT_PATTERN_DIM, Value::Short(vec![2, 2])); + raw.set(t::CFA_PATTERN_EP, Value::Byte(vec![0, 1, 1, 2])); + raw.set_image(ImageData::Strips { rows_per_strip: h, strips: vec![strip] }); + let mut ifd0 = IfdBuilder::new(); + ifd0.set(t::MAKE, Value::Ascii("NIKON CORPORATION".into())); + ifd0.set(t::MODEL, Value::Ascii("NIKON TEST".into())); + ifd0.set_child(t::EXIF_IFD, exif); + ifd0.add_sub_ifd(raw); + TiffWriter::new(ByteOrder::Big, false).write(&[ifd0]).unwrap() + } + + #[test] + fn decodes_through_the_nef_container() { + // lossless 14-bit, big-endian maker note + let (w, h) = (40usize, 6usize); + let img = image(w, h, 16383, 11); + let mut table = vec![0x46, 0x30]; + for _ in 0..4 { + table.extend_from_slice(&2048u16.to_be_bytes()); + } + table.resize(46, 0); + let src = encode(&img, w, &LOSSLESS_14, [[2048; 2]; 2]); + let bytes = nef_file(src, w as u32, h as u32, 14, table, ByteOrder::Big); + let r = crate::decode(&bytes).unwrap(); + assert_eq!(r.data, crate::RawData::U16(img.iter().map(|&v| v as u16).collect())); + assert_eq!(r.cfa.as_ref().unwrap().name(), "RGGB"); + assert_eq!(crate::probe_info(&bytes).unwrap(), r.info()); + // lossy 12-bit type 2, little-endian maker note (newer bodies) + let table = lossy_table(12, ByteOrder::Little); + let t = parse_table(&table, ByteOrder::Little, 12).unwrap(); + let Encoding::Lossy(curve) = &t.encoding else { panic!() }; + let img = image(w, h, 1023, 5); + let src = encode(&img, w, &LOSSY_12, t.seeds); + let r = crate::decode(&nef_file(src.clone(), w as u32, h as u32, 12, table.clone(), ByteOrder::Little)).unwrap(); + assert_eq!(r.data, crate::RawData::U16(img.iter().map(|&v| curve[v as usize]).collect())); + // the split variant falls back to the preview + let mut split = table; + split[562..564].copy_from_slice(&3u16.to_le_bytes()); + let e = crate::decode(&nef_file(src, w as u32, h as u32, 12, split, ByteOrder::Little)); + assert!(matches!(e, Err(RawError::Unsupported(_))), "{e:?}"); + } + + #[test] + fn truncated_and_corrupted_streams_error_never_panic() { + let (w, h) = (64, 16); + let t = lossless(14); + let img = image(w, h, 16383, 3); + let src = encode(&img, w, &LOSSLESS_14, t.seeds); + // truncation: too short for the image, or ends early + for cut in [0, 1, 10, src.len() / 2, src.len() - 40] { + assert!(decode(&src[..cut], w, h, 14, &t).is_err(), "cut {cut}"); + } + // garbage and bit flips: an error or (rarely) some image, never a panic + let mut s = 12345u32; + for i in 0..200 { + let mut bad = src.clone(); + for _ in 0..1 + i % 8 { + s = s.wrapping_mul(1_103_515_245).wrapping_add(12345); + let at = (s >> 8) as usize % bad.len(); + bad[at] ^= 1 << (s % 8); + } + let _ = decode(&bad, w, h, 14, &t); + let noise: Vec = (0..src.len()).map(|k| ((k as u32).wrapping_mul(2_654_435_761) >> (i % 24)) as u8).collect(); + let _ = decode(&noise, w, h, 14, &t); + let _ = decode(&noise, w, h, 12, &lossless(12)); + } + // all-ones never forms a valid lossy 12-bit code + assert!(decode(&[0xff; 64], 8, 8, 12, &parse_table(&lossy_table(12, ByteOrder::Big), ByteOrder::Big, 12).unwrap()).is_err()); + // absurd dimensions are limited before allocating + assert!(decode(&src, usize::MAX, 2, 14, &t).is_err()); + assert!(decode(&src, 1 << 20, 1 << 20, 14, &t).is_err()); + assert!(decode(&src, 0, 2, 14, &t).is_err()); + } +} diff --git a/crates/raw/src/vendor/pef.rs b/crates/raw/src/vendor/pef.rs index 78cfa87a..c9421d3b 100644 --- a/crates/raw/src/vendor/pef.rs +++ b/crates/raw/src/vendor/pef.rs @@ -99,6 +99,10 @@ impl<'a> Bits<'a> { self.skip(k); v } + /// Bits handed out so far (may exceed the source length: zeros are read past its end). + pub fn consumed_bits(&self) -> usize { + self.pos * 8 - self.n as usize + } pub fn overrun(&self) -> bool { self.pos > self.src.len() + 8 } diff --git a/crates/raw/tests/corpus.rs b/crates/raw/tests/corpus.rs index 2db239ed..fa8a2626 100644 --- a/crates/raw/tests/corpus.rs +++ b/crates/raw/tests/corpus.rs @@ -23,8 +23,6 @@ const KNOWN_UNSUPPORTED: &[&str] = &[ "rw2-panasonic-gx80", // " "rw2-panasonic-g9-b", // " "orf-olympus-em", // Olympus compressed ORF - "nef-nikon-d5100-lossless", // Nikon Huffman NEF - "nef-nikon-d7000-lossy", // " "sraw", // Canon sRAW / mRAW ]; @@ -94,3 +92,56 @@ fn corpus_raw_decodes() { } eprintln!("corpus/raw: {ok} decoded, {unsupported} known-unsupported (preview only)"); } + +/// Green-channel means of 32×32 blocks. +fn block_means(img: &lightcraft_raw::RawImage) -> Vec { + let lightcraft_raw::RawData::U16(d) = &img.data else { panic!("float data") }; + let (w, h, b) = (img.width, img.height, 32); + let mut out = Vec::new(); + for by in 0..h / b { + for bx in 0..w / b { + let (mut s, mut n) = (0f64, 0f64); + for y in by * b..by * b + b { + for x in (bx * b..bx * b + b).step_by(2) { + s += d[y * w + x + (y & 1)] as f64; + n += 1.0; + } + } + out.push(s / n); + } + } + out +} + +fn correlation(a: &[f64], b: &[f64]) -> f64 { + let n = a.len() as f64; + let (ma, mb) = (a.iter().sum::() / n, b.iter().sum::() / n); + let cov: f64 = a.iter().zip(b).map(|(x, y)| (x - ma) * (y - mb)).sum(); + let va: f64 = a.iter().map(|x| (x - ma).powi(2)).sum(); + let vb: f64 = b.iter().map(|y| (y - mb).powi(2)).sum(); + cov / (va * vb).sqrt() +} + +/// raw.pixls.us has the same D5100 scene as 14-bit lossless compressed and uncompressed NEF: the Huffman decode must +/// match the uncompressed image (up to the small differences between two exposures). +#[test] +fn corpus_nef_compressed_matches_uncompressed() { + let dir = corpus_root().join("raw"); + let (Ok(a), Ok(b)) = (std::fs::read(dir.join("nef-nikon-d5100-lossless.nef")), std::fs::read(dir.join("nef-nikon-d5100-uncompressed.nef"))) + else { + eprintln!("skip: D5100 NEF pair absent"); + return; + }; + let (a, b) = (decode(&a).unwrap(), decode(&b).unwrap()); + assert_eq!((a.width, a.height, a.bits), (b.width, b.height, b.bits)); + let r = correlation(&block_means(&a), &block_means(&b)); + eprintln!("D5100 lossless vs uncompressed: block correlation {r:.4}"); + assert!(r > 0.98, "correlation {r}"); + // the lossy 12-bit D7000 file decodes into the curve's range and isn't flat + if let Ok(c) = std::fs::read(dir.join("nef-nikon-d7000-lossy12.nef")) { + let c = decode(&c).unwrap(); + let m = block_means(&c); + let (lo, hi) = m.iter().fold((f64::MAX, 0f64), |(l, h), &v| (l.min(v), h.max(v))); + assert!(hi <= 4095.0 && hi - lo > 500.0, "D7000 block means {lo}..{hi}"); + } +} diff --git a/docs/parity.md b/docs/parity.md index 0e63ca1f..6479d7d6 100644 --- a/docs/parity.md +++ b/docs/parity.md @@ -57,9 +57,9 @@ Weighted completion (✅ = 1, 🟡 = ½, 🚫 left out): **79.4%** of 502 in-sco Ordered by tier, then user value, then (low) effort. Take the first one nobody is working on. -1. **LR-IMP-FORMATS** (P0) — CR3, compressed NEF/RAF/ORF, RW2 v4 (preview only today: such photos are marked - `preview_only` and the UI says they are edited from the camera's embedded JPEG, issue #10); HEIC/AVIF decode (no - permissive pure-Rust decoder yet). High value, high effort (clean sources needed). +1. **LR-IMP-FORMATS** (P0) — CR3, compressed RAF/ORF, RW2 v4, Nikon "lossy after split" NEF (preview only today: + such photos are marked `preview_only` and the UI says they are edited from the camera's embedded JPEG); HEIC/AVIF + decode (no permissive pure-Rust decoder yet). High value, high effort (clean sources needed). 2. **LR-EDIT-OPTICS-PROFILE** (P1) — a lens-profile database of our own (embedded DNG/maker corrections work today). ## Shortcuts: conflicts and missing bindings @@ -110,7 +110,7 @@ under the UI command that wraps it (a no-op error); the UI command now wins. | LR-IMP-PROFILES | Import profiles & presets | P1 | ✅ | `cmd:file.importPresets`, `cmd:preset.import`, `cmd:profile.import`, `crates/engine/src/preset_import.rs`, `crates/engine/src/preset_luminar.rs` | presets: .lcpreset, XMP, classic .lrtemplate, photos carrying edits ("DNG presets"), Luminar looks (.lmp, .mplumpack collections; sliders with a counterpart), .zip bundles, folders (folder → group), drag & drop; masks carried over; unmapped settings reported; profiles: .cube 3D LUTs; Adobe profile formats deliberately unsupported | | LR-IMP-LOCAL | Work on files in place | P0 | ✅ | `cmd:library.browse`, `cmd:photo.addToLibrary`, `cmd:library.import` (mode add), `cmd:local.addRoot`, `cmd:local.hide`, `cmd:local.restoreHidden`, `crates/engine/src/cmd/browse.rs`, `crates/ui-egui/src/panels/left.rs` (`local_section`), `crates/ui-egui/src/panels/grid.rs` (`folder_header`) | Local: browse Pictures / Desktop / Downloads / Home or any folder without adding it (breadcrumb, Include subfolders, Add N to My Photos); browsed photos stay out of All Photos, albums and counts; folders picked with Browse Folder… (or Keep in Local / Add to Local on a folder row) stay listed across restarts, and browsing a folder inside a listed one highlights it inside that tree (opened down to it, siblings reachable) instead of adding a root; edits go to XMP sidecars; importing promotes them; right-click a top-level location → Remove from Local hides the shortcut only (nothing on disk or in the catalog changes), saved with the UI state, and “Show N hidden locations” (or Browse Folder… on it) brings it back; a folder is one location however its path is spelled (`/` or `\`, trailing separator, `.`/`..`, drive-letter case; `folder_key` in `crates/catalog/src/query.rs`); browsed photos nobody changed are forgotten once their folder has not been browsed for 30 days (Settings → Performance → Local folders, 0 = never; checked when the library opens, or `cmd:library.forgetLocal` with `dryRun`; files and sidecars stay, browsing again brings them back; see `crates/catalog/src/local.rs` for what counts as a change) | | LR-IMP-SIDECAR-SPLIT | Separate XMP sidecar variants | P2 | ⬜ | `cmd:library.xmpPreferences` | sidecar naming option exists (stem/full), no split sidecars | -| LR-IMP-FORMATS | Supported formats | P0 | 🟡 | `crates/codecs/src/lib.rs`, `crates/raw/src/lib.rs` | JPEG, PNG, TIFF, WebP, JXL, PSD, GIF, BMP; DNG (incl. lossy / Smart Preview DNG), CR2, ARW, NEF, RAF, RW2, PEF, ORF. Missing: CR3, compressed NEF/RAF/ORF, RW2 v4 (preview only: imported with `preview_only` = the decoder's reason, rendered as a rendered JPEG, flagged by a grid badge and loupe/Edit/Info notices, `previewOnly` in `catalog.query`; Reload clears it once the file decodes), HEIC/AVIF decode | +| LR-IMP-FORMATS | Supported formats | P0 | 🟡 | `crates/codecs/src/lib.rs`, `crates/raw/src/lib.rs` | JPEG, PNG, TIFF, WebP, JXL, PSD, GIF, BMP; DNG (incl. lossy / Smart Preview DNG), CR2, ARW, NEF (uncompressed, lossless and lossy compressed 12/14-bit, `crates/raw/src/vendor/nefc.rs`), RAF, RW2, PEF, ORF. Missing: CR3, compressed RAF/ORF, RW2 v4, NEF "lossy after split" (preview only: imported with `preview_only` = the decoder's reason, rendered as a rendered JPEG, flagged by a grid badge and loupe/Edit/Info notices, `previewOnly` in `catalog.query`; Reload clears it once the file decodes), HEIC/AVIF decode | | LR-IMP-CULL-AT-IMPORT | Culling analysis at import | P2 | 🟡 | `cmd:photo.analyze` | run Assisted Culling on the imported photos (they're selected after an import); not automatic | | LR-IMP-MOVE | Move on import [Classic] | P1 | ✅ | `cmd:library.import` (`mode` move), `crates/engine/src/import_move.rs`, `crates/engine/src/tests_import_move.rs`, `crates/ui-egui/src/import.rs` | Transfer ▸ Move with Copy's destination, folders (day / month / one folder / custom template), rename template and example destination; XMP sidecars (both namings) move along; each source is removed only after its destination is written (hard link on the same volume, else copied, synced and compared byte for byte) and its catalog record is saved; failed, duplicate, unchecked files keep their sources, taken names get -1, -2…, never overwritten; files already in the destination/library are added in place; sources that can't be removed (read-only card) are kept and reported (`kept`). Undo removes the photos from the library but leaves the moved files at the destination. No Copy as DNG while moving | | LR-IMP-DNG-CONVERT | Convert to DNG on import [Classic] | P2 | ✅ | `cmd:library.import` (`dng`), `crates/ui-egui/src/import.rs` | copy imports: Raw files ▸ Copy as DNG (lossless; the card is untouched) | diff --git a/docs/whats-new.md b/docs/whats-new.md index a6327a73..05de11b2 100644 --- a/docs/whats-new.md +++ b/docs/whats-new.md @@ -45,3 +45,7 @@ - Slideshow, second window, All Metadata, System Info. - Edit in External Editor (⇧⌘E): a 16-bit TIFF copy, stacked, refreshed when you come back. - Lossy DNG files and Smart Previews open as raw photos. +- Compressed Nikon NEFs (lossless and lossy compressed, 12- and 14-bit — e.g. D3200, D5100, D7000, D750, D850, Z 50) + now develop from the raw data instead of the camera's embedded JPEG, so a B&W or other picture style set in the + camera no longer gets baked in. Photos already imported as "preview only" switch over on Reload. (Files that + Nikon splits into two differently compressed halves still use the preview for now.)