diff --git a/.gitignore b/.gitignore
index 51531a34..8c2047fe 100644
--- a/.gitignore
+++ b/.gitignore
@@ -8,6 +8,9 @@ plan/
.DS_Store
.claude/
/craft-fonts
+# nix build / nix develop
+/result
+/result-*
# model weights are never committed (SAM 3: docs/ai-masks.md)
*.safetensors
*.safetensors.part
diff --git a/AGENTS.md b/AGENTS.md
index 588b6f41..eb5b9430 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -119,4 +119,7 @@ of it. Full standard: `../craftrules/standards/never-crash.md`
## Map of the code
`geom`, `color`, `raster`, `tiff` (L0) → `raw`, `codecs`, `meta`, `develop` (L1) → `pipeline` → `catalog` → `engine`
→ `ui-egui`, `mcp` (L5) → apps `lightcraft` (desktop), `lightcraft-cli` (render/commands/MCP). `scenes` generates demo
-photos. `xtask` = tooling (`ci`, `layers`, `assets`, `parity`, `wasm`, `corpus`, `stats`).
+photos. `xtask` = tooling (`ci`, `layers`, `assets`, `parity`, `wasm`, `corpus`, `stats`). `flake.nix` +
+`nix/package.nix` = the Nix package (`nix build` builds both binaries with the craft-fonts input, installs the
+desktop file/icons/AppStream metadata and runs `cargo test --workspace`; `nix develop` = dev shell). Community-maintained and not
+in CI: it may lag behind the workspace; see README → Quick start.
diff --git a/CLAUDE.md b/CLAUDE.md
index 588b6f41..eb5b9430 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -119,4 +119,7 @@ of it. Full standard: `../craftrules/standards/never-crash.md`
## Map of the code
`geom`, `color`, `raster`, `tiff` (L0) → `raw`, `codecs`, `meta`, `develop` (L1) → `pipeline` → `catalog` → `engine`
→ `ui-egui`, `mcp` (L5) → apps `lightcraft` (desktop), `lightcraft-cli` (render/commands/MCP). `scenes` generates demo
-photos. `xtask` = tooling (`ci`, `layers`, `assets`, `parity`, `wasm`, `corpus`, `stats`).
+photos. `xtask` = tooling (`ci`, `layers`, `assets`, `parity`, `wasm`, `corpus`, `stats`). `flake.nix` +
+`nix/package.nix` = the Nix package (`nix build` builds both binaries with the craft-fonts input, installs the
+desktop file/icons/AppStream metadata and runs `cargo test --workspace`; `nix develop` = dev shell). Community-maintained and not
+in CI: it may lag behind the workspace; see README → Quick start.
diff --git a/Cargo.lock b/Cargo.lock
index 3f086161..8de1ee49 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -2900,6 +2900,17 @@ dependencies = [
"web-time",
]
+[[package]]
+name = "lightcraft-fetch"
+version = "0.2.1"
+dependencies = [
+ "log",
+ "rustls",
+ "rustls-rustcrypto",
+ "sha2",
+ "webpki-roots",
+]
+
[[package]]
name = "lightcraft-geom"
version = "0.2.1"
@@ -3039,14 +3050,11 @@ version = "0.2.1"
dependencies = [
"candle-core",
"candle-nn",
+ "lightcraft-fetch",
"log",
- "rustls",
- "rustls-rustcrypto",
"serde",
"serde_json",
- "sha2",
"thiserror 2.0.21",
- "webpki-roots",
]
[[package]]
diff --git a/Cargo.toml b/Cargo.toml
index d59f8f45..7af2f898 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -29,6 +29,7 @@ lightcraft-merge = { path = "crates/merge" }
lightcraft-engine = { path = "crates/engine" }
lightcraft-ui-egui = { path = "crates/ui-egui" }
lightcraft-mcp = { path = "crates/mcp" }
+lightcraft-fetch = { path = "crates/fetch" }
lightcraft-segment = { path = "crates/segment" }
# foundation
diff --git a/README.md b/README.md
index a1dea118..0cf23b19 100644
--- a/README.md
+++ b/README.md
@@ -258,8 +258,8 @@ lightcraft --control 7980 ~/Pictures/trip
## Fast, native, private
- **Pure Rust, no C.** Our own RAW decoders (DNG, Canon CR2, Sony ARW, Nikon NEF, Fujifilm RAF incl. X-Trans,
- Panasonic RW2, Pentax PEF, Olympus ORF), our own colour science, our own pipeline. JPEG, PNG, TIFF, WebP, PSD
- composites and JPEG XL open today.
+ Panasonic RW2 / Leica RWL, Pentax PEF, Olympus ORF), our own colour science, our own pipeline. JPEG, PNG, TIFF, WebP,
+ PSD composites and JPEG XL open today.
- **Scene-referred & wide-gamut.** Linear Rec.2020 float internally, Bradford-adapted white balance, gamut mapping
instead of clipping, a filmic shoulder for raw and pixel-exact pass-through for JPEGs you haven't touched.
- **Resolution-independent edits.** Radii and brush sizes are relative to the image, so a 400 px preview, your
@@ -301,9 +301,9 @@ LightCraft is young and moving fast. **Where we honestly stand** (details in the
| Masking: brush, linear, radial, luminance/colour range, add/subtract/intersect | ✅ (AI subject/sky use classical heuristics for now) |
| Crop, straighten tool + auto straighten, flip, rotate, aspect ratios, overlays | ✅ |
| Profiles (Color, Neutral, Vivid, Landscape, Portrait, Monochrome: our own looks), presets, versions, history, copy/paste/sync settings | ✅ |
-| Camera colour: DNG files use their own matrices | ✅ · our own calibration for other raws ⬜ (top priority; neutral fallback today) |
+| Camera colour: DNG files use their own matrices | ✅ · our own calibration for other raws ⬜ (top priority; ARW, NEF and RW2 start from a look fitted to their own JPEG, other raws from a neutral fallback) |
| Native macOS menu bar (generated from the command registry), control channel + every widget addressable, headless UI snapshots | ✅ |
-| RAW: DNG, CR2, ARW, NEF (uncompressed + lossless/lossy compressed), Fujifilm RAF (uncompressed, Bayer + X-Trans), Panasonic RW2, Pentax PEF, Olympus ORF (uncompressed); embedded previews for every format incl. CR3 | ✅ · CR3, compressed RAF/ORF decode ⬜ |
+| RAW: DNG, CR2, ARW, NEF (uncompressed + lossless/lossy compressed), Fujifilm RAF (uncompressed, Bayer + X-Trans), Panasonic RW2 / Leica RWL / Panasonic RAW (every raw format, DMC-LX1 to DC-S1RM2), Pentax PEF, Olympus ORF (uncompressed); embedded previews for every format incl. CR3 | ✅ · CR3, compressed RAF/ORF decode ⬜ |
| Detail: sharpening, luminance + colour noise reduction | ✅ · AI Denoise, Super Resolution ⬜ |
| Remove / Heal / Clone spots (auto source), Visualize Spots, Red Eye and Pet Eye (auto pupil detection, catchlight) | ✅ · content-aware fill, spot pin editing 🚧 |
| Export: JPEG / PNG / TIFF / WebP / AVIF / DNG / original, sizing, file-size limit, output sharpening, naming templates, batch, metadata policy, text or image watermark | ✅ · HDR export ⬜ |
@@ -357,6 +357,31 @@ Language**; the choice applies immediately and persists. See [docs/localization.
The web build needs the `wasm32-unknown-unknown` target and the matching `wasm-bindgen` CLI
(`cargo xtask web` prints the exact install command); see [docs/web.md](docs/web.md).
+**Nix** builds the desktop app and `lightcraft-cli` (the Nix build always includes the craft-fonts input, so Japanese
+text has glyphs):
+
+```sh
+nix run github:storytold/lightcraft # the desktop app
+nix build github:storytold/lightcraft # → ./result/bin/{lightcraft,lightcraft-cli}
+nix develop github:storytold/lightcraft # rust toolchain + native deps + fonts
+```
+
+In a flake configuration (NixOS, home-manager, nix-darwin):
+
+```nix
+# flake.nix
+inputs.lightcraft.url = "github:storytold/lightcraft";
+# optional: build against your own nixpkgs instead of the one LightCraft pins
+# inputs.lightcraft.inputs.nixpkgs.follows = "nixpkgs";
+
+# then, in a NixOS or home-manager module (where `inputs` is in scope):
+nixpkgs.overlays = [ inputs.lightcraft.overlays.default ]; # makes `pkgs.lightcraft` available
+environment.systemPackages = [ pkgs.lightcraft ]; # home-manager: home.packages = [ pkgs.lightcraft ];
+```
+
+`nix build` installs the same desktop file, hicolor icons and AppStream metadata as the .deb/.rpm, and runs
+`cargo test --workspace` as its check phase (skip it with `pkgs.lightcraft.overrideAttrs { doCheck = false; }`).
+
**Keyboard:** G grid · D detail · E edit · C crop · M masking ·
Shift+P presets · \\ original · Y before/after · Z zoom ·
J clipping · ⌘Z undo · ⌘/ all shortcuts.
diff --git a/ROADMAP.md b/ROADMAP.md
index f2dcb9bc..dcfba518 100644
--- a/ROADMAP.md
+++ b/ROADMAP.md
@@ -23,8 +23,8 @@ duplicate Local entries #22, black GPU exports on an Intel iGPU #78).
| Dimension | Estimate | What's true today | Biggest gaps |
|---|---:|---|---|
| **Feature checklist** | 79% | P0 core and P1 nearly complete: import (Add / Copy / Move, templates, devices), library, grid/loupe/compare/survey, every Edit slider, curves, colour grading, masking tools, crop/Upright, heal/clone, presets/profiles, versions/history, sync, export, menus, shortcuts | P1: lens-profile database, content-aware fill (patch synthesis), video playback/trim |
-| **RAW coverage** (formats people shoot) | ~50% | DNG (all kinds), CR2, ARW, NEF (uncompressed + Huffman lossless/lossy), uncompressed RAF/ORF, packed RW2, PEF; every container's embedded preview (incl. CR3) | **CR3** (every Canon since ~2018), compressed RAF/ORF, RW2 v4, Nikon lossy-after-split, Canon sRAW, HEIC/AVIF. NEFs labelled compressed but stored uncompressed (Z 6 packed 14-bit, D850 12-bit uncompressed). Per-model verification is thin (~40 corpus files vs >1,000 models) |
-| **Colour & image quality** | ~55–65% | Pipeline is complete and fast; GPU path CPU-exact within 1/255 | **No measured camera calibration database**: ARW and NEF have a guarded per-file embedded-JPEG colour estimate (docs/camera-preview-colour.md); other non-DNG raws and rejected estimates use a neutral matrix. Colour fidelity remains incomplete. No lens-profile database. No measured fidelity against Lightroom (tone, highlights, texture/clarity, NR, sharpening are tuned by eye) |
+| **RAW coverage** (formats people shoot) | ~55% | DNG (all kinds), CR2, ARW, NEF (uncompressed + Huffman lossless/lossy), uncompressed RAF/ORF, RW2 / RWL / Panasonic RAW (every raw format, checked on 178 files from 118 Panasonic and Leica bodies), PEF; every container's embedded preview (incl. CR3) | **CR3** (every Canon since ~2018), compressed RAF/ORF, Nikon lossy-after-split, Canon sRAW, HEIC/AVIF. NEFs labelled compressed but stored uncompressed (Z 6 packed 14-bit, D850 12-bit uncompressed). Per-model verification is thin (~55 corpus files vs >1,000 models) |
+| **Colour & image quality** | ~55–65% | Pipeline is complete and fast; GPU path CPU-exact within 1/255 | **No measured camera calibration database**: ARW, NEF and RW2 have a guarded per-file embedded-JPEG colour estimate (docs/camera-preview-colour.md); other non-DNG raws and rejected estimates use a neutral matrix. Colour fidelity remains incomplete. No lens-profile database. No measured fidelity against Lightroom (tone, highlights, texture/clarity, NR, sharpening are tuned by eye) |
| **AI & computational** | ~15–20% | Assisted culling (focus, bursts), auto tone, HDR/panorama merge; subject/sky/background masks as classical heuristics | Real segmentation masks (subject, sky, people, objects, landscape, depth), AI denoise, super resolution, lens blur, generative remove, faces/people, natural-language search. **Blocked on a model strategy** (licensable weights or our own training; pure-Rust inference is feasible) |
| **Workflow & library** | ~85% (single machine) | Robust catalog (journal + snapshots, background compaction, crash-tested), 85k-photo libraries stay responsive, Local browsing with automatic cleanup, XMP interop, keywords, smart albums, Move import | Opening an 85k library takes 1.7–4.7 s; no cloud sync (out of scope), no tablet companion (#74, roadmap), shared albums, publish services, tethering |
| **Classic modules** | ~30% | Geotagging from GPX track logs, soft proofing (partial), slideshow (basic) | **Map view, Book, Print, Slideshow module, Web, publish services**: ~40 tracker rows ⬜ |
@@ -36,7 +36,7 @@ duplicate Local entries #22, black GPU exports on an Intel iGPU #78).
| User | Readiness | What blocks them |
|---|---:|---|
| JPEG / DNG shooter, single machine | ~85% | Fidelity polish, AI masks |
-| Nikon / Sony / older-Canon raw shooter | ~65% | Camera colour fidelity and coverage (ARW and NEF preview estimates are only a starting point) |
+| Nikon / Sony / Panasonic / older-Canon raw shooter | ~65% | Camera colour fidelity and coverage (ARW, NEF and RW2 preview estimates are only a starting point); Panasonic compacts and kit zooms also miss their embedded distortion correction |
| Canon CR3 / Fujifilm / Olympus shooter | ~35% | Their raws open as embedded previews only (CR3: the full-size JPEG with full metadata) |
| Lightroom Classic power user | ~45% | Print, Book, Map, publish, tethering |
| Relies on AI (masks, denoise) | ~25% | Object / Describe masks via optional SAM 3 (download not yet hosted); no AI denoise |
@@ -46,9 +46,9 @@ duplicate Local entries #22, black GPU exports on an Intel iGPU #78).
Priorities, in order. Each points at tracker rows in [`docs/parity.md`](docs/parity.md) → *Top gaps*.
1. **Camera colour calibration of our own** (LR-PROF-CAMERACOLOR, P0): fit each camera to its own embedded JPEG, use
- matrices the files carry themselves, then chart shots. Sony ARW and Nikon NEF have the file-local fit (matrix + tone
- curve from their own JPEG); generalise it to the other makes' raws (RW2, PEF, ORF…), then validate fidelity.
-2. **Raw formats, clean-room** (LR-IMP-FORMATS, P0): **CR3** first, then compressed RAF / ORF, RW2 v4, NEF
+ matrices the files carry themselves, then chart shots. Sony ARW, Nikon NEF and Panasonic RW2 have the file-local fit
+ (matrix + tone curve from their own JPEG); generalise it to the other makes' raws (PEF, ORF…), then validate fidelity.
+2. **Raw formats, clean-room** (LR-IMP-FORMATS, P0): **CR3** first, then compressed RAF / ORF, NEF
lossy-after-split, sRAW. Decided 2026-10-05: write our own decoders from prose descriptions (never decoder source,
no LGPL dependency); compressed NEF (#86) is the template.
3. **Verified camera coverage** (LR-IMP-CAMERA-COVERAGE, P0): a CC0 sample per model in the corpus, each decoded and
@@ -84,7 +84,7 @@ hardening (#78), copyright metadata (#51), GPX geotagging (#60), import tag help
| M8 | Heal / Remove | content-aware remove (PatchMatch), heal, clone, brush spots, visualize spots, red/pet eye | 6–10 | 🚧 (heal, clone, auto source, visualize spots, red/pet eye ✅; PatchMatch remove ⬜) |
| M9 | Presets, profiles, versions, sync | preset browser + amount, create/import presets, profile browser, versions, history, copy/paste/sync settings | 5–8 | ✅ |
| M10 | Export & share | export dialog (JPEG/PNG/TIFF/DNG/AVIF/JXL/original), sizing, sharpening, metadata, watermark, naming, batch jobs, XMP sidecars, HDR export | 6–10 | 🚧 (all formats incl. DNG/original, sizing, presets, background jobs ✅; JXL encode, HDR export ⬜) |
-| M11 | RAW II | CR3, RAF (X-Trans), ORF, RW2, PEF, SRW, 3FR, IIQ + long tail; camera calibration DB; HEIC/AVIF/JXL import | 20–35 | 🚧 (RAF uncompressed, RW2 packed, PEF, ORF uncompressed ✅; **camera colour calibration** 🚧 (guarded ARW and NEF preview fitting; measured database still missing), CR3, compressed ORF/RAF ⬜) |
+| M11 | RAW II | CR3, RAF (X-Trans), ORF, RW2, PEF, SRW, 3FR, IIQ + long tail; camera calibration DB; HEIC/AVIF/JXL import | 20–35 | 🚧 (RAF uncompressed, RW2 (every raw format), PEF, ORF uncompressed ✅; **camera colour calibration** 🚧 (guarded ARW, NEF and RW2 preview fitting; measured database still missing), CR3, compressed ORF/RAF ⬜) |
| M12 | AI & smart features | subject/sky/background/people/object masks, semantic search, faces/People (permissively licensed models, pure-Rust inference) | 20–40 | ⬜ |
| M13 | Merge | HDR merge (deghost), panorama (projections, boundary warp, fill edges), HDR panorama | 10–15 | ✅ |
| M14 | Video | import/playback/trim via FilmCraft crates, global edits + presets on video, video export | 6–10 | ⬜ |
@@ -118,7 +118,7 @@ trip, slideshow, auto import…) in ≈ 12 h on 2026-10-02; four to six parallel
| Work package | Tracker rows | Agent-hours | Risk |
|---|---|---:|---|
| Remaining P0/P1 UI and library features (folder rename/move, keyword painter, people view…) | ≈ 8 | 5–10 | low |
-| Raw codecs: CR3 (CRX), compressed ORF / RAF, NEF lossy-after-split, RW2 v4, HEIC/AVIF decode, JPEG XL DNG | LR-IMP-FORMATS | 40–80 | **high** — clean-room black-box analysis, no permissive specs |
+| Raw codecs: CR3 (CRX), compressed ORF / RAF, NEF lossy-after-split, HEIC/AVIF decode, JPEG XL DNG | LR-IMP-FORMATS | 40–80 | **high** — clean-room black-box analysis, no permissive specs |
| Lens-profile database of our own (calibration targets, fitting, data) | LR-EDIT-OPTICS-PROFILE | 15–30 | data collection |
| Video: playback, trim, edits, export (pure-Rust decode, ideally shared with FilmCraft) | R. Video | 20–40 | medium |
| AI: subject / sky / background / people / object masks, object-aware remove, AI denoise, super resolution, lens blur, people & faces, natural-language search, culling | ≈ 30 | 80–150 | **high** — permissively licensed weights, pure-Rust inference, maybe training |
@@ -165,7 +165,9 @@ The milestone estimates in the table above were made before work started and are
Decoded (CC0 corpus from raw.pixls.us, `cargo xtask corpus --download`, `crates/raw/tests/corpus.rs`): DNG (uncompressed,
LJ92, lossy JPEG / Smart Previews, Deflate, float, linear), CR2, ARW (uncompressed, ARW2, LJ92; as-shot white balance and black level of pre-2017 bodies from the enciphered
maker-note `Tag2010` and the encrypted `SR2SubIFD`, both recovered by black-box analysis, `crates/raw/src/vendor/arw.rs`), NEF/NRW uncompressed and Huffman-compressed (lossless, lossy type 1/2, 12/14-bit), RAF uncompressed (Bayer and
-X-Trans), RW2 packed 12/14-bit, PEF (uncompressed and Huffman), ORF uncompressed (16-bit and 12-bit packed). Every
+X-Trans), RW2 / Leica RWL / Panasonic RAW in every raw format (compressed formats 4 and 6, the prefix-coded strips of format 8,
+packed formats 2, 5 and 7, the 16-bit words of the 2005–2007 bodies; all recovered by black-box analysis of 178 CC0 files from
+118 bodies, `crates/raw/src/vendor/rw2.rs`), PEF (uncompressed and Huffman), ORF uncompressed (16-bit and 12-bit packed). Every
supported container also yields its embedded JPEG preview (CR3 too), and the engine shows that preview for raw variants
it can't decode yet.
@@ -174,14 +176,11 @@ Not decoded yet — preview only (no permissively licensed description; black-bo
files): rows above the split decode with the regular lossy table; from the split row on a different code is used
that our black-box analysis has not recovered yet (none of the four regular tables fits, also not with byte
alignment or reset predictors at the split row). The other Nikon Huffman variants are decoded (`crates/raw/src/vendor/nefc.rs` documents the analysis).
-- **Panasonic RW2 raw format 4** (quantised): the block layout is known (0x4000-byte chunks rotated by 0x1ff8; 128-bit
- blocks of two 12-bit seeds plus four groups of a 2-bit scale and three 8-bit codes; scales 0/1 are ×1/×2 differences
- from the same-colour pixel two to the left), the reconstruction rule for scales 2/3 is not established.
- **Olympus compressed ORF**, **Fujifilm compressed RAF**, **Canon CR3/CRX** (M11.1), **Canon sRAW/mRAW**, lossy DNG.
-**Camera colour matrices:** ARW files can use guarded, separate chromaticity and tone estimates from their own embedded JPEG (see `docs/camera-preview-colour.md`); this is a per-file camera-look estimate with relative WB, not measured calibration or absolute-Kelvin WB. Other non-DNG raws and rejected fits use the documented neutral fallback (camera RGB ≈ linear sRGB, flagged
+**Camera colour matrices:** ARW, NEF and RW2 files can use guarded, separate chromaticity and tone estimates from their own embedded JPEG (see `docs/camera-preview-colour.md`); this is a per-file camera-look estimate with relative WB, not measured calibration or absolute-Kelvin WB. Other non-DNG raws and rejected fits use the documented neutral fallback (camera RGB ≈ linear sRGB, flagged
`matrix_is_fallback`) with the file's as-shot white-balance multipliers. Clean sources to evaluate next: manufacturer
-matrices stored in the files themselves (Olympus ImageProcessing `ColorMatrix`, Pentax/Panasonic equivalents) and our
+matrices stored in the files themselves (Olympus ImageProcessing `ColorMatrix`, Pentax equivalents; Panasonic RW2 files carry none) and our
own chart-based calibration (M11.4). Adobe matrices are never used.
## Log
@@ -193,6 +192,9 @@ own chart-based calibration (M11.4). Adobe matrices are never used.
Local roots and cleanup, 85k-photo grid and catalog performance, background compaction, failed-save errors, GPU
export hardening, clippy 1.99. Added the honest *Where we stand* assessment and *Where we're going* priorities; added
tracker rows for camera colour, camera coverage and render fidelity.
+- 2026-10-07: Panasonic RW2 / Leica RWL / Panasonic RAW decode in every raw format (compressed 4/6/8, packed 2/5/7,
+ 16-bit words), black levels, in-camera aspect crops and mapped-out defects established on 178 CC0 files from 118
+ bodies; RW2 gets the file-local camera look (accepted on 140 of 174 files with a preview).
## Chinese and Japanese interfaces, and text watermarks
diff --git a/apps/lightcraft-cli/tests/cli.rs b/apps/lightcraft-cli/tests/cli.rs
index 6e015af3..7cd0710b 100644
--- a/apps/lightcraft-cli/tests/cli.rs
+++ b/apps/lightcraft-cli/tests/cli.rs
@@ -292,7 +292,9 @@ fn devices_are_listed_and_imported_from() {
assert_eq!(dev["name"], "CARD", "{line}");
// the import review of its DCIM folder finds the photo
let dcim = dev["path"].as_str().unwrap().to_string();
- let o = Command::new(BIN).args(["run", "library.importPreview", &format!("paths={}", serde_json::json!([dcim]))]).output().unwrap();
+ // JSON-escaped: a Windows path has backslashes
+ let paths = serde_json::to_string(&[&dcim]).unwrap();
+ let o = Command::new(BIN).args(["run", "library.importPreview", &format!("paths={paths}")]).output().unwrap();
let line: Value = serde_json::from_slice(o.stdout.split(|b| *b == b'\n').next().unwrap()).unwrap();
assert_eq!(line["result"]["candidates"].as_array().map(Vec::len), Some(1), "{line}");
}
diff --git a/apps/lightcraft-web/src/web.rs b/apps/lightcraft-web/src/web.rs
index 6d0483a9..2efabebb 100644
--- a/apps/lightcraft-web/src/web.rs
+++ b/apps/lightcraft-web/src/web.rs
@@ -428,8 +428,9 @@ impl WebApp {
let cores = window().map_or(1, |w| w.navigator().hardware_concurrency() as usize);
cores.saturating_sub(1).clamp(1, 4)
});
+ let cache = app.session.media.rendered.clone();
let workers =
- (n > 0).then(|| Workers::start(n, backend.as_ref().map_or("memory", |b| b.kind()), backend.clone(), index, cc.egui_ctx.clone()));
+ (n > 0).then(|| Workers::start(n, backend.as_ref().map_or("memory", |b| b.kind()), backend.clone(), index, &cache, cc.egui_ctx.clone()));
if let Some(w) = &workers {
app.renderer.set_offload(Box::new(w.clone()));
}
diff --git a/apps/lightcraft-web/src/wire.rs b/apps/lightcraft-web/src/wire.rs
index 372e84c1..0474e2a0 100644
--- a/apps/lightcraft-web/src/wire.rs
+++ b/apps/lightcraft-web/src/wire.rs
@@ -11,13 +11,13 @@
//! ([`ThumbIndex`]).
use std::collections::HashMap;
-use std::sync::Arc;
+use std::sync::{Arc, Weak};
use lightcraft_engine::catalog::Source;
use lightcraft_engine::develop::{DevelopSettings, EmbeddedLens};
use lightcraft_engine::media::{DecodedSource, MediaCache, RenderJob, SourceLevel, SourceRef};
use lightcraft_engine::pipeline::{Quality, RenderRequest, Rendered, SourceInfo, StageCache};
-use lightcraft_preview::{Hash128, Lru};
+use lightcraft_preview::{Hash128, Lru, PreviewCache};
use serde::{Deserialize, Serialize};
use crate::store::hash_of_path;
@@ -299,6 +299,33 @@ impl ThumbIndex {
}
}
+/// Which rendered-thumbnail cache, at which generation, the stored thumbnails ([`ThumbIndex`])
+/// belong to (main thread). Clearing the previews bumps the cache's generation, and opening
+/// another library replaces the cache: either makes every stored thumbnail obsolete.
+pub struct CacheWatch(Option<(Weak, u64)>);
+
+impl CacheWatch {
+ /// Watch `active`, the session's cache when the workers start: the persisted index belongs
+ /// to it at its current generation. (Without this baseline, the first observation after a
+ /// clear would only record the cleared generation and keep the obsolete thumbnails.)
+ pub fn new(active: &Arc) -> CacheWatch {
+ CacheWatch(Some((Arc::downgrade(active), active.generation())))
+ }
+
+ /// The watched cache, while it's alive.
+ pub fn cache(&self) -> Option> {
+ self.0.as_ref().and_then(|(cache, _)| cache.upgrade())
+ }
+
+ /// `cache` at `generation` is the active one now. If it was cleared or replaced since the
+ /// last observation, invalidate `index` and return the keys whose files should be deleted.
+ pub fn observe(&mut self, cache: &Arc, generation: u64, index: &mut ThumbIndex) -> Vec {
+ let changed = self.0.as_ref().is_some_and(|(old, g)| old.as_ptr() != Arc::as_ptr(cache) || *g != generation);
+ self.0 = Some((Arc::downgrade(cache), generation));
+ if changed { index.invalidate() } else { Vec::new() }
+ }
+}
+
#[cfg(test)]
mod tests {
use super::*;
@@ -381,6 +408,62 @@ mod tests {
assert_eq!(core.needs_original(&refreshed).as_deref(), Some(hash.as_str()), "explicit refresh must not reuse the old decoded source");
}
+ /// The persisted index as a restarted page loads it: one stored thumbnail.
+ fn persisted_index(key: Hash128) -> (ThumbIndex, String) {
+ let mut index = ThumbIndex::default();
+ let hex = index.cache_key(key);
+ index.insert(&hex, 100);
+ (ThumbIndex::from_json(&index.to_json()), hex)
+ }
+
+ #[test]
+ fn clearing_previews_before_the_first_request_invalidates_stored_thumbnails() {
+ let key = lightcraft_preview::hash_bytes(b"thumbnail");
+ let (mut index, old) = persisted_index(key);
+ let cache = Arc::new(PreviewCache::memory(1 << 20));
+ let mut watch = CacheWatch::new(&cache);
+ let mut gone = Vec::new();
+ // each frame (`Workers::finished`) observes the watched cache, even with no requests
+ let mut frame = |watch: &mut CacheWatch, index: &mut ThumbIndex| {
+ if let Some(c) = watch.cache() {
+ gone.extend(watch.observe(&c, c.generation(), index));
+ }
+ };
+ frame(&mut watch, &mut index); // an empty filtered view: no render request yet
+ cache.clear(); // File ▸ Clear Preview Cache
+ frame(&mut watch, &mut index);
+ // the filter is removed: the first thumbnail request (`Workers::try_start`)
+ gone.extend(watch.observe(&cache, cache.generation(), &mut index));
+ let disk_key = index.cache_key(key);
+ assert!(!index.touch(&disk_key), "the pre-clear thumbnail must not be read back");
+ assert_eq!(gone, vec![old.clone()], "the stored file is deleted");
+ assert_ne!(disk_key, old, "a fresh namespace: a late old write can't come back");
+ assert!(index.dirty, "the invalidated index is saved");
+ }
+
+ #[test]
+ fn replacing_the_cache_before_the_first_request_invalidates_stored_thumbnails() {
+ let key = lightcraft_preview::hash_bytes(b"thumbnail");
+ let (mut index, old) = persisted_index(key);
+ let first = Arc::new(PreviewCache::memory(1 << 20));
+ let mut watch = CacheWatch::new(&first);
+ let other = Arc::new(PreviewCache::memory(1 << 20)); // another library, generation 0 again
+ assert_eq!(watch.observe(&other, other.generation(), &mut index), vec![old]);
+ }
+
+ #[test]
+ fn a_normal_start_keeps_stored_thumbnails() {
+ let key = lightcraft_preview::hash_bytes(b"thumbnail");
+ let (mut index, old) = persisted_index(key);
+ let cache = Arc::new(PreviewCache::memory(1 << 20));
+ let mut watch = CacheWatch::new(&cache);
+ for _ in 0..3 {
+ assert!(watch.observe(&cache, cache.generation(), &mut index).is_empty());
+ }
+ assert_eq!(index.cache_key(key), old);
+ assert!(index.touch(&old), "persisted thumbnails survive a restart");
+ }
+
#[test]
fn thumb_index_prunes_lru() {
let mut i = ThumbIndex::default();
diff --git a/apps/lightcraft-web/src/workers.rs b/apps/lightcraft-web/src/workers.rs
index 21e90f7d..a90ed61e 100644
--- a/apps/lightcraft-web/src/workers.rs
+++ b/apps/lightcraft-web/src/workers.rs
@@ -27,7 +27,7 @@ use wasm_bindgen::JsCast;
use wasm_bindgen::prelude::*;
use crate::backend::Backend;
-use crate::wire::{ThumbIndex, WireJob, WorkerCore, thumb_storage_key};
+use crate::wire::{CacheWatch, ThumbIndex, WireJob, WorkerCore, thumb_storage_key};
/// Storage key of the thumbnail index.
pub const THUMB_INDEX: &str = "thumbs/index.json";
@@ -175,7 +175,8 @@ struct W {
}
struct Inner {
- preview_generation: Option<(std::sync::Weak, u64)>,
+ /// The rendered-thumbnail cache the stored thumbnails belong to.
+ preview_generation: CacheWatch,
workers: Vec,
done: Vec<(Slot, RenderResult, f64)>,
next_id: u32,
@@ -192,17 +193,16 @@ struct Inner {
impl Inner {
fn observe_cache(&mut self, cache: &Arc, generation: u64) {
- if self.preview_generation.as_ref().is_some_and(|(old, g)| old.as_ptr() != Arc::as_ptr(cache) || *g != generation) {
- let gone = self.index.invalidate();
- if let Some(backend) = self.backend.clone() {
- wasm_bindgen_futures::spawn_local(async move {
- for key in gone {
- let _ = backend.remove(&thumb_storage_key(&key)).await;
- }
- });
- }
+ let gone = self.preview_generation.observe(cache, generation, &mut self.index);
+ if !gone.is_empty()
+ && let Some(backend) = self.backend.clone()
+ {
+ wasm_bindgen_futures::spawn_local(async move {
+ for key in gone {
+ let _ = backend.remove(&thumb_storage_key(&key)).await;
+ }
+ });
}
- self.preview_generation = Some((Arc::downgrade(cache), generation));
}
}
@@ -212,9 +212,11 @@ pub struct Workers(Rc>);
impl Workers {
/// Start `n` workers (none: every job runs inline). `store` is the backend kind for workers.
- pub fn start(n: usize, store: &str, backend: Option, index: ThumbIndex, ctx: egui::Context) -> Workers {
+ /// `index` lists the stored thumbnails of `cache`, the session's rendered-thumbnail cache
+ /// now: watching it from the start, a clear before the first render request still counts.
+ pub fn start(n: usize, store: &str, backend: Option, index: ThumbIndex, cache: &Arc, ctx: egui::Context) -> Workers {
let w = Workers(Rc::new(RefCell::new(Inner {
- preview_generation: None,
+ preview_generation: CacheWatch::new(cache),
workers: Vec::new(),
done: Vec::new(),
next_id: 0,
@@ -471,7 +473,7 @@ impl RenderOffload for Workers {
fn finished(&mut self) -> Vec<(Slot, RenderResult, f64)> {
let mut g = self.0.borrow_mut();
// Persist invalidation even when the cleared library has no visible render requests.
- if let Some(cache) = g.preview_generation.as_ref().and_then(|(cache, _)| cache.upgrade()) {
+ if let Some(cache) = g.preview_generation.cache() {
g.observe_cache(&cache, cache.generation());
}
g.inline_this_frame = false;
diff --git a/apps/lightcraft/src/main.rs b/apps/lightcraft/src/main.rs
index e46fd2ee..4bb1c6ff 100644
--- a/apps/lightcraft/src/main.rs
+++ b/apps/lightcraft/src/main.rs
@@ -286,8 +286,8 @@ fn services() -> Services {
.add_filter(
lightcraft_ui_egui::i18n::tr("Photos"),
&[
- "jpg", "jpeg", "png", "tif", "tiff", "webp", "dng", "cr2", "cr3", "nef", "nrw", "arw", "raf", "orf", "rw2", "pef", "psd",
- "jxl", "gif", "bmp",
+ "jpg", "jpeg", "png", "tif", "tiff", "webp", "dng", "cr2", "cr3", "nef", "nrw", "arw", "raf", "orf", "rw2", "rwl", "raw",
+ "pef", "psd", "jxl", "gif", "bmp",
],
)
.pick_files()
diff --git a/crates/catalog/src/model.rs b/crates/catalog/src/model.rs
index 9248b220..fe8d1b05 100644
--- a/crates/catalog/src/model.rs
+++ b/crates/catalog/src/model.rs
@@ -336,11 +336,12 @@ impl Photo {
pub fn develops_raw(&self) -> bool {
self.kind == MediaKind::Raw && self.preview_only.is_none()
}
- /// The current ARW and NEF readers have vendor WB multipliers but no measured camera
+ /// The current ARW, NEF and RW2 readers have vendor WB multipliers but no measured camera
/// illuminant. Use adjustments relative to the camera's as-shot look, as for rendered
- /// photographs (the engine's `camera_preview::file_local_look` covers the same formats).
+ /// photographs (the engine's `camera_preview::file_local_look` covers the same formats; RWL and
+ /// RAW are Leica's and the oldest Panasonic bodies' names for RW2 files).
pub fn relative_wb(&self) -> bool {
- self.develops_raw() && ["ARW", "NEF", "NRW"].iter().any(|f| self.format.eq_ignore_ascii_case(f))
+ self.develops_raw() && ["ARW", "NEF", "NRW", "RW2", "RWL", "RAW"].iter().any(|f| self.format.eq_ignore_ascii_case(f))
}
/// The develop settings import gave this photo: [`Photo::camera_defaults`], or the user's
/// default preset applied on top of them ([`Photo::import_look`]).
@@ -430,7 +431,15 @@ mod edited_tests {
#[test]
fn sony_and_nikon_raws_use_relative_white_balance() {
- for (name, format, relative) in [("a.arw", "ARW", true), ("a.nef", "NEF", true), ("a.nrw", "nrw", true), ("a.dng", "DNG", false)] {
+ for (name, format, relative) in [
+ ("a.arw", "ARW", true),
+ ("a.nef", "NEF", true),
+ ("a.nrw", "nrw", true),
+ ("a.rw2", "RW2", true),
+ ("a.rwl", "RWL", true),
+ ("a.raw", "RAW", true),
+ ("a.dng", "DNG", false),
+ ] {
let mut p = Photo::new(PhotoId(1), Source::Demo { scene: 0 }, name, format, 10, 10, "2026-10-01T00:00:00");
p.kind = MediaKind::Raw;
p.as_shot_wb = Some((5200.0, 4.0));
diff --git a/crates/engine/src/camera_preview.rs b/crates/engine/src/camera_preview.rs
index f0120641..1a3e9a9b 100644
--- a/crates/engine/src/camera_preview.rs
+++ b/crates/engine/src/camera_preview.rs
@@ -1,5 +1,5 @@
-//! Estimate the starting look of a raw without a camera colour matrix (Sony ARW, Nikon NEF) from its
-//! own JPEG. Colour and luminance are fitted separately; the JPEG supplies correspondences only,
+//! Estimate the starting look of a raw without a camera colour matrix (Sony ARW, Nikon NEF, Panasonic
+//! RW2) from its own JPEG. Colour and luminance are fitted separately; the JPEG supplies correspondences only,
//! never output pixels or a replacement for RAW editing.
//! A global matrix can't follow the camera's hue-dependent rendering (the best matrix rendered a
//! lime shirt olive that the camera kept lime): a hue/saturation table fitted to the residuals
@@ -31,7 +31,7 @@ pub(crate) const PROFILE_PROXY: usize = 192;
/// as-shot look (`docs/camera-preview-colour.md`). The catalog's `Photo::relative_wb` matches the
/// same formats by file extension.
pub(crate) fn file_local_look(format: RawFormat) -> bool {
- matches!(format, RawFormat::Arw | RawFormat::Nef | RawFormat::Nrw)
+ matches!(format, RawFormat::Arw | RawFormat::Nef | RawFormat::Nrw | RawFormat::Rw2)
}
pub(crate) fn fit_preview(raw: &RawImage, bytes: &[u8], transform: &CameraTransform) -> Option {
@@ -63,14 +63,24 @@ fn proxies(raw: &RawImage, bytes: &[u8], transform: &CameraTransform, size: usiz
let jpeg = lightcraft_raw::embedded_preview(bytes)?;
let edge = (2 * size).max(384) as u32;
let decoded = lightcraft_codecs::decode(&jpeg, lightcraft_codecs::DecodeOptions { max_size: Some((edge, edge)), max_pixels: 64_000_000 }).ok()?;
- let reference = decoded.to_working();
- let crop = raw.crop.clipped(raw.active_area.width, raw.active_area.height);
+ let mut reference = decoded.to_working();
+ let (a, crop) = (raw.active_area, raw.crop.clipped(raw.active_area.width, raw.active_area.height));
if crop.width == 0 || crop.height == 0 || reference.width == 0 || reference.height == 0 {
return None;
}
- let aspect = crop.width as f64 / crop.height as f64;
- if (reference.width as f64 / reference.height as f64 / aspect - 1.0).abs() > 0.02 {
- return None;
+ let matches = |w: usize, h: usize, rw: usize, rh: usize| (rw as f64 / rh as f64 / (w as f64 / h as f64) - 1.0).abs() <= 0.02;
+ if !matches(crop.width, crop.height, reference.width, reference.height) {
+ // Panasonic previews show the whole active area while the default crop is the in-camera aspect ratio
+ if !matches(a.width, a.height, reference.width, reference.height) {
+ return None;
+ }
+ let (sx, sy) = (reference.width as f64 / a.width as f64, reference.height as f64 / a.height as f64);
+ let (x, y) = ((crop.x as f64 * sx).round() as usize, (crop.y as f64 * sy).round() as usize);
+ let (w, h) = ((crop.width as f64 * sx).round() as usize, (crop.height as f64 * sy).round() as usize);
+ if w < 16 || h < 16 || x + w > reference.width || y + h > reference.height {
+ return None;
+ }
+ reference = reference.into_crop(x, y, w, h);
}
// Fixed, bounded proxy: the selected look cannot depend on thumbnail/export resolution.
let k = (crop.width.max(crop.height).div_ceil(edge as usize).max(2)).div_ceil(2) * 2;
@@ -731,8 +741,8 @@ mod tests {
}
#[test]
- fn sony_and_nikon_raws_get_a_file_local_look() {
- assert!([RawFormat::Arw, RawFormat::Nef, RawFormat::Nrw].into_iter().all(file_local_look));
+ fn sony_nikon_and_panasonic_raws_get_a_file_local_look() {
+ assert!([RawFormat::Arw, RawFormat::Nef, RawFormat::Nrw, RawFormat::Rw2].into_iter().all(file_local_look));
assert!(![RawFormat::Dng, RawFormat::Cr2, RawFormat::Raf].into_iter().any(file_local_look));
}
@@ -753,6 +763,24 @@ mod tests {
assert!(info.relative_wb && info.as_shot_temp == 6500.0 && info.as_shot_tint == 0.0);
}
+ /// A public DC-FZ1000 II RW2 shot at 4:3 on its 3:2 sensor (skipped without the corpus): the default crop is 4:3
+ /// while the embedded JPEG shows the whole sensor; the look is still fitted, against the matching part of it.
+ #[test]
+ fn corpus_rw2_with_an_in_camera_crop_gets_a_camera_look() {
+ let path = std::env::var_os("LIGHTCRAFT_CORPUS")
+ .map(std::path::PathBuf::from)
+ .unwrap_or_else(|| std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("../../corpus"))
+ .join("raw/rw2-panasonic-fz1000m2-4x3.rw2");
+ let Ok(bytes) = std::fs::read(&path) else {
+ eprintln!("skip: {} absent", path.display());
+ return;
+ };
+ let (img, info) = crate::files::load_bytes(&bytes, 400).unwrap();
+ assert_eq!((img.width, img.height), (400, 300), "framed in the in-camera aspect ratio");
+ assert!(info.camera_tone.is_some(), "no camera look fitted");
+ assert!(info.relative_wb && info.as_shot_temp == 6500.0 && info.as_shot_tint == 0.0);
+ }
+
#[test]
fn rejects_monochrome_invalid_and_unrelated_previews() {
let mut sensor = Rgb32f::new(32, 32);
diff --git a/crates/engine/src/import.rs b/crates/engine/src/import.rs
index df0e96e8..4895eee7 100644
--- a/crates/engine/src/import.rs
+++ b/crates/engine/src/import.rs
@@ -32,8 +32,8 @@ use crate::media::ProbeInfo;
/// File extensions LightCraft imports (lower case).
pub const EXTENSIONS: &[&str] = &[
- "jpg", "jpeg", "png", "tif", "tiff", "webp", "dng", "cr2", "cr3", "nef", "nrw", "arw", "raf", "orf", "rw2", "pef", "psd", "jxl", "gif", "bmp",
- "heic", "avif",
+ "jpg", "jpeg", "png", "tif", "tiff", "webp", "dng", "cr2", "cr3", "nef", "nrw", "arw", "raf", "orf", "rw2", "rwl", "raw", "pef", "psd", "jxl",
+ "gif", "bmp", "heic", "avif",
];
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
diff --git a/crates/engine/src/media.rs b/crates/engine/src/media.rs
index e45cc92e..76a9bf1a 100644
--- a/crates/engine/src/media.rs
+++ b/crates/engine/src/media.rs
@@ -244,8 +244,11 @@ fn source_bytes(img: &Rgb32f) -> usize {
}
impl MediaCache {
- /// Keep rendered thumbnails on disk in `dir` as well.
+ /// Keep rendered thumbnails on disk in `dir` as well. The cache this replaces is retired
+ /// (its files stay): render jobs still holding it can't write through it any more, even
+ /// into a directory a later clear emptied (`dir` may be the same one).
pub fn attach_disk_cache(&mut self, dir: &std::path::Path, disk_bytes: u64) {
+ self.rendered.retire();
self.rendered = Arc::new(PreviewCache::with_disk(rendered_budget(self.budget), dir, disk_bytes));
}
diff --git a/crates/engine/src/rename.rs b/crates/engine/src/rename.rs
index 0479a4e5..259d51fc 100644
--- a/crates/engine/src/rename.rs
+++ b/crates/engine/src/rename.rs
@@ -774,12 +774,13 @@ mod tests {
fail: RefCell::new(Vec::new()),
}
}
+ /// The model volume spells paths with `/`; the code under test joins them with the host's separator.
+ fn norm(p: &Path) -> String {
+ p.to_string_lossy().replace('\\', "/")
+ }
fn idx(&self, p: &Path) -> Option {
- let p = p.to_string_lossy();
- self.files
- .borrow()
- .iter()
- .position(|(f, _)| if self.ci { folded_path(f) == folded_path(&p) } else { Path::new(f) == Path::new(p.as_ref()) })
+ let p = Self::norm(p);
+ self.files.borrow().iter().position(|(f, _)| if self.ci { folded_path(f) == folded_path(&p) } else { *f == p })
}
/// The listing: (exact path, contents), sorted.
pub fn listing(&self) -> Vec<(String, String)> {
@@ -800,14 +801,14 @@ mod tests {
self.idx(a).is_some() && self.idx(a) == self.idx(b)
}
fn rename_no_replace(&self, a: &Path, b: &Path) -> std::io::Result<()> {
- if self.fail.borrow().iter().any(|f| a.to_string_lossy().contains(f.as_str())) {
+ if self.fail.borrow().iter().any(|f| Self::norm(a).contains(f.as_str())) {
return Err(std::io::Error::other("injected failure"));
}
if self.exists(b) {
return Err(std::io::Error::new(std::io::ErrorKind::AlreadyExists, "exists"));
}
let i = self.idx(a).ok_or_else(|| std::io::Error::new(std::io::ErrorKind::NotFound, "no such file"))?;
- self.files.borrow_mut()[i].0 = b.to_string_lossy().to_string();
+ self.files.borrow_mut()[i].0 = Self::norm(b);
Ok(())
}
fn copy_no_replace(&self, a: &Path, b: &Path) -> std::io::Result<()> {
@@ -816,7 +817,7 @@ mod tests {
}
let i = self.idx(a).ok_or_else(|| std::io::Error::new(std::io::ErrorKind::NotFound, "no such file"))?;
let c = self.files.borrow()[i].1.clone();
- self.files.borrow_mut().push((b.to_string_lossy().to_string(), c));
+ self.files.borrow_mut().push((Self::norm(b), c));
Ok(())
}
fn remove_file(&self, p: &Path) -> std::io::Result<()> {
@@ -825,17 +826,19 @@ mod tests {
Ok(())
}
fn stem_sibling(&self, sidecar: &Path) -> Option {
- let stem = sidecar.with_extension("").to_string_lossy().to_string();
+ let stem = Self::norm(&sidecar.with_extension(""));
self.files.borrow().iter().map(|(f, _)| f.clone()).find(|f| {
let p = Path::new(f);
- p.with_extension("").to_string_lossy() == stem && !p.extension().is_some_and(|e| e.eq_ignore_ascii_case("xmp"))
+ Self::norm(&p.with_extension("")) == stem && !p.extension().is_some_and(|e| e.eq_ignore_ascii_case("xmp"))
})
}
}
fn file_photo(id: u64, path: &str) -> Photo {
let name = Path::new(path).file_name().map(|n| n.to_string_lossy().to_string()).unwrap_or_default();
- Photo::new(PhotoId(id), Source::File { path: path.into() }, &name, "JPG", 1, 1, "2026-01-01T00:00:00")
+ // the host's separator, so `dir.join(name)` in the planner spells the same path the same way
+ let path = path.replace('/', std::path::MAIN_SEPARATOR_STR);
+ Photo::new(PhotoId(id), Source::File { path }, &name, "JPG", 1, 1, "2026-01-01T00:00:00")
}
#[cfg(windows)]
@@ -845,8 +848,11 @@ mod tests {
// Any attempt to move the unchanged photo (including rollback) must fail.
fs.fail.borrow_mut().push("IMG".into());
let mut s = Session::new();
+ // spelled with `/` on purpose (not `file_photo`, which uses the host's separator)
for (id, path) in [(1, "N:/photos/IMG.jpg"), (2, "N:/photos/other.jpg")] {
- s.catalog.apply(Op::AddPhoto { photo: Box::new(file_photo(id, path)) }).unwrap();
+ let name = Path::new(path).file_name().map(|n| n.to_string_lossy().to_string()).unwrap_or_default();
+ let photo = Photo::new(PhotoId(id), Source::File { path: path.into() }, &name, "JPG", 1, 1, "2026-01-01T00:00:00");
+ s.catalog.apply(Op::AddPhoto { photo: Box::new(photo) }).unwrap();
}
let before = fs.listing();
let plans = s.plan_rename_with(&fs, &[PhotoId(1)], "{name}", 1);
diff --git a/crates/engine/src/segment/mod.rs b/crates/engine/src/segment/mod.rs
index 84397042..3cf14712 100644
--- a/crates/engine/src/segment/mod.rs
+++ b/crates/engine/src/segment/mod.rs
@@ -260,7 +260,7 @@ impl Segmenter {
{
let mirrors = self.mirrors();
if mirrors.is_empty() {
- return Err(lightcraft_segment::fetch::DownloadError::NoMirrors.to_string());
+ return Err(lightcraft_segment::fetch::no_mirrors_message());
}
self.download.start(lightcraft_segment::fetch::SAM3_FILES, mirrors, dir, lightcraft_segment::fetch::Options::default())
}
diff --git a/crates/engine/src/tests_libops.rs b/crates/engine/src/tests_libops.rs
index ac0e5c57..f9e0f492 100644
--- a/crates/engine/src/tests_libops.rs
+++ b/crates/engine/src/tests_libops.rs
@@ -273,7 +273,7 @@ fn import_review_preview_and_options() {
let mut s = Session::new().with_fs();
s.open_library(&lib, false).unwrap();
// c.png is already in the library
- s.execute("library.import", &json!({"paths": [src.join("sub/c.png").to_string_lossy()]})).unwrap();
+ s.execute("library.import", &json!({"paths": [src.join("sub").join("c.png").to_string_lossy()]})).unwrap();
let n0 = s.catalog.len();
let undo0 = s.undo.len();
let pv = s.execute("library.importPreview", &json!({"paths": [src.to_string_lossy()]})).unwrap();
diff --git a/crates/engine/src/tests_prefs.rs b/crates/engine/src/tests_prefs.rs
index 9b343bfd..3534064f 100644
--- a/crates/engine/src/tests_prefs.rs
+++ b/crates/engine/src/tests_prefs.rs
@@ -288,3 +288,32 @@ fn resizing_the_disk_cache_keeps_the_cache_and_its_thumbnails() {
drop(s);
let _ = std::fs::remove_dir_all(&lib);
}
+
+#[test]
+fn reopening_a_library_retires_the_old_cache_so_its_jobs_cannot_write_after_a_clear() {
+ let lib = temp_dir("reopen-writer");
+ let mut s = Session::new().with_fs();
+ s.open_library(&lib, true).unwrap();
+ let id = s.catalog.photos().next().unwrap().id;
+ // a thumbnail job still holding the first open's cache object
+ let job = s.thumb_job(id, 128).unwrap();
+ let (old, key) = job.cache.clone().unwrap();
+ // a thumbnail written before the reopen is still valid afterwards
+ let kept = lightcraft_preview::hash_bytes(b"kept thumbnail");
+ let img = lightcraft_raster::Rgba8 { width: 4, height: 4, data: vec![[10, 20, 30, 255]; 16] };
+ old.put(kept, Arc::new(img.clone()));
+ s.open_library(&lib, false).unwrap();
+ assert!(!Arc::ptr_eq(&old, &s.media.rendered));
+ assert!(s.media.rendered.get(kept).is_some(), "a plain reopen keeps the disk cache");
+ s.execute("library.clearPreviews", &json!({})).unwrap();
+ // the old job finishes: its render must not land in the cleared directory
+ let r = job.run();
+ assert!(r.rendered.is_ok());
+ old.put(kept, Arc::new(img));
+ let fresh = lightcraft_preview::PreviewCache::with_disk(1 << 20, &lib.join("thumbs"), 1 << 30);
+ assert!(fresh.get(key).is_none(), "stale thumbnail written through the replaced cache");
+ assert!(fresh.get(kept).is_none(), "stale put through the replaced cache");
+ assert!(s.media.rendered.get(key).is_none());
+ drop(s);
+ let _ = std::fs::remove_dir_all(&lib);
+}
diff --git a/crates/fetch/Cargo.toml b/crates/fetch/Cargo.toml
new file mode 100644
index 00000000..2e65583f
--- /dev/null
+++ b/crates/fetch/Cargo.toml
@@ -0,0 +1,17 @@
+[package]
+name = "lightcraft-fetch"
+version.workspace = true
+edition.workspace = true
+license = "Apache-2.0"
+description = "LightCraft model downloads: mirrors, resume, pinned SHA-256, over HTTPS in pure Rust (L0)"
+
+[lints]
+workspace = true
+
+# downloads run natively only (the web build fetches no models)
+[target.'cfg(not(target_arch = "wasm32"))'.dependencies]
+log = { workspace = true }
+rustls = { version = "0.23", default-features = false, features = ["std", "tls12"] }
+rustls-rustcrypto = "0.0.2-alpha"
+webpki-roots = "1"
+sha2 = { version = "0.10", default-features = false, features = ["std"] }
diff --git a/crates/segment/src/fetch/http.rs b/crates/fetch/src/http.rs
similarity index 100%
rename from crates/segment/src/fetch/http.rs
rename to crates/fetch/src/http.rs
diff --git a/crates/fetch/src/lib.rs b/crates/fetch/src/lib.rs
new file mode 100644
index 00000000..499a6eda
--- /dev/null
+++ b/crates/fetch/src/lib.rs
@@ -0,0 +1,394 @@
+//! Downloading a model's files, only when the user asked for it: from an ordered list of mirrors
+//! (the next one when a mirror fails), resuming partial files, with connect and stall timeouts,
+//! a size cap, progress and cancellation, and checked (exact size, SHA-256 where pinned) before
+//! each file is moved into the model folder. Runs on whatever thread calls [`download`] (the
+//! engine uses a background thread; the UI thread never waits for the network). Pure Rust: the
+//! HTTPS client is rustls with the RustCrypto provider (see [`http`]).
+//!
+//! Files are written to `.part` next to their final place and renamed when verified, so a
+//! model folder never holds a half-written file under a real name, and an interrupted download
+//! resumes from where it stopped. A file that fails its check is deleted, never kept.
+//!
+//! What to download and where from belongs to the caller: it passes the [`FileSpec`]s and the
+//! default mirrors (see [`mirrors`]). `lightcraft-segment` does so for the SAM 3 weights, which
+//! are never part of LightCraft (SAM License, see docs/ai-masks.md).
+//!
+//! Native only: on wasm32 this crate is empty (the web build downloads no models).
+
+#![cfg(not(target_arch = "wasm32"))]
+#![forbid(unsafe_code)]
+#![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable)]
+
+pub mod http;
+
+use std::fs::{File, OpenOptions};
+use std::io::{Read, Write};
+use std::path::{Path, PathBuf};
+use std::sync::atomic::{AtomicBool, Ordering};
+use std::time::Duration;
+
+use sha2::{Digest, Sha256};
+
+use http::{HttpError, Limits, Url};
+
+/// One file of the model.
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub struct FileSpec {
+ pub name: &'static str,
+ /// Exact size in bytes, when pinned (then the download resumes and is checked against it).
+ pub size: Option,
+ /// SHA-256 (lower-case hex), when pinned.
+ pub sha256: Option<&'static str>,
+ /// Largest size accepted when `size` is not pinned.
+ pub max: u64,
+}
+
+/// Most mirrors used (a hostile list can't make a download loop forever).
+const MAX_MIRRORS: usize = 16;
+/// Redirects followed per request.
+const MAX_REDIRECTS: usize = 8;
+
+/// The mirrors to try, in order: `env`'s (base URLs separated by commas, spaces or newlines), then
+/// the mirrors file's (one base URL per line, `#` comments), then `defaults`; duplicates and
+/// unusable URLs dropped, at most 16.
+pub fn mirrors(env: Option<&str>, file: Option<&Path>, defaults: &[&str]) -> Vec {
+ let mut out: Vec = Vec::new();
+ let mut add = |s: &str| {
+ let s = s.trim().trim_end_matches('/');
+ if !s.is_empty() && Url::parse(s).is_ok() && !out.iter().any(|m| m == s) && out.len() < MAX_MIRRORS {
+ out.push(s.to_string());
+ }
+ };
+ if let Some(env) = env {
+ env.split([',', ' ', '\n', '\t']).for_each(&mut add);
+ }
+ // a small text file; anything larger isn't a mirror list
+ if let Some(text) = file.and_then(|f| std::fs::metadata(f).ok().filter(|m| m.len() < 64 * 1024).and_then(|_| std::fs::read_to_string(f).ok())) {
+ text.lines().map(|l| l.split('#').next().unwrap_or_default()).for_each(&mut add);
+ }
+ defaults.iter().for_each(|m| add(m));
+ out
+}
+
+/// Download settings.
+#[derive(Clone, Debug)]
+pub struct Options {
+ /// Longest wait to connect to a mirror.
+ pub connect_timeout: Duration,
+ /// Longest wait for the next bytes once connected.
+ pub stall_timeout: Duration,
+ /// Tries per mirror and file (each resumes where the last one stopped).
+ pub attempts: u32,
+}
+
+impl Default for Options {
+ fn default() -> Self {
+ Options { connect_timeout: Duration::from_secs(15), stall_timeout: Duration::from_secs(30), attempts: 3 }
+ }
+}
+
+/// Where a download is.
+#[derive(Clone, Debug, Default, PartialEq, Eq)]
+pub struct Progress {
+ pub file: String,
+ /// Bytes of all files on disk so far, and the total expected (pinned sizes + announced ones).
+ pub done: u64,
+ pub total: u64,
+ /// The mirror in use (without query).
+ pub mirror: String,
+}
+
+/// Why a download failed.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum DownloadError {
+ /// No mirror is configured.
+ NoMirrors,
+ Cancelled,
+ /// A local file problem (folder not writable, disk full…).
+ Disk(String),
+ /// Every mirror failed for `file`; the reason for each.
+ AllMirrorsFailed {
+ file: String,
+ errors: Vec,
+ },
+}
+
+impl std::fmt::Display for DownloadError {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ match self {
+ DownloadError::NoMirrors => write!(f, "no download location is configured for this model in this build"),
+ DownloadError::Cancelled => write!(f, "the download was cancelled"),
+ DownloadError::Disk(e) => write!(f, "{e}"),
+ DownloadError::AllMirrorsFailed { file, errors } => write!(f, "could not download {file}: {}", errors.join("; ")),
+ }
+ }
+}
+
+/// One mirror's failure for one file; `Retry` resumes on the same mirror.
+enum Fail {
+ Retry(String),
+ NextMirror(String),
+ Cancelled,
+ Disk(String),
+}
+
+/// Download `files` into `dir` from `mirrors` (in order; see [`mirrors`]). Files already there
+/// and verified are kept. `progress` is called as bytes arrive (often: throttle in it).
+pub fn download(
+ files: &[FileSpec],
+ mirrors: &[String],
+ dir: &Path,
+ opts: &Options,
+ cancel: &AtomicBool,
+ progress: &mut dyn FnMut(&Progress),
+) -> Result<(), DownloadError> {
+ std::fs::create_dir_all(dir).map_err(|e| DownloadError::Disk(format!("can't create {}: {e}", dir.display())))?;
+ // what's left to fetch
+ let mut todo = Vec::new();
+ for f in files {
+ if !verified(f, &dir.join(f.name))? {
+ todo.push(f);
+ }
+ }
+ if todo.is_empty() {
+ return Ok(());
+ }
+ if mirrors.is_empty() {
+ return Err(DownloadError::NoMirrors);
+ }
+ let total: u64 = files.iter().map(|f| f.size.unwrap_or(0)).sum();
+ let mut before: u64 = files.iter().filter(|f| !todo.contains(f)).filter_map(|f| f.size).sum();
+ for f in todo {
+ let mut errors = Vec::new();
+ let mut ok = false;
+ 'mirrors: for m in mirrors {
+ let url = format!("{m}/{}", f.name);
+ for _ in 0..opts.attempts.max(1) {
+ if cancel.load(Ordering::Relaxed) {
+ return Err(DownloadError::Cancelled);
+ }
+ let mut report = |n: u64, announced: u64| {
+ let mirror = Url::parse(m).map(|u| u.display()).unwrap_or_default();
+ progress(&Progress {
+ file: f.name.to_string(),
+ done: before.saturating_add(n),
+ total: total.max(before.saturating_add(announced)),
+ mirror,
+ })
+ };
+ match fetch_file(f, &url, dir, opts, cancel, &mut report) {
+ Ok(()) => {
+ ok = true;
+ break 'mirrors;
+ }
+ Err(Fail::Cancelled) => return Err(DownloadError::Cancelled),
+ Err(Fail::Disk(e)) => return Err(DownloadError::Disk(e)),
+ Err(Fail::Retry(e)) => {
+ log::warn!("download of {url}: {e} (retrying)");
+ errors.push(format!("{}: {e}", short(m)));
+ }
+ Err(Fail::NextMirror(e)) => {
+ log::warn!("download of {url}: {e}");
+ errors.push(format!("{}: {e}", short(m)));
+ continue 'mirrors;
+ }
+ }
+ }
+ }
+ if !ok {
+ return Err(DownloadError::AllMirrorsFailed { file: f.name.to_string(), errors });
+ }
+ before = before.saturating_add(std::fs::metadata(dir.join(f.name)).map(|m| m.len()).unwrap_or(0));
+ }
+ Ok(())
+}
+
+/// A mirror for messages (no query string: it may hold a token).
+fn short(m: &str) -> String {
+ Url::parse(m).map(|u| u.display()).unwrap_or_else(|_| "mirror".into())
+}
+
+/// Whether `path` is already the right file (exact size and hash when pinned).
+fn verified(f: &FileSpec, path: &Path) -> Result {
+ let Ok(meta) = std::fs::metadata(path) else { return Ok(false) };
+ if !meta.is_file() || meta.len() == 0 {
+ return Ok(false);
+ }
+ if f.size.is_some_and(|s| s != meta.len()) || meta.len() > f.max {
+ return Ok(false);
+ }
+ match f.sha256 {
+ Some(want) => Ok(sha256_file(path).map_err(DownloadError::Disk)? == want),
+ None => Ok(true),
+ }
+}
+
+fn sha256_file(path: &Path) -> Result {
+ let mut file = File::open(path).map_err(|e| format!("{}: {e}", path.display()))?;
+ let mut h = Sha256::new();
+ let mut buf = vec![0u8; 1 << 20];
+ loop {
+ let n = file.read(&mut buf).map_err(|e| format!("{}: {e}", path.display()))?;
+ if n == 0 {
+ break;
+ }
+ h.update(buf.get(..n).unwrap_or_default());
+ }
+ Ok(h.finalize().iter().map(|b| format!("{b:02x}")).collect())
+}
+
+fn part_path(dir: &Path, name: &str) -> PathBuf {
+ dir.join(format!("{name}.part"))
+}
+
+/// Fetch one file from one mirror into `.part` (resuming it when its size is pinned),
+/// check it and move it into place.
+fn fetch_file(f: &FileSpec, url: &str, dir: &Path, opts: &Options, cancel: &AtomicBool, report: &mut dyn FnMut(u64, u64)) -> Result<(), Fail> {
+ let part = part_path(dir, f.name);
+ let limit = f.size.unwrap_or(f.max);
+ let disk = |e: std::io::Error| Fail::Disk(format!("{}: {e}", part.display()));
+ // resume only a pinned file (an unpinned one can't be checked after resuming)
+ let mut have = match std::fs::metadata(&part) {
+ Ok(m) if f.size.is_some() && m.len() <= limit => m.len(),
+ Ok(_) => {
+ std::fs::remove_file(&part).map_err(disk)?;
+ 0
+ }
+ Err(_) => 0,
+ };
+ let fetched = f.size != Some(have) || have == 0;
+ if fetched {
+ let limits = Limits { connect: opts.connect_timeout, stall: opts.stall_timeout, cancel };
+ let mut url = Url::parse(url).map_err(|e| Fail::NextMirror(e.to_string()))?;
+ let origin = url.host.clone();
+ let mut redirects = 0;
+ let mut resp = loop {
+ let mut headers = vec![];
+ if have > 0 {
+ headers.push(("Range", format!("bytes={have}-")));
+ }
+ // a token for the configured host only (never sent on to a redirect's host)
+ if let Ok(t) = std::env::var("LIGHTCRAFT_SAM3_TOKEN")
+ && !t.trim().is_empty()
+ && url.tls
+ && url.host == origin
+ {
+ headers.push(("Authorization", format!("Bearer {}", t.trim())));
+ }
+ let r = http::get(&url, &headers, &limits).map_err(http_fail)?;
+ match r.status {
+ 301 | 302 | 303 | 307 | 308 => {
+ redirects += 1;
+ if redirects > MAX_REDIRECTS {
+ return Err(Fail::NextMirror("too many redirects".into()));
+ }
+ let loc = r.header("location").ok_or_else(|| Fail::NextMirror("redirect without a location".into()))?;
+ url = url.join(loc).map_err(|e| Fail::NextMirror(e.to_string()))?;
+ }
+ _ => break r,
+ }
+ };
+ let mut file = match resp.status {
+ 200 => {
+ // the whole file (from the start, also when a range was asked for)
+ if resp.content_length().is_some_and(|n| n > limit || f.size.is_some_and(|s| s != n)) {
+ return Err(Fail::NextMirror(format!("the server's file has the wrong size ({:?} bytes)", resp.content_length())));
+ }
+ have = 0;
+ File::create(&part).map_err(disk)?
+ }
+ 206 if have > 0 => {
+ let range = resp.header("content-range").unwrap_or_default();
+ let (start, total) = parse_content_range(range).ok_or_else(|| Fail::NextMirror(format!("bad Content-Range `{range}`")))?;
+ if start != have || total.is_some_and(|t| Some(t) != f.size) {
+ return Err(Fail::NextMirror(format!("the server resumed at the wrong place ({range})")));
+ }
+ OpenOptions::new().append(true).open(&part).map_err(disk)?
+ }
+ 416 => {
+ // the partial file doesn't fit what the server has: start over
+ std::fs::remove_file(&part).map_err(disk)?;
+ return Err(Fail::Retry("the partial download didn't match; starting over".into()));
+ }
+ 401 | 403 => return Err(Fail::NextMirror(format!("access denied (HTTP {})", resp.status))),
+ 404 | 410 => return Err(Fail::NextMirror(format!("not found (HTTP {})", resp.status))),
+ s if s >= 500 => return Err(Fail::Retry(format!("server error (HTTP {s})"))),
+ s => return Err(Fail::NextMirror(format!("HTTP {s}"))),
+ };
+ let announced = f.size.unwrap_or_else(|| resp.content_length().unwrap_or(0));
+ report(have, announced);
+ let mut buf = vec![0u8; 256 * 1024];
+ loop {
+ let n = match resp.read(&mut buf, &limits) {
+ Ok(n) => n,
+ Err(e) => {
+ let _ = file.flush();
+ if f.size.is_none() {
+ drop(file);
+ let _ = std::fs::remove_file(&part);
+ }
+ return Err(http_fail(e));
+ }
+ };
+ if n == 0 {
+ break;
+ }
+ if have.saturating_add(n as u64) > limit {
+ drop(file);
+ let _ = std::fs::remove_file(&part);
+ return Err(Fail::NextMirror(format!("the server sent more than the expected {limit} bytes")));
+ }
+ file.write_all(buf.get(..n).unwrap_or_default()).map_err(disk)?;
+ have = have.saturating_add(n as u64);
+ report(have, announced);
+ }
+ file.sync_all().map_err(disk)?;
+ }
+ // checks: exact size, hash
+ let len = std::fs::metadata(&part).map_err(disk)?.len();
+ if f.size.is_some_and(|s| s != len) {
+ return Err(Fail::Retry(format!("incomplete ({len} of {} bytes)", f.size.unwrap_or(0))));
+ }
+ if len == 0 {
+ let _ = std::fs::remove_file(&part);
+ return Err(Fail::NextMirror("the server sent an empty file".into()));
+ }
+ if let Some(want) = f.sha256 {
+ let got = sha256_file(&part).map_err(Fail::Disk)?;
+ if got != want {
+ std::fs::remove_file(&part).map_err(disk)?;
+ let why = format!("the file is damaged or not the official one (SHA-256 {got})");
+ // a damaged partial file from an earlier run isn't this mirror's fault
+ return Err(if fetched { Fail::NextMirror(why) } else { Fail::Retry(why) });
+ }
+ }
+ std::fs::rename(&part, dir.join(f.name)).map_err(disk)?;
+ Ok(())
+}
+
+fn http_fail(e: HttpError) -> Fail {
+ match e {
+ HttpError::Cancelled => Fail::Cancelled,
+ HttpError::Stalled | HttpError::Io(_) | HttpError::Protocol(_) => Fail::Retry(e.to_string()),
+ HttpError::BadUrl(_) | HttpError::Connect(_) | HttpError::Tls(_) => Fail::NextMirror(e.to_string()),
+ }
+}
+
+/// `bytes a-b/total` → (a, total) (`*` total → None).
+fn parse_content_range(v: &str) -> Option<(u64, Option)> {
+ let v = v.trim().strip_prefix("bytes")?.trim();
+ let (range, total) = v.split_once('/')?;
+ let (a, b) = range.split_once('-')?;
+ let (a, b) = (a.trim().parse::().ok()?, b.trim().parse::().ok()?);
+ if b < a {
+ return None;
+ }
+ let total = match total.trim() {
+ "*" => None,
+ t => Some(t.parse::().ok()?),
+ };
+ Some((a, total))
+}
+
+#[cfg(test)]
+mod tests;
diff --git a/crates/segment/src/fetch/tests.rs b/crates/fetch/src/tests.rs
similarity index 97%
rename from crates/segment/src/fetch/tests.rs
rename to crates/fetch/src/tests.rs
index 81e38f01..57c168a2 100644
--- a/crates/segment/src/fetch/tests.rs
+++ b/crates/fetch/src/tests.rs
@@ -294,13 +294,15 @@ fn no_mirrors_is_a_clear_error_and_mirror_lists_are_parsed() {
let dir = tmp("nomirror");
let e = run(&[pinned()], &[], &dir, &AtomicBool::new(false)).0.unwrap_err();
assert_eq!(e, DownloadError::NoMirrors);
- assert!(e.to_string().contains(MIRRORS_ENV));
+ assert!(e.to_string().contains("no download location"));
let file = std::env::temp_dir().join(format!("lc-sam3-mirrors-{}.txt", std::process::id()));
std::fs::write(&file, "# mine\nhttps://b.example/m/ # second\n\nnot a url\nhttps://a.example/x\n").unwrap();
- let m = mirrors(Some("https://a.example/x/, ftp://no"), Some(&file));
- assert_eq!(m[..2], ["https://a.example/x".to_string(), "https://b.example/m".to_string()]);
- assert_eq!(m.len(), 2 + DEFAULT_MIRRORS.len());
- assert_eq!(mirrors(None, Some(Path::new("/nonexistent/mirrors.txt"))).len(), DEFAULT_MIRRORS.len());
+ let defaults = ["https://c.example/d", "https://a.example/x"];
+ let m = mirrors(Some("https://a.example/x/, ftp://no"), Some(&file), &defaults);
+ // the environment's, then the file's, then the defaults; a repeated one is kept once
+ assert_eq!(m, ["https://a.example/x", "https://b.example/m", "https://c.example/d"]);
+ assert_eq!(mirrors(None, Some(Path::new("/nonexistent/mirrors.txt")), &defaults).len(), 2);
+ assert!(mirrors(None, None, &[]).is_empty());
let _ = std::fs::remove_file(&file);
let _ = std::fs::remove_dir_all(&dir);
}
diff --git a/crates/gpu/tests/equivalence.rs b/crates/gpu/tests/equivalence.rs
index c16b46cb..2c1e70e6 100644
--- a/crates/gpu/tests/equivalence.rs
+++ b/crates/gpu/tests/equivalence.rs
@@ -73,7 +73,23 @@ fn check(name: &str, src: &Arc, info: &SourceInfo, s: &DevelopSettings,
let gpu = lightcraft_gpu::render(src, info, s, req, None).expect("gpu render");
let (mean, max, over) = diff(&cpu, &gpu.image);
eprintln!("{name:<28} {}x{} mean {mean:.4} max {max} >1: {:.4}%", cpu.width, cpu.height, over * 100.0);
- assert!(mean < MEAN_LSB && max <= MAX_LSB, "{name}: mean {mean:.4} LSB, max {max} LSB");
+ // A pixel on the very edge of the picture can fall either side of the sampling boundary (the
+ // lens warp with a shifted red plane lands exactly outside it): the CPU and a GPU's rounding
+ // may then disagree on one edge pixel. Tolerate a couple of those, nothing in the interior.
+ let (w, h) = (cpu.width, cpu.height);
+ let strays: Vec<(usize, usize)> = cpu
+ .data
+ .iter()
+ .zip(&gpu.image.data)
+ .enumerate()
+ .filter(|(_, (p, q))| (0..3).any(|c| p[c].abs_diff(q[c]) > MAX_LSB))
+ .map(|(i, _)| (i % w, i / w))
+ .collect();
+ let on_the_edge = |&(x, y): &(usize, usize)| x == 0 || y == 0 || x + 1 == w || y + 1 == h;
+ assert!(
+ mean < MEAN_LSB && (max <= MAX_LSB || (strays.len() <= 2 && strays.iter().all(on_the_edge))),
+ "{name}: mean {mean:.4} LSB, max {max} LSB, pixels over {MAX_LSB}: {strays:?}"
+ );
(mean, max)
}
diff --git a/crates/mcp/src/headless.rs b/crates/mcp/src/headless.rs
index 2ae21b25..162ccce4 100644
--- a/crates/mcp/src/headless.rs
+++ b/crates/mcp/src/headless.rs
@@ -11,8 +11,10 @@ use serde_json::{Value, json};
use crate::backend::Backend;
/// File extensions recognised as photos when expanding folders.
-pub const PHOTO_EXTENSIONS: &[&str] =
- &["jpg", "jpeg", "png", "tif", "tiff", "webp", "dng", "cr2", "cr3", "nef", "arw", "raf", "orf", "rw2", "pef", "psd", "jxl", "gif", "bmp", "avif"];
+pub const PHOTO_EXTENSIONS: &[&str] = &[
+ "jpg", "jpeg", "png", "tif", "tiff", "webp", "dng", "cr2", "cr3", "nef", "arw", "raf", "orf", "rw2", "rwl", "raw", "pef", "psd", "jxl", "gif",
+ "bmp", "avif",
+];
/// Headless backend: a [`Session`] with filesystem hooks.
pub struct Headless {
diff --git a/crates/preview/src/lib.rs b/crates/preview/src/lib.rs
index 50dc68e5..a571ecda 100644
--- a/crates/preview/src/lib.rs
+++ b/crates/preview/src/lib.rs
@@ -28,7 +28,8 @@ pub use pool::JobPool;
/// Rendered-thumbnail cache: memory LRU over an optional disk cache.
pub struct PreviewCache {
- generation: RwLock,
+ /// Current generation; `None` once [`Self::retire`]d (nothing is read or written any more).
+ generation: RwLock>,
mem: Mutex>>,
disk: Option,
}
@@ -36,12 +37,12 @@ pub struct PreviewCache {
impl PreviewCache {
/// Memory-only cache of at most `mem_bytes`.
pub fn memory(mem_bytes: usize) -> PreviewCache {
- PreviewCache { generation: RwLock::new(0), mem: Mutex::new(Lru::new(mem_bytes)), disk: None }
+ PreviewCache { generation: RwLock::new(Some(0)), mem: Mutex::new(Lru::new(mem_bytes)), disk: None }
}
/// Memory LRU + disk cache in `dir` (at most `disk_bytes`).
pub fn with_disk(mem_bytes: usize, dir: &Path, disk_bytes: u64) -> PreviewCache {
- PreviewCache { generation: RwLock::new(0), mem: Mutex::new(Lru::new(mem_bytes)), disk: Some(DiskCache::new(dir, disk_bytes)) }
+ PreviewCache { generation: RwLock::new(Some(0)), mem: Mutex::new(Lru::new(mem_bytes)), disk: Some(DiskCache::new(dir, disk_bytes)) }
}
pub fn disk(&self) -> Option<&DiskCache> {
@@ -53,13 +54,14 @@ impl PreviewCache {
}
/// Changes when explicitly cleared; jobs captured before that clear cannot repopulate it.
+ /// A retired cache reports `u64::MAX`, which no job's generation matches.
pub fn generation(&self) -> u64 {
- *self.generation.read().unwrap_or_else(std::sync::PoisonError::into_inner)
+ self.generation.read().unwrap_or_else(std::sync::PoisonError::into_inner).unwrap_or(u64::MAX)
}
pub fn get_at(&self, generation: u64, key: Hash128) -> Option> {
let current = self.generation.read().unwrap_or_else(std::sync::PoisonError::into_inner);
- if *current != generation {
+ if *current != Some(generation) {
return None;
}
if let Some(v) = self.mem.lock().unwrap_or_else(|e| e.into_inner()).get(&key) {
@@ -76,7 +78,7 @@ impl PreviewCache {
pub fn put_at(&self, generation: u64, key: Hash128, img: Arc) {
let current = self.generation.read().unwrap_or_else(std::sync::PoisonError::into_inner);
- if *current != generation {
+ if *current != Some(generation) {
return;
}
if let Some(d) = &self.disk {
@@ -94,7 +96,7 @@ impl PreviewCache {
pub fn put_deferred_at(self: &Arc, generation: u64, key: Hash128, img: Arc) {
let current = self.generation.read().unwrap_or_else(std::sync::PoisonError::into_inner);
- if *current != generation {
+ if *current != Some(generation) {
return;
}
let c = cost(&img);
@@ -106,7 +108,7 @@ impl PreviewCache {
let me = self.clone();
let write = move || {
let current = me.generation.read().unwrap_or_else(std::sync::PoisonError::into_inner);
- if *current != generation {
+ if *current != Some(generation) {
return;
}
if let Some(d) = &me.disk {
@@ -121,16 +123,30 @@ impl PreviewCache {
write();
}
- /// Drop everything (memory and disk).
+ /// Drop everything (memory and disk). Does nothing once [`Self::retire`]d.
pub fn clear(&self) {
let mut generation = self.generation.write().unwrap_or_else(std::sync::PoisonError::into_inner);
- *generation = generation.wrapping_add(1);
+ // a retired cache no longer owns its directory (its replacement may)
+ let Some(g) = generation.as_mut() else { return };
+ // (never reaches u64::MAX, which stands for "retired")
+ *g = g.wrapping_add(1) % u64::MAX;
self.mem.lock().unwrap_or_else(|e| e.into_inner()).clear();
if let Some(d) = &self.disk {
d.clear();
}
}
+ /// Stop this cache object for good, leaving its disk files alone: called when another cache
+ /// object replaces it (a library reopened or another one opened). Jobs still holding it then
+ /// can neither read nor write through it, so a later clear of the replacement's directory
+ /// (possibly the same one) can't be repopulated by them. Waits for writes in progress, like
+ /// [`Self::clear`].
+ pub fn retire(&self) {
+ let mut generation = self.generation.write().unwrap_or_else(std::sync::PoisonError::into_inner);
+ *generation = None;
+ self.mem.lock().unwrap_or_else(|e| e.into_inner()).clear();
+ }
+
/// Use tick ([`next_tick`]) of the least recently used image in memory.
pub fn oldest_tick(&self) -> Option {
self.mem.lock().unwrap_or_else(|e| e.into_inner()).oldest_tick()
diff --git a/crates/preview/src/tests.rs b/crates/preview/src/tests.rs
index ca4d39f8..3d634003 100644
--- a/crates/preview/src/tests.rs
+++ b/crates/preview/src/tests.rs
@@ -26,6 +26,30 @@ fn cleared_generation_cannot_read_or_repopulate_memory_or_disk() {
cache.clear();
}
+#[test]
+fn retired_cache_keeps_its_files_but_refuses_every_read_and_write() {
+ let dir = temp_dir("retire");
+ let old = Arc::new(PreviewCache::with_disk(1 << 20, &dir, 1 << 20));
+ let (kept, late) = (hash_bytes(b"kept"), hash_bytes(b"late"));
+ let img = Arc::new(gradient(8, 8, 7));
+ let generation = old.generation();
+ old.put_at(generation, kept, img.clone());
+ old.retire();
+ assert_ne!(old.generation(), generation);
+ assert!(old.get_at(generation, kept).is_none());
+ assert!(old.get(kept).is_none());
+ old.put_at(generation, late, img.clone());
+ old.put_deferred_at(generation, late, img.clone());
+ old.put(late, img.clone());
+ old.put_deferred(late, img.clone());
+ old.clear();
+ // a replacement for the same directory still finds the valid file, and only that
+ let new = PreviewCache::with_disk(1 << 20, &dir, 1 << 20);
+ assert_eq!(new.get(kept).map(|i| (i.width, i.height)), Some((8, 8)), "the disk file stays (JPEG: lossy)");
+ assert!(new.get(late).is_none());
+ new.clear();
+}
+
fn temp_dir(tag: &str) -> std::path::PathBuf {
let d = std::env::temp_dir().join(format!("lc-preview-{tag}-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&d);
diff --git a/crates/raw/src/dngwrite.rs b/crates/raw/src/dngwrite.rs
index 3df8d706..bd46a2c5 100644
--- a/crates/raw/src/dngwrite.rs
+++ b/crates/raw/src/dngwrite.rs
@@ -177,6 +177,9 @@ pub fn write_dng(raw: &RawImage, opts: &DngWriteOptions) -> Result> {
ifd.set(t::AS_SHOT_NEUTRAL, rat_vec(&v));
} else if let Some(xy) = c.as_shot_white_xy {
ifd.set(t::AS_SHOT_WHITE_XY, rat_vec(&[xy.x, xy.y]));
+ } else if let Some(m) = raw.wb_multipliers.filter(|m| m.iter().all(|v| *v > 0.0 && v.is_finite())) {
+ // vendor raws (NEF, ARW…) carry white balance as multipliers: the DNG neutral is their reciprocal
+ ifd.set(t::AS_SHOT_NEUTRAL, rat_vec(&m.map(|v| 1.0 / v as f64)));
}
ifd.set(t::BASELINE_EXPOSURE, Value::SRational(vec![srational(c.baseline_exposure)]));
// the source's own profile look travels with its data (conversions, smart previews, merges)
diff --git a/crates/raw/src/lib.rs b/crates/raw/src/lib.rs
index d4791e7f..f67649de 100644
--- a/crates/raw/src/lib.rs
+++ b/crates/raw/src/lib.rs
@@ -13,9 +13,10 @@
//!
//! Formats: DNG (uncompressed, lossless JPEG, lossy JPEG (Smart Previews), Deflate incl. floating point, tiled/stripped, CFA and LinearRaw),
//! Canon CR2, Nikon NEF/NRW (uncompressed, Huffman lossless / lossy compressed), Sony ARW (uncompressed, ARW2, lossless), Fujifilm RAF (uncompressed Bayer
-//! and X-Trans), Panasonic RW2 (packed 12/14-bit), Pentax PEF (uncompressed, Huffman), Olympus ORF (uncompressed).
+//! and X-Trans), Panasonic RW2 / Leica RWL / Panasonic RAW (every raw format: compressed 4 and 6, the prefix-coded strips of 8,
+//! packed 2/5/7, the 16-bit words of the oldest bodies), Pentax PEF (uncompressed, Huffman), Olympus ORF (uncompressed).
//! [`embedded_preview`] covers all of them plus CR3. Variants we can't decode yet (Nikon "lossy after split" NEF,
-//! Panasonic quantised RW2, compressed ORF/RAF, CR3) return [`RawError::Unsupported`]; each vendor module documents its sources
+//! compressed ORF/RAF, CR3) return [`RawError::Unsupported`]; each vendor module documents its sources
//! (public specifications, tag-name documentation, black-box analysis of CC0 samples) and gaps. Non-DNG files carry no
//! colour matrix: [`color`] falls back to a documented neutral model. The decoders never panic on malformed input.
#![forbid(unsafe_code)]
diff --git a/crates/raw/src/vendor/rw2.rs b/crates/raw/src/vendor/rw2.rs
index efd515f2..10ade627 100644
--- a/crates/raw/src/vendor/rw2.rs
+++ b/crates/raw/src/vendor/rw2.rs
@@ -1,28 +1,80 @@
-//! Panasonic RW2 — the packed (unquantised) variant.
+//! Panasonic RW2, Leica RWL and the older Panasonic / Leica `.RAW` files: every raw encoding these cameras write.
//!
-//! Sources: the ExifTool PanasonicRaw tag-name documentation (`0x0002/0x0003` sensor size, `0x0004–0x0007`
-//! sensor borders, `0x0009` CFA pattern, `0x000a` bits per sample, `0x000e` linearity limit, `0x001c–0x001e`
-//! black levels, `0x0024–0x0026` WB levels, `0x0017` ISO, `0x002d` raw format, `0x002e` JpgFromRaw, `0x0118` raw data
-//! offset) and our own black-box analysis of CC0 samples from raw.pixls.us (DC-G9 12-bit, DC-GH5S 14-bit):
+//! **Clean-room.** No raw-decoder source code (dcraw, LibRaw, rawspeed, rawloader/rawler, darktable, RawTherapee,
+//! libopenraw, ExifTool's Perl code, …) was read or consulted. What we used:
//!
-//! - The raw data is a sequence of 0x4000-byte chunks; each chunk is stored rotated: its logical content is
-//! bytes `0x1ff8..0x4000` followed by bytes `0..0x1ff8`. (Found by trying every 8-byte rotation and keeping
-//! the one under which the block at each chunk seam is as smooth as its neighbours.)
-//! - The logical stream is made of 16-byte blocks, each a little-endian 128-bit word read LSB-first from bit 0:
-//! 12-bit data packs 10 samples (8 padding bits at the top), 14-bit data 9 samples (2 padding bits at the top).
-//! Found from the per-bit-position probability of a 1 (padding is always 0; the top bits of 12-bit samples and
-//! the two lowest bits of the GH5S's 14-bit samples are almost always 0), confirmed by image smoothness and by
-//! the decoded masked-free dark level matching the recorded black levels.
-//! - Pixels fill rows left to right, rows top to bottom.
+//! - Tag documentation: ExifTool's PanasonicRaw tag-name tables (IFD0 `0x0002`–`0x0007` sensor size and borders,
+//! `0x0009` CFAPattern with its four values, `0x000a` BitsPerSample, `0x000b` Compression, `0x000e`–`0x0010`
+//! LinearityLimit, `0x0017`/`0x0037` ISO, `0x001c`–`0x001e` black levels, `0x0024`–`0x0026` WB levels, `0x002d`
+//! RawFormat, `0x002e` JpgFromRaw, `0x002f`–`0x0032` crop, `0x0118` RawDataOffset) and the libopenraw format
+//! notes (prose: the `IIU` magic; Leica RWL files are RW2).
+//! - Our own black-box analysis of 178 CC0 files from raw.pixls.us, about 120 bodies from the DMC-LX1 (2005) to
+//! the DC-S1RM2 (2025), every encoding below. The measurement that settled each rule is noted in brackets.
//!
-//! Panasonic's quantised variant (raw format 4: 14 pixels per block, two 12-bit seeds plus four groups of a 2-bit
-//! scale and three 8-bit codes) is not decoded: our black-box analysis identified the block layout but not the
-//! reconstruction rule for every scale, and no permissively licensed description exists. Those files report
-//! [`RawError::Unsupported`]; their embedded preview still works.
+//! Encodings, by raw format (tag `0x002d`; the oldest files have none and are told apart by compression `0x000b`):
+//!
+//! - **16-bit words** (compression 34828 and 34830: DMC-LX1, DMC-FZ50, Digilux 3): little-endian words at
+//! `StripOffsets`, the sample left-justified [every value is a multiple of 2^(16 − bits)].
+//! - **Packed** (formats 2, 5 and 7): 16-byte blocks read as a little-endian 128-bit number holding ⌊128 / bits⌋
+//! samples from bit 0 up (12-bit: 10, 14-bit: 9). Format 5 stores every whole 0x4000-byte chunk rotated (its
+//! logical content is bytes `0x1ff8..0x4000` followed by `0..0x1ff8`); formats 2 (DMC-FZ8) and 7 (DC-S5 with
+//! later firmware) are not rotated [vertical same-colour differences 20–180× smaller with the right choice].
+//! - **Format 4** (most bodies, 2008–2022): chunks rotated as format 5; each 16-byte block, read as a little-endian
+//! 128-bit number from bit 127 down, codes 14 pixels of a row. Before pixels 2, 5, 8 and 11 comes a 2-bit scale
+//! `s` for the next three pixels, step `m` = 1, 2, 4 or 16 [per-bit statistics show the field boundaries]. The
+//! first pixel of each colour (parity of the pixel index `i`) is a seed: 8 bits `h`, then, if `h ≠ 0` or
+//! `i ≥ 12`, 4 bits `l`, value `16h + l`. `h = 0` (with `i < 12`) marks a mapped-out defective pixel, decoded as
+//! 0, and the next pixel of that colour is read as the seed instead [these blocks decode smoothly only this way,
+//! 10× closer to the rows above and below, and they sit at the same sensor sites in every shot of a body]. Every
+//! later pixel is an 8-bit code `j` against the colour's previous value `p`: `j = 0` keeps `p` [3–4× closer to
+//! the neighbouring rows than reading it as a difference]; otherwise `p + (j − 128)·m` when `s < 3` and
+//! `p ≥ 128·m`, else `j·m + (p mod m)` [scale-3 codes track the absolute level, ≈ level / 16; values stay on the
+//! camera's 4k + 3 lattice (GX80, 99.9–100 % per branch) only when the absolute reading keeps the low bits of `p`].
+//! - **Format 6** (DC-S1, S1R, S1H, S5; DC-GH5M2): not rotated. 14-bit: two 14-bit seeds, then three groups of a
+//! 2-bit scale and three 10-bit codes (11 pixels; the 4 lowest bits are unused); 12-bit: two 12-bit seeds and four
+//! groups of a 2-bit scale and three 8-bit codes (14 pixels). Read from bit 127 down and reconstructed as format 4
+//! with the bias 512 (14-bit) or 128 (12-bit) [code statistics per scale as in format 4; flat saturated blocks
+//! decode flat only when the absolute reading keeps the low bits]. Seeds are never deferred and zero codes don't
+//! occur in any sample; a zero code keeps the previous value as in format 4.
+//! - **Format 8** (DC-GH6 and later: GH6, GH7, G9M2, S5M2, S5M2X, S9, S1RM2, S1M2): the image is split into vertical
+//! strips listed in `u16` arrays (a count, then the values; 32-bit values as low/high pairs): `0x0044` absolute
+//! offsets, `0x0045` left columns, `0x0046` lengths in bits, `0x0047` widths, `0x0048` heights. Each strip is an
+//! independent bit stream whose bits are taken from each byte starting at bit 0 [a constant run at a strip's end
+//! reads as repeats of one code only in this order; MSB-first reading gives code frequencies of exactly
+//! 2^−length, the signature of random bits]. Tag `0x0040` lists (length, code) for 17 prefix codes, the difference
+//! categories 0–16 of ITU-T T.81 (category `c`, then `c` additional bits, a leading 0 meaning negative). Pixels
+//! are coded two rows at a time, cell by 2×2 cell (top-left, top-right, bottom-left, bottom-right); each is
+//! predicted from the same site of the previous cell, the first cell of a row pair from the first cell of the
+//! previous pair, 0 at the top [the strips decode to the camera JPEG's image without seams]. The streams end a
+//! few padding rows before their stated length. Tags `0x0039`–`0x003f`, `0x0041` and `0x0043` carry the same
+//! values in every sample; files with other values are reported as unsupported rather than guessed at.
+//!
+//! Levels and geometry:
+//!
+//! - Black: per colour from `0x001c`–`0x001e`. Formats 4 and 6 store every sample 15 above it [flat dark regions
+//! (σ ≈ 1–3) of ~100 files sit 13–20 above the tag, e.g. 143.1 on a D-Lux 7 tagged 128], the other formats
+//! don't [131.4 on a format-5 G9 tagged 127.5; 128.7 on a format-8 S9 tagged 128]. Files that record no black
+//! (tags 0: bodies up to ~2010) use 15 [their darkest flat regions sit at 15–22].
+//! - White: the saturation plateau of the active area's samples, else the full range of the bit depth. The linearity
+//! limits (`0x000e`–`0x0010`) are not used: extended low-ISO shots record limits far below their samples (TZ200D
+//! at ISO 100: 3277, samples up to 3529) while the camera's own JPEG renders them against the full range [the
+//! JPEG/raw brightness ratio matches normal shots of the same bodies only then], the Digilux 3 records 4095 but
+//! saturates at 15871 of 16383, and the GF1 records 16383 for 12-bit samples.
+//! - Samples of 0 mark mapped-out defects (format 4 above; isolated zeros between normal neighbours in the 16-bit
+//! files) and are interpolated from the same colour two pixels away.
+//! - The CFA pattern applies at sample (0, 0) of the data, not at the sensor borders [the two files with odd borders
+//! show the greens on the diagonal this predicts]; red and blue were checked against every file's JPEG.
+//! - Active area: the sensor borders. Default crop: `0x002f`–`0x0032`, the in-camera aspect ratio (e.g. 4:3 or 1:1
+//! on a 3:2 sensor; the embedded JPEG always shows the whole sensor). Some bodies write zeros there: ignored.
+//!
+//! Not supported ([`RawError::Unsupported`], the embedded preview is shown instead): compression 34826, raw formats
+//! 1 and 3, and bit depths other than the ones above (no samples of any of them).
-use crate::{BlackLevel, Cfa, ColorData, Mode, OpcodeLists, RawData, RawError, RawFormat, RawImage, Rect, Result};
+use super::pef::{Bits, Huffman, diff};
+use super::white_from_data;
+use crate::{BlackLevel, Cfa, ColorData, MAX_SAMPLES, Mode, OpcodeLists, RawData, RawError, RawFormat, RawImage, Rect, Result};
use lightcraft_geom::Orientation;
-use lightcraft_tiff::{Tiff, tags as t};
+use lightcraft_tiff::{ByteOrder, Ifd, Tiff, Value, tags as t};
use rayon::prelude::*;
const SENSOR_WIDTH: u16 = 0x0002;
@@ -33,96 +85,444 @@ const BORDER_BOTTOM: u16 = 0x0006;
const BORDER_RIGHT: u16 = 0x0007;
const CFA_PATTERN: u16 = 0x0009;
const BITS: u16 = 0x000a;
-const LINEARITY_LIMIT: u16 = 0x000e;
+const COMPRESSION: u16 = 0x000b;
+const LINEARITY_LIMIT: [u16; 3] = [0x000e, 0x000f, 0x0010];
+const ISO: u16 = 0x0017;
const BLACK_RGB: [u16; 3] = [0x001c, 0x001d, 0x001e];
const WB_RGB: [u16; 3] = [0x0024, 0x0025, 0x0026];
const RAW_FORMAT: u16 = 0x002d;
-const ISO: u16 = 0x0017;
+/// The in-camera aspect-ratio crop: top, left, bottom, right in sensor coordinates.
+const CROP: [u16; 4] = [0x002f, 0x0030, 0x0031, 0x0032];
+const ISO_32: u16 = 0x0037;
const RAW_OFFSET: u16 = 0x0118;
+const HUFFMAN_CODES: u16 = 0x0040;
+const STRIP_OFFSETS: u16 = 0x0044;
+const STRIP_LEFT: u16 = 0x0045;
+const STRIP_BITS: u16 = 0x0046;
+const STRIP_WIDTHS: u16 = 0x0047;
+const STRIP_HEIGHTS: u16 = 0x0048;
+/// Format-8 tags of unknown meaning and the values every sample carries (as `u16` arrays).
+const FORMAT8_FIXED: [(u16, &[u16]); 9] = [
+ (0x0039, &[6, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1]),
+ (0x003a, &[6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]),
+ (0x003b, &[65535]),
+ (0x003c, &[0]),
+ (0x003d, &[0]),
+ (0x003e, &[0]),
+ (0x003f, &[0]),
+ (0x0041, &[17, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]),
+ (0x0043, &[1]),
+];
+/// Most strips seen in a file is 3 (the 12008 × 8008 high-resolution mode).
+const MAX_STRIPS: usize = 64;
const CHUNK: usize = 0x4000;
const ROTATE: usize = 0x1ff8;
+/// Formats 4 and 6 store samples this far above the recorded black level.
+const CODEC_BLACK_OFFSET: f32 = 15.0;
+/// Black level of files that record none.
+const DEFAULT_BLACK: f32 = 15.0;
-/// The logical byte stream of the raw data (chunks un-rotated). A trailing partial chunk is kept as is.
-pub(crate) fn unrotate(src: &[u8]) -> Vec {
- let mut out = Vec::with_capacity(src.len());
- for c in src.chunks(CHUNK) {
- if c.len() == CHUNK {
- out.extend_from_slice(&c[ROTATE..]);
- out.extend_from_slice(&c[..ROTATE]);
- } else {
- out.extend_from_slice(c);
+/// How the samples are stored.
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+enum Encoding {
+ /// Little-endian 16-bit words, samples left-justified.
+ Words,
+ /// Continuous 16-byte blocks; `rotated`: whole 0x4000-byte chunks are stored rotated by 0x1ff8.
+ Blocks { kind: Block, rotated: bool },
+ /// Format 8: prefix-coded strips.
+ Strips,
+}
+
+/// What one 16-byte block holds.
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+enum Block {
+ /// ⌊128 / bits⌋ samples from bit 0 up.
+ Packed {
+ bits: u32,
+ },
+ Format4,
+ Format6 {
+ bits14: bool,
+ },
+}
+
+impl Block {
+ fn per(self) -> usize {
+ match self {
+ Block::Packed { bits } => (128 / bits.max(1)) as usize,
+ Block::Format4 | Block::Format6 { bits14: false } => 14,
+ Block::Format6 { bits14: true } => 11,
}
}
- out
-}
-/// Samples per 16-byte block and the bit position of the first sample (kept for other layouts).
-fn layout(bits: u32) -> Option<(usize, u32)> {
- match bits {
- 12 => Some((10, 0)),
- 14 => Some((9, 0)),
- _ => None,
+ fn decode(self, b: &[u8; 16], out: &mut [u16]) {
+ match self {
+ Block::Packed { bits } => packed(b, bits, out),
+ Block::Format4 => format4(b, out),
+ Block::Format6 { bits14 } => format6(b, bits14, out),
+ }
}
}
-/// Unpack `out.len()` samples from the logical stream.
-pub(crate) fn unpack(stream: &[u8], bits: u32, out: &mut [u16]) -> Result<()> {
- let (per, start) = layout(bits).ok_or_else(|| RawError::Unsupported(format!("RW2 packed {bits}-bit data")))?;
- let mask = (1u128 << bits) - 1;
- out.par_chunks_mut(per * 1024).enumerate().for_each(|(ci, o)| {
- let first_block = ci * 1024;
- for (bi, px) in o.chunks_mut(per).enumerate() {
- let at = (first_block + bi) * 16;
- let Some(b) = stream.get(at..at + 16).and_then(|b| <[u8; 16]>::try_from(b).ok()) else {
- px.fill(0);
- continue;
- };
- let word = u128::from_le_bytes(b);
- for (i, p) in px.iter_mut().enumerate() {
- *p = ((word >> (start + i as u32 * bits)) & mask) as u16;
+fn encoding(ifd: &Ifd, bits: u32) -> Result {
+ let blocks = |kind, rotated| Ok(Encoding::Blocks { kind, rotated });
+ match (ifd.u16(RAW_FORMAT), ifd.u16(COMPRESSION)) {
+ (Some(2), _) if bits == 12 => blocks(Block::Packed { bits }, false),
+ (Some(4), _) if bits == 12 => blocks(Block::Format4, true),
+ (Some(5), _) if bits == 12 || bits == 14 => blocks(Block::Packed { bits }, true),
+ (Some(6), _) if bits == 12 || bits == 14 => blocks(Block::Format6 { bits14: bits == 14 }, false),
+ (Some(7), _) if bits == 14 => blocks(Block::Packed { bits }, false),
+ (Some(8), _) if (12..=16).contains(&bits) => Ok(Encoding::Strips),
+ (None, Some(34828 | 34830)) if (12..=16).contains(&bits) => Ok(Encoding::Words),
+ (Some(f), _) => Err(RawError::Unsupported(format!("Panasonic raw format {f} with {bits}-bit samples"))),
+ (None, c) => Err(RawError::Unsupported(format!("Panasonic raw without a raw format (compression {c:?})"))),
+ }
+}
+
+/// Bit fields of a 16-byte block taken as a little-endian 128-bit number, read from bit 127 down.
+struct Fields {
+ word: u128,
+ left: u32,
+}
+
+impl Fields {
+ fn new(b: &[u8; 16]) -> Fields {
+ Fields { word: u128::from_le_bytes(*b), left: 128 }
+ }
+
+ /// The next `n` (1..=16) bits.
+ #[inline]
+ fn take(&mut self, n: u32) -> i32 {
+ self.left = self.left.saturating_sub(n);
+ ((self.word >> self.left) & ((1u128 << n) - 1)) as i32
+ }
+}
+
+/// Step and absolute flag of the 2-bit scales of formats 4 and 6.
+const STEPS: [(i32, bool); 4] = [(1, false), (2, false), (4, false), (16, true)];
+
+/// Formats 4 and 6: the next value of a colour from its previous value `p` and code `j` (see the module docs).
+#[inline]
+fn next(p: i32, j: i32, (m, absolute): (i32, bool), bias: i32) -> i32 {
+ if j == 0 {
+ p
+ } else if !absolute && p >= bias * m {
+ p + (j - bias) * m
+ } else {
+ j * m + (p & (m - 1))
+ }
+}
+
+#[inline]
+fn put(out: &mut [u16], i: usize, v: i32) {
+ if let Some(o) = out.get_mut(i) {
+ *o = v.clamp(0, u16::MAX as i32) as u16;
+ }
+}
+
+fn packed(b: &[u8; 16], bits: u32, out: &mut [u16]) {
+ let word = u128::from_le_bytes(*b);
+ let mask = (1u128 << bits.min(16)) - 1;
+ for (i, o) in out.iter_mut().enumerate().take((128 / bits.max(1)) as usize) {
+ *o = ((word >> (i as u32 * bits)) & mask) as u16;
+ }
+}
+
+fn format4(b: &[u8; 16], out: &mut [u16]) {
+ let mut f = Fields::new(b);
+ let (mut prev, mut seeded, mut step) = ([0i32; 2], [false; 2], STEPS[0]);
+ for i in 0..14 {
+ if i % 3 == 2 {
+ step = STEPS[f.take(2) as usize & 3];
+ }
+ let c = i & 1;
+ let v = if seeded[c] {
+ prev[c] = next(prev[c], f.take(8), step, 128);
+ prev[c]
+ } else {
+ let hi = f.take(8);
+ if hi != 0 || i >= 12 {
+ prev[c] = hi << 4 | f.take(4);
+ seeded[c] = hi != 0;
+ prev[c]
+ } else {
+ 0 // a defect marker; this colour's seed comes with its next pixel
+ }
+ };
+ put(out, i, v);
+ }
+}
+
+fn format6(b: &[u8; 16], bits14: bool, out: &mut [u16]) {
+ let (seed, code, groups, bias) = if bits14 { (14, 10, 3, 512) } else { (12, 8, 4, 128) };
+ let mut f = Fields::new(b);
+ let mut prev = [f.take(seed), f.take(seed)];
+ put(out, 0, prev[0]);
+ put(out, 1, prev[1]);
+ for g in 0..groups {
+ let step = STEPS[f.take(2) as usize & 3];
+ for k in 0..3 {
+ let i = 2 + 3 * g + k;
+ prev[i & 1] = next(prev[i & 1], f.take(code), step, bias);
+ put(out, i, prev[i & 1]);
+ }
+ }
+}
+
+/// Decode a continuous stream of 16-byte blocks into `out`, in parallel by 0x4000-byte chunks (whole chunks
+/// rotated when `rotated`). Blocks missing from `src` give zeros.
+fn decode_blocks(src: &[u8], kind: Block, rotated: bool, out: &mut [u16]) {
+ let per = kind.per();
+ out.par_chunks_mut(CHUNK / 16 * per).enumerate().for_each(|(c, o)| {
+ let raw = src.get(c * CHUNK..).unwrap_or_default();
+ let raw = raw.get(..CHUNK).unwrap_or(raw);
+ let mut logical = Vec::new();
+ let chunk = if rotated && raw.len() == CHUNK {
+ logical.extend_from_slice(raw.get(ROTATE..).unwrap_or_default());
+ logical.extend_from_slice(raw.get(..ROTATE).unwrap_or_default());
+ &logical
+ } else {
+ raw
+ };
+ for (i, px) in o.chunks_mut(per).enumerate() {
+ match chunk.get(i * 16..i * 16 + 16).and_then(|s| <&[u8; 16]>::try_from(s).ok()) {
+ Some(b) => kind.decode(b, px),
+ None => px.fill(0),
}
}
});
+}
+
+fn decode_words(src: &[u8], shift: u32, out: &mut [u16]) {
+ out.par_chunks_mut(1 << 16).enumerate().for_each(|(k, o)| {
+ for (i, v) in o.iter_mut().enumerate() {
+ let at = ((k << 16) + i) * 2;
+ *v = src.get(at..at + 2).and_then(|b| <[u8; 2]>::try_from(b).ok()).map_or(0, |b| u16::from_le_bytes(b) >> shift);
+ }
+ });
+}
+
+/// A tag as a `u16` array: SHORT values, or the little-endian contents of an UNDEFINED / BYTE value.
+fn u16_array(ifd: &Ifd, tag: u16, order: ByteOrder) -> Option> {
+ match ifd.value(tag)? {
+ Value::Undefined(b) | Value::Byte(b) => Some(b.as_chunks::<2>().0.iter().map(|c| order.u16(*c)).collect()),
+ Value::Short(v) => Some(v.clone()),
+ other => Some(other.to_u64_vec().into_iter().map(|v| v.min(u16::MAX as u64) as u16).collect()),
+ }
+}
+
+/// A format-8 `u16` list (count, then the values; `wide`: 32-bit values as low/high pairs).
+fn list(ifd: &Ifd, tag: u16, order: ByteOrder, wide: bool) -> Result> {
+ let a = u16_array(ifd, tag, order).ok_or_else(|| RawError::Corrupt(format!("RW2 format 8 without tag {tag:#06x}")))?;
+ let n = a.first().map_or(0, |&n| n as usize);
+ let at = |i: usize| a.get(i).map(|&v| v as u64);
+ let values: Option> = (0..n).map(|i| if wide { Some(at(1 + 2 * i)? | at(2 + 2 * i)? << 16) } else { at(1 + i) }).collect();
+ values.filter(|v| !v.is_empty() && v.len() <= MAX_STRIPS).ok_or_else(|| RawError::Corrupt(format!("RW2 format 8: bad list in tag {tag:#06x}")))
+}
+
+/// One format-8 strip: its bit stream and placement.
+struct Strip<'a> {
+ data: &'a [u8],
+ width: usize,
+ height: usize,
+ bits: u64,
+}
+
+/// The format-8 strips (tiling the full width, top to bottom) and code table.
+fn format8<'a>(ifd: &Ifd, order: ByteOrder, bytes: &'a [u8], w: usize, h: usize) -> Result<(Vec>, Huffman)> {
+ for (tag, want) in FORMAT8_FIXED {
+ if let Some(v) = u16_array(ifd, tag, order)
+ && v != want
+ {
+ return Err(RawError::Unsupported(format!("Panasonic raw format 8 with tag {tag:#06x} = {v:?}")));
+ }
+ }
+ let (offsets, lefts, bits) = (list(ifd, STRIP_OFFSETS, order, true)?, list(ifd, STRIP_LEFT, order, true)?, list(ifd, STRIP_BITS, order, true)?);
+ let (widths, heights) = (list(ifd, STRIP_WIDTHS, order, false)?, list(ifd, STRIP_HEIGHTS, order, false)?);
+ let n = offsets.len();
+ if [lefts.len(), bits.len(), widths.len(), heights.len()].iter().any(|&l| l != n) {
+ return Err(RawError::Corrupt("RW2 format 8: strip lists of different lengths".into()));
+ }
+ let mut strips = Vec::with_capacity(n);
+ let mut x = 0usize;
+ for i in 0..n {
+ let (left, width, height) = (lefts[i] as usize, widths[i] as usize, heights[i] as usize);
+ let len = usize::try_from(bits[i].div_ceil(8)).unwrap_or(usize::MAX);
+ let data = usize::try_from(offsets[i]).ok().and_then(|o| bytes.get(o..o.checked_add(len)?));
+ let Some(data) = data.filter(|_| left == x && width > 0 && height == h) else {
+ return Err(RawError::Corrupt(format!("RW2 format 8: strip {i} outside the image or file")));
+ };
+ x += width;
+ strips.push(Strip { data, width, height, bits: bits[i] });
+ }
+ if x != w {
+ return Err(RawError::Corrupt(format!("RW2 format 8: strips cover {x} of {w} columns")));
+ }
+ let codes = u16_array(ifd, HUFFMAN_CODES, order).unwrap_or_default();
+ let pairs: Vec<(u32, u32)> = match (codes.first(), codes.get(1..35)) {
+ (Some(17), Some(c)) => c.as_chunks::<2>().0.iter().map(|&[l, c]| (l as u32, c as u32)).collect(),
+ _ => return Err(RawError::Corrupt("RW2 format 8: bad code table".into())),
+ };
+ Ok((strips, code_table(&pairs)?))
+}
+
+/// The prefix-code table from (length, code) per difference category 0..=16. In 12- and 14-bit files the codes of the
+/// categories the bit depth can't produce start with a shorter code (8 ones): read bit by bit, the shorter code wins
+/// and the longer ones are unreachable.
+fn code_table(pairs: &[(u32, u32)]) -> Result {
+ let mut table = Vec::with_capacity(pairs.len());
+ for (i, &(la, ca)) in pairs.iter().enumerate() {
+ if la > 12 {
+ return Err(RawError::Unsupported(format!("RW2 format 8: {la}-bit code")));
+ }
+ if la > 0 && ca >> la != 0 {
+ return Err(RawError::Corrupt("RW2 format 8: code longer than its length".into()));
+ }
+ let shadowed = pairs.iter().any(|&(lb, cb)| lb > 0 && lb < la && ca >> (la - lb) == cb);
+ let duplicate = pairs.iter().enumerate().any(|(j, &(lb, cb))| j != i && la > 0 && (lb, cb) == (la, ca));
+ if duplicate {
+ return Err(RawError::Corrupt("RW2 format 8: two categories with the same code".into()));
+ }
+ table.push(if la == 0 || shadowed { (0, 0) } else { ((ca << (12 - la)) as u16, la as u8) });
+ }
+ Huffman::new(&table)
+}
+
+/// Decode one strip into its column range of each output row.
+fn decode_strip(s: &Strip, codes: &Huffman, mut rows: Vec<&mut [u16]>) -> Result<()> {
+ // the stream's bits are taken from each byte starting at bit 0: reversed, it is an ordinary MSB-first stream
+ let reversed: Vec = s.data.iter().map(|b| b.reverse_bits()).collect();
+ let mut bits = Bits::new(&reversed);
+ let mut first = [0i32; 4];
+ for pair in 0..s.height.div_ceil(2) {
+ let mut cell = first;
+ for cx in 0..s.width.div_ceil(2) {
+ for (k, v) in cell.iter_mut().enumerate() {
+ let d = diff(&mut bits, codes).ok_or_else(|| RawError::Corrupt(format!("RW2 format 8: invalid code in row {}", 2 * pair)))?;
+ *v = v.saturating_add(d);
+ if let Some(px) = rows.get_mut(2 * pair + k / 2).and_then(|r| r.get_mut(2 * cx + k % 2)) {
+ *px = (*v).clamp(0, u16::MAX as i32) as u16;
+ }
+ }
+ if cx == 0 {
+ first = cell;
+ }
+ }
+ if bits.consumed_bits() as u64 > s.bits {
+ return Err(RawError::Corrupt(format!("RW2 format 8: strip data ends at row {} of {}", 2 * pair, s.height)));
+ }
+ }
Ok(())
}
+fn decode_strips(strips: &[Strip], codes: &Huffman, w: usize, out: &mut [u16]) -> Result<()> {
+ let mut parts: Vec> = strips.iter().map(|s| Vec::with_capacity(s.height)).collect();
+ for row in out.chunks_mut(w) {
+ let mut rest = row;
+ for (s, rows) in strips.iter().zip(parts.iter_mut()) {
+ let (part, tail) = rest.split_at_mut(s.width.min(rest.len()));
+ rows.push(part);
+ rest = tail;
+ }
+ }
+ strips.par_iter().zip(parts).try_for_each(|(s, rows)| decode_strip(s, codes, rows))
+}
+
+/// Replace samples of 0 inside `area` (mapped-out defects) by the mean of the non-zero samples of the same colour two
+/// pixels away.
+fn fill_defects(data: &mut [u16], w: usize, area: Rect) {
+ for y in area.y..area.y + area.height {
+ let Some(row) = data.get(y * w + area.x..y * w + area.x + area.width) else { break };
+ if !row.contains(&0) {
+ continue;
+ }
+ for x in area.x..area.x + area.width {
+ let i = y * w + x;
+ if data.get(i) != Some(&0) {
+ continue;
+ }
+ let near = [(x.checked_sub(2), Some(y)), (Some(x + 2).filter(|&x| x < w), Some(y)), (Some(x), y.checked_sub(2)), (Some(x), Some(y + 2))];
+ let (mut sum, mut n) = (0u32, 0u32);
+ for (nx, ny) in near {
+ if let (Some(nx), Some(ny)) = (nx, ny)
+ && let Some(&v) = data.get(ny * w + nx).filter(|&&v| v != 0)
+ {
+ sum += v as u32;
+ n += 1;
+ }
+ }
+ if n > 0
+ && let Some(v) = data.get_mut(i)
+ {
+ *v = (sum / n) as u16;
+ }
+ }
+ }
+}
+
+/// Samples of up to 512 evenly spaced rows of the active area (the borders hold junk such as 0xffff runs).
+fn active_rows(data: &[u16], w: usize, a: Rect) -> Vec {
+ let step = (a.height / 512).max(1);
+ (a.y..a.y + a.height).step_by(step).filter_map(|y| data.get(y * w + a.x..y * w + a.x + a.width)).flatten().copied().collect()
+}
+
fn cfa(code: u16) -> Cfa {
- let s = match code {
+ Cfa::bayer_static(match code {
2 => "GRBG",
3 => "GBRG",
4 => "BGGR",
_ => "RGGB",
- };
- Cfa::bayer_static(s)
+ })
}
pub(crate) fn decode(bytes: &[u8], mode: Mode) -> Result {
let tiff = Tiff::parse(bytes)?;
- let ifd0 = &tiff.ifds[0];
- let get = |tag: u16| ifd0.u64(tag).map(|v| v as usize);
+ let ifd0 = tiff.ifds.first().ok_or_else(|| RawError::Corrupt("RW2 without IFD0".into()))?;
+ let get = |tag: u16| ifd0.u64(tag).and_then(|v| usize::try_from(v).ok());
let (w, h) = (get(SENSOR_WIDTH).unwrap_or(0), get(SENSOR_HEIGHT).unwrap_or(0));
- let bits = get(BITS).unwrap_or(12) as u32;
- let format = get(RAW_FORMAT).unwrap_or(0);
if w == 0 || h == 0 {
return Err(RawError::Corrupt("RW2 without sensor size".into()));
}
- let n = w.checked_mul(h).filter(|n| *n <= crate::MAX_SAMPLES).ok_or(RawError::Limit("image too large"))?;
- if format != 5 {
- return Err(RawError::Unsupported(format!("Panasonic RW2 raw format {format} (quantised)")));
- }
- let (per, _) = layout(bits).ok_or_else(|| RawError::Unsupported(format!("RW2 packed {bits}-bit data")))?;
- let off = get(RAW_OFFSET).or_else(|| get(t::STRIP_OFFSETS)).ok_or_else(|| RawError::Corrupt("RW2 without raw data offset".into()))?;
- let src = bytes.get(off..).ok_or_else(|| RawError::Corrupt("RW2 raw data outside file".into()))?;
- let need = n.div_ceil(per) * 16;
- if src.len() < need {
- return Err(RawError::Corrupt(format!("RW2 raw data truncated ({} of {need} bytes)", src.len())));
- }
+ let n = w.checked_mul(h).filter(|n| *n <= MAX_SAMPLES).ok_or(RawError::Limit("image too large"))?;
+ let bits = ifd0.u32(BITS).unwrap_or(12);
+ let encoding = encoding(ifd0, bits)?;
+ let src = || {
+ let off = get(RAW_OFFSET).or_else(|| get(t::STRIP_OFFSETS)).ok_or_else(|| RawError::Corrupt("RW2 without raw data offset".into()))?;
+ bytes.get(off..).ok_or_else(|| RawError::Corrupt("RW2 raw data outside file".into()))
+ };
+ let truncated = |have: usize, need: usize| RawError::Corrupt(format!("RW2 raw data truncated ({have} of {need} bytes)"));
let mut data = Vec::new();
- if mode == Mode::Full {
- let stream = unrotate(&src[..(need.div_ceil(CHUNK) * CHUNK).min(src.len())]);
- data = vec![0u16; n];
- unpack(&stream, bits, &mut data)?;
+ match encoding {
+ Encoding::Words => {
+ let src = src()?;
+ let need = n * 2;
+ if src.len() < need {
+ return Err(truncated(src.len(), need));
+ }
+ if mode == Mode::Full {
+ data = vec![0u16; n];
+ decode_words(src, 16u32.saturating_sub(bits), &mut data);
+ }
+ }
+ Encoding::Blocks { kind, rotated } => {
+ let src = src()?;
+ let need = n.div_ceil(kind.per()) * 16;
+ if src.len() < need {
+ return Err(truncated(src.len(), need));
+ }
+ if mode == Mode::Full {
+ data = vec![0u16; n];
+ decode_blocks(src, kind, rotated, &mut data);
+ }
+ }
+ Encoding::Strips => {
+ let (strips, codes) = format8(ifd0, tiff.order, bytes, w, h)?;
+ if mode == Mode::Full {
+ data = vec![0u16; n];
+ decode_strips(&strips, &codes, w, &mut data)?;
+ }
+ }
}
let active = match (get(BORDER_TOP), get(BORDER_LEFT), get(BORDER_BOTTOM), get(BORDER_RIGHT)) {
@@ -130,26 +530,45 @@ pub(crate) fn decode(bytes: &[u8], mode: Mode) -> Result {
_ => Rect::new(0, 0, w, h),
};
let active = if active.width == 0 || active.height == 0 { Rect::new(0, 0, w, h) } else { active };
- let cfa = cfa(ifd0.u16(CFA_PATTERN).unwrap_or(1));
- let black = match BLACK_RGB.map(|tag| ifd0.f64(tag)) {
- [Some(r), Some(g), Some(b)] => {
- // per CFA position, anchored at the active area
- let a = cfa.shifted(active.x, active.y);
- let values = a.pattern.iter().map(|&c| [r, g, b][c as usize] as f32).collect();
- BlackLevel { repeat_rows: 2, repeat_cols: 2, values, ..Default::default() }
+ let crop = match CROP.map(get) {
+ [Some(tp), Some(l), Some(b), Some(r)]
+ if b > tp && r > l && l >= active.x && tp >= active.y && r <= active.x + active.width && b <= active.y + active.height =>
+ {
+ Rect::new(l - active.x, tp - active.y, r - l, b - tp)
}
- _ => BlackLevel::uniform(0.0),
+ _ => Rect::new(0, 0, active.width, active.height),
};
- let white = ifd0.f64(LINEARITY_LIMIT).filter(|v| *v > 0.0).map(|v| v as f32).unwrap_or(((1u32 << bits) - 1) as f32);
+ if !data.is_empty() {
+ fill_defects(&mut data, w, active);
+ }
+
+ let cfa = cfa(ifd0.u16(CFA_PATTERN).unwrap_or(1));
+ let tags = BLACK_RGB.map(|tag| ifd0.f64(tag).filter(|v| v.is_finite() && *v >= 0.0).unwrap_or(0.0) as f32);
+ let offset = match encoding {
+ Encoding::Blocks { kind: Block::Format4 | Block::Format6 { .. }, .. } => CODEC_BLACK_OFFSET,
+ _ => 0.0,
+ };
+ let rgb = if tags.iter().all(|&v| v == 0.0) { [DEFAULT_BLACK; 3] } else { tags.map(|v| v + offset) };
+ // per CFA position, anchored at the active area
+ let values = cfa.shifted(active.x, active.y).pattern.iter().map(|&c| rgb[c as usize]).collect();
+ let black = BlackLevel { repeat_rows: 2, repeat_cols: 2, values, ..Default::default() };
+ let full = ((1u32 << bits.clamp(1, 16)) - 1) as f32;
+ let white = if data.is_empty() {
+ // headers only (white isn't part of `RawInfo`): the recorded limit as a placeholder
+ LINEARITY_LIMIT.iter().filter_map(|&tag| ifd0.f64(tag)).map(|v| v as f32).filter(|v| *v > 0.0 && *v <= full).fold(full, f32::min)
+ } else {
+ white_from_data(&active_rows(&data, w, active), bits)
+ };
+ let white = if white > black.mean() { white } else { full };
let wb = match WB_RGB.map(|tag| ifd0.f64(tag)) {
[Some(r), Some(g), Some(b)] if r > 0.0 && g > 0.0 && b > 0.0 => Some([(r / g) as f32, 1.0, (b / g) as f32]),
_ => None,
};
let mut metadata = lightcraft_meta::from_tiff(&tiff);
- metadata.width = Some(active.width as u32);
- metadata.height = Some(active.height as u32);
+ metadata.width = Some(crop.width as u32);
+ metadata.height = Some(crop.height as u32);
if metadata.iso.is_none() {
- metadata.iso = ifd0.u32(ISO).filter(|v| *v > 0);
+ metadata.iso = ifd0.u32(ISO).or_else(|| ifd0.u32(ISO_32)).filter(|v| *v > 0);
}
let img = RawImage {
format: RawFormat::Rw2,
@@ -162,7 +581,7 @@ pub(crate) fn decode(bytes: &[u8], mode: Mode) -> Result {
black,
white: vec![white],
active_area: active,
- crop: Rect::new(0, 0, active.width, active.height),
+ crop,
orientation: Orientation::from_exif(ifd0.u16(t::ORIENTATION).unwrap_or(1)),
color: ColorData::default(),
wb_multipliers: wb,
@@ -177,79 +596,483 @@ pub(crate) fn decode(bytes: &[u8], mode: Mode) -> Result {
#[cfg(test)]
mod tests {
use super::*;
- use lightcraft_tiff::{ByteOrder, IfdBuilder, TiffWriter, Value};
+ use lightcraft_tiff::{IfdBuilder, TiffWriter};
- /// Inverse of the decoder: pack, then rotate every whole chunk.
- pub(crate) fn encode(px: &[u16], bits: u32) -> Vec {
- let (per, start) = layout(bits).unwrap();
- let mut stream = Vec::new();
- for blk in px.chunks(per) {
- let mut word = 0u128;
- for (i, &v) in blk.iter().enumerate() {
- word |= (v as u128) << (start + i as u32 * bits);
- }
- stream.extend_from_slice(&word.to_le_bytes());
+ /// Writes bit fields into a 16-byte block from bit 127 down (the inverse of [`Fields`]).
+ struct FieldWriter {
+ word: u128,
+ left: u32,
+ }
+
+ impl FieldWriter {
+ fn new() -> Self {
+ FieldWriter { word: 0, left: 128 }
}
- stream.resize(stream.len().div_ceil(CHUNK) * CHUNK, 0);
- let mut out = Vec::with_capacity(stream.len());
- for c in stream.chunks(CHUNK) {
- // logical = file[ROTATE..] ++ file[..ROTATE] => file = logical[CHUNK-ROTATE..] ++ logical[..CHUNK-ROTATE]
- out.extend_from_slice(&c[CHUNK - ROTATE..]);
- out.extend_from_slice(&c[..CHUNK - ROTATE]);
+ fn put(&mut self, n: u32, v: u32) -> &mut Self {
+ assert!(v < 1 << n, "{v} doesn't fit {n} bits");
+ self.left -= n;
+ self.word |= (v as u128) << self.left;
+ self
}
+ fn block(&self) -> [u8; 16] {
+ self.word.to_le_bytes()
+ }
+ }
+
+ /// A seed of format 4: 8 bits `h`, 4 bits `l`.
+ fn seed(f: &mut FieldWriter, v: u32) {
+ f.put(8, v >> 4).put(4, v & 15);
+ }
+
+ fn decode4(b: [u8; 16]) -> [u16; 14] {
+ let mut out = [0u16; 14];
+ format4(&b, &mut out);
out
}
- pub(crate) fn rw2(w: u16, h: u16, bits: u16, format: u16, payload: &[u8]) -> Vec {
+ #[test]
+ fn format4_known_blocks() {
+ // every branch: differences at each scale, a zero code, absolute scale-3 codes keeping the low bits
+ let mut f = FieldWriter::new();
+ seed(&mut f, 1000);
+ seed(&mut f, 2000);
+ f.put(2, 0).put(8, 130).put(8, 120).put(8, 0); // 1002, 1992, 1002 (kept)
+ f.put(2, 1).put(8, 129).put(8, 100).put(8, 128); // 1994, 946, 1994
+ f.put(2, 2).put(8, 200).put(8, 64).put(8, 131); // 1234, 1738, 1246
+ f.put(2, 3).put(8, 50).put(8, 255).put(8, 0); // 50·16 + 1738 % 16 = 810, 4080 + 14 = 4094, 810 (kept)
+ assert_eq!(f.left, 0);
+ assert_eq!(decode4(f.block()), [1000, 2000, 1002, 1992, 1002, 1994, 946, 1994, 1234, 1738, 1246, 810, 4094, 810]);
+
+ // a deferred seed (defect marker, 8 bits only) and absolute codes because the previous value is too small
+ let mut f = FieldWriter::new();
+ f.put(8, 0); // pixel 0: defect; the next even pixel brings the seed
+ seed(&mut f, 325);
+ f.put(2, 1);
+ seed(&mut f, 481); // pixel 2: the even seed
+ f.put(8, 140).put(8, 10); // 325 + 24 = 349, 481 − 236 = 245
+ f.put(2, 2).put(8, 100).put(8, 129).put(8, 129); // < 512: 400 + 349 % 4 = 401, 516 + 245 % 4 = 517, 517
+ f.put(2, 0).put(8, 127).put(8, 0).put(8, 255); // 516, 517 (kept), 643
+ f.put(2, 0).put(8, 1).put(8, 128).put(8, 128); // 390, 643, 390
+ assert_eq!(f.left, 0);
+ assert_eq!(decode4(f.block()), [0, 325, 481, 349, 245, 401, 517, 517, 516, 517, 643, 390, 643, 390]);
+
+ // a colour without a seed until pixel 12, which then always reads its 4 low bits
+ let mut f = FieldWriter::new();
+ f.put(8, 0);
+ seed(&mut f, 1600);
+ for g in 0..4 {
+ f.put(2, 0);
+ for k in 0..3 {
+ let i = 2 + 3 * g + k;
+ if i % 2 == 1 {
+ f.put(8, 128);
+ } else if i < 12 {
+ f.put(8, 0);
+ } else {
+ f.put(8, 0).put(4, 7);
+ }
+ }
+ }
+ assert_eq!(f.left, 0);
+ assert_eq!(decode4(f.block()), [0, 1600, 0, 1600, 0, 1600, 0, 1600, 0, 1600, 0, 1600, 7, 1600]);
+ }
+
+ #[test]
+ fn format6_known_blocks() {
+ let mut f = FieldWriter::new();
+ f.put(14, 5000).put(14, 6000);
+ f.put(2, 0).put(10, 520).put(10, 500).put(10, 512); // 5008, 5988, 5008
+ f.put(2, 3).put(10, 1023).put(10, 300).put(10, 0); // 16368 + 5988 % 16, 4800 + 5008 % 16, kept
+ f.put(2, 1).put(10, 600).put(10, 400).put(10, 10); // 4976, 16148, 3972
+ let mut out = [0u16; 11];
+ format6(&f.block(), true, &mut out);
+ assert_eq!(out, [5000, 6000, 5008, 5988, 5008, 16372, 4800, 16372, 4976, 16148, 3972]);
+
+ let mut f = FieldWriter::new();
+ f.put(12, 300).put(12, 4000);
+ f.put(2, 2).put(8, 140).put(8, 100).put(8, 130); // 300 < 512: 560 + 0; 3888; 568
+ f.put(2, 0).put(8, 129).put(8, 127).put(8, 128); // 3889, 567, 3889
+ f.put(2, 3).put(8, 10).put(8, 255).put(8, 20); // 160 + 7, 4080 + 1, 320 + 7
+ f.put(2, 1).put(8, 128).put(8, 130).put(8, 0); // 4081, 331, 4081 (kept)
+ assert_eq!(f.left, 0);
+ let mut out = [0u16; 14];
+ format6(&f.block(), false, &mut out);
+ assert_eq!(out, [300, 4000, 560, 3888, 568, 3889, 567, 3889, 167, 4081, 327, 4081, 331, 4081]);
+ }
+
+ /// A camera-like format 4/6 encoder: per group the smallest scale that codes the targets exactly, else scale 3.
+ /// Returns the block and what a decoder must reconstruct.
+ fn encode_block(target: &[u16], fmt6_bits: Option) -> ([u8; 16], Vec) {
+ let (seed_bits, code_bits, bias) = match fmt6_bits {
+ Some(14) => (14, 10, 512),
+ Some(_) => (12, 8, 128),
+ None => (12, 8, 128),
+ };
+ let n = target.len();
+ let mut f = FieldWriter::new();
+ let mut prev = [target[0] as i32, target[1] as i32];
+ let mut out = vec![target[0], target[1]];
+ for &s in &target[..2] {
+ if fmt6_bits.is_some() {
+ f.put(seed_bits, s as u32);
+ } else {
+ seed(&mut f, s as u32);
+ }
+ }
+ for g in 0..(n - 2) / 3 {
+ let idx: Vec = (0..3).map(|k| 2 + 3 * g + k).collect();
+ let fits = |s: usize| -> Option> {
+ let (m, absolute) = STEPS[s];
+ let mut p = prev;
+ let mut codes = vec![];
+ for &i in &idx {
+ let (v, c) = (target[i] as i32, i & 1);
+ let j = if !absolute && p[c] >= bias * m {
+ let d = v - p[c];
+ (d % m == 0 && (d / m).abs() < bias).then_some(d / m + bias)?
+ } else {
+ // nearest step, as a camera would
+ let j = (v - (p[c] & (m - 1)) + m / 2) / m;
+ (absolute || (v - (p[c] & (m - 1))) % m == 0).then_some(j.clamp(1, (1 << code_bits) - 1))?
+ };
+ p[c] = next(p[c], j, (m, absolute), bias);
+ codes.push(j);
+ }
+ Some(codes)
+ };
+ let s = (0..4).find(|&s| fits(s).is_some()).unwrap_or(3);
+ let codes = fits(s).unwrap();
+ f.put(2, s as u32);
+ for (k, &i) in idx.iter().enumerate() {
+ f.put(code_bits, codes[k] as u32);
+ prev[i & 1] = next(prev[i & 1], codes[k], STEPS[s], bias);
+ out.push(prev[i & 1] as u16);
+ }
+ }
+ (f.block(), out)
+ }
+
+ /// A smooth synthetic Bayer-like image with some sharp edges, values `lo..hi`.
+ fn image(w: usize, h: usize, lo: u32, hi: u32) -> Vec {
+ (0..w * h)
+ .map(|i| {
+ let (x, y) = (i % w, i / w);
+ let smooth = lo + (hi - lo) * ((x * 7 + y * 3) % 997) as u32 / 997 / 2;
+ let edge = if (x / 37 + y / 23) % 3 == 0 { (hi - lo) / 3 } else { 0 };
+ (smooth + edge + (x % 2 * 40) as u32).min(hi) as u16
+ })
+ .collect()
+ }
+
+ /// Rotate every whole 0x4000-byte chunk the way formats 4 and 5 store them.
+ fn rotate(mut stream: Vec) -> Vec {
+ stream.resize(stream.len().div_ceil(CHUNK) * CHUNK, 0);
+ stream.chunks(CHUNK).flat_map(|c| c[CHUNK - ROTATE..].iter().chain(&c[..CHUNK - ROTATE]).copied().collect::>()).collect()
+ }
+
+ /// Pack samples LSB-first into 16-byte blocks (formats 2, 5 and 7).
+ fn pack(px: &[u16], bits: u32) -> Vec {
+ let per = (128 / bits) as usize;
+ px.chunks(per).flat_map(|blk| blk.iter().enumerate().fold(0u128, |w, (i, &v)| w | (v as u128) << (i as u32 * bits)).to_le_bytes()).collect()
+ }
+
+ /// A minimal RW2: IFD0 with `tags` plus sensor size, borders (2 rows, 4 columns at the top/left) and Make, the
+ /// payload at the end. `at_payload` adds tags that hold the payload's offset.
+ fn file(w: u16, h: u16, tags: &[(u16, Value)], payload: &[u8], at_payload: &dyn Fn(u32) -> Vec<(u16, Value)>) -> Vec {
let build = |off: u32| {
let mut ifd = IfdBuilder::new();
for (tag, v) in [(SENSOR_WIDTH, w), (SENSOR_HEIGHT, h), (BORDER_TOP, 2), (BORDER_LEFT, 4), (BORDER_BOTTOM, h), (BORDER_RIGHT, w)] {
ifd.set(tag, Value::Short(vec![v]));
}
- ifd.set(CFA_PATTERN, Value::Short(vec![1]));
- ifd.set(BITS, Value::Short(vec![bits]));
- ifd.set(RAW_FORMAT, Value::Short(vec![format]));
- ifd.set(LINEARITY_LIMIT, Value::Short(vec![(1 << bits) - 1]));
- for (i, tag) in BLACK_RGB.iter().enumerate() {
- ifd.set(*tag, Value::Short(vec![100 + i as u16]));
- }
- for (tag, v) in WB_RGB.iter().zip([512u16, 256, 384]) {
- ifd.set(*tag, Value::Short(vec![v]));
- }
ifd.set(t::MAKE, Value::Ascii("Panasonic".into()));
- ifd.set(RAW_OFFSET, Value::Long(vec![off]));
+ for (tag, v) in tags.iter().cloned().chain(at_payload(off)) {
+ ifd.set(tag, v);
+ }
let mut b = TiffWriter::new(ByteOrder::Little, false).write(&[ifd]).unwrap();
b[2] = 0x55;
b
};
- let head = build(0);
- let off = head.len().div_ceil(16) * 16;
+ let off = build(0).len().div_ceil(16) * 16;
let mut b = build(off as u32);
b.resize(off, 0);
b.extend_from_slice(payload);
b
}
+ fn short(tag: u16, v: u16) -> (u16, Value) {
+ (tag, Value::Short(vec![v]))
+ }
+
+ /// Tags of a format-`format`, `bits`-bit file with black levels `black` and the payload at `RawDataOffset`.
+ fn rw2(w: u16, h: u16, bits: u16, format: u16, black: u16, payload: &[u8]) -> Vec {
+ let mut tags = vec![short(CFA_PATTERN, 1), short(BITS, bits), short(RAW_FORMAT, format)];
+ tags.extend(BLACK_RGB.iter().enumerate().map(|(i, &tag)| short(tag, black + i as u16)));
+ tags.extend(WB_RGB.iter().zip([512, 256, 384]).map(|(&tag, v)| short(tag, v)));
+ file(w, h, &tags, payload, &|off| vec![(RAW_OFFSET, Value::Long(vec![off]))])
+ }
+
+ fn samples(r: &RawImage) -> &[u16] {
+ match &r.data {
+ RawData::U16(d) => d,
+ RawData::F32(_) => panic!("float samples"),
+ }
+ }
+
#[test]
- fn packed_12_and_14_bit() {
- let (w, h) = (90usize, 200usize); // > one chunk of data
- for bits in [12u32, 14] {
- let px: Vec = (0..w * h).map(|i| ((i * 2654435761usize) >> 7) as u16 & ((1 << bits) - 1)).collect();
- let bytes = rw2(w as u16, h as u16, bits as u16, 5, &encode(&px, bits));
- assert_eq!(crate::probe(&bytes), Some(RawFormat::Rw2));
+ fn format4_and_6_round_trip_through_files() {
+ // format 4: 14 pixels per block, rotated chunks, blocks continuous across rows (here 70 = 5 blocks per row)
+ let (w, h) = (70usize, 300usize);
+ let target = image(w, h, 200, 4000);
+ let (mut stream, mut want) = (vec![], vec![]);
+ for px in target.chunks(14) {
+ let (b, out) = encode_block(px, None);
+ stream.extend_from_slice(&b);
+ want.extend(out);
+ }
+ assert!(want.iter().zip(&target).all(|(a, b)| a.abs_diff(*b) <= 8), "the encoder is too lossy for the test image");
+ let bytes = rw2(w as u16, h as u16, 12, 4, 128, &rotate(stream));
+ let r = crate::decode(&bytes).unwrap();
+ assert_eq!(samples(&r), want);
+ assert_eq!(crate::probe_info(&bytes).unwrap(), r.info());
+ assert_eq!(r.black.values, vec![143.0; 4].into_iter().zip([0., 1., 1., 2.]).map(|(b, o)| b + o).collect::>());
+
+ // format 6: not rotated; 11 (14-bit) or 14 (12-bit) pixels per block
+ for (bits, per, top) in [(14u16, 11usize, 16000u32), (12, 14, 4000)] {
+ let w = per * 6;
+ let target = image(w, 40, 600, top);
+ let (mut stream, mut want) = (vec![], vec![]);
+ for px in target.chunks(per) {
+ let (b, out) = encode_block(px, Some(bits as u32));
+ stream.extend_from_slice(&b);
+ want.extend(out);
+ }
+ let bytes = rw2(w as u16, 40, bits, 6, 512, &stream);
let r = crate::decode(&bytes).unwrap_or_else(|e| panic!("{bits}: {e}"));
+ assert_eq!(samples(&r), want, "{bits}-bit");
+ assert_eq!(r.black.at(0, 0, 0, 1), 527.0, "format 6 stores samples 15 above the black level");
+ }
+ }
+
+ #[test]
+ fn packed_and_word_formats() {
+ let (w, h) = (90usize, 200usize); // > one chunk of data
+ for (format, bits, rotated) in [(5u16, 12u32, true), (5, 14, true), (7, 14, false), (2, 12, false)] {
+ let px: Vec = (0..w * h).map(|i| (((i * 2654435761usize) >> 7) as u16 & ((1 << bits) - 1)).max(1)).collect();
+ let stream = pack(&px, bits);
+ let bytes = rw2(w as u16, h as u16, bits as u16, format, 100, &if rotated { rotate(stream) } else { stream });
+ assert_eq!(crate::probe(&bytes), Some(RawFormat::Rw2));
+ let r = crate::decode(&bytes).unwrap_or_else(|e| panic!("format {format}, {bits}-bit: {e}"));
assert_eq!(crate::probe_info(&bytes).unwrap(), r.info());
- assert_eq!(r.data, RawData::U16(px), "{bits}-bit");
+ assert_eq!(samples(&r), px, "format {format}, {bits}-bit");
assert_eq!(r.active_area, Rect::new(4, 2, w - 4, h - 2));
assert_eq!(r.cfa.as_ref().unwrap().name(), "RGGB");
- assert_eq!(r.black.at(0, 0, 0, 1), 100.0);
- assert_eq!(r.black.at(1, 0, 0, 1), 101.0);
- assert_eq!(r.black.at(1, 1, 0, 1), 102.0);
+ // packed formats store samples at the recorded black level, per colour, anchored at the active area
+ assert_eq!([r.black.at(0, 0, 0, 1), r.black.at(1, 0, 0, 1), r.black.at(1, 1, 0, 1)], [100.0, 101.0, 102.0]);
assert_eq!(r.wb_multipliers, Some([2.0, 1.0, 1.5]));
}
- let bytes = rw2(28, 2, 12, 4, &[0; 64]);
+ // the oldest files: 16-bit words with the sample left-justified, no black level recorded
+ for (compression, bits) in [(34828u16, 12u32), (34830, 14)] {
+ let px: Vec = (0..w * h).map(|i| 100 + (i * 7 % 3000) as u16).collect();
+ let payload: Vec = px.iter().flat_map(|&v| (v << (16 - bits)).to_le_bytes()).collect();
+ let tags = [short(BITS, bits as u16), short(COMPRESSION, compression), short(CFA_PATTERN, 4)];
+ let bytes = file(w as u16, h as u16, &tags, &payload, &|off| vec![(t::STRIP_OFFSETS, Value::Long(vec![off]))]);
+ let r = crate::decode(&bytes).unwrap_or_else(|e| panic!("{compression}: {e}"));
+ assert_eq!(samples(&r), px);
+ assert_eq!(r.black.mean(), DEFAULT_BLACK);
+ assert_eq!(r.white_at(0), ((1 << bits) - 1) as f32, "no saturation plateau: the full range");
+ }
+ }
+
+ #[test]
+ fn levels_crop_defects_and_cfa() {
+ let (w, h) = (90usize, 60usize);
+ let mut px: Vec = (0..w * h).map(|i| 300 + (i % 7) as u16).collect();
+ // a saturation plateau sets the white level; zeros inside the active area are defects
+ px[10 * w..20 * w].fill(3900);
+ px[30 * w + 40] = 0;
+ px[1] = 0; // in the border: left alone
+ let mut tags = vec![short(CFA_PATTERN, 2), short(BITS, 12), short(RAW_FORMAT, 5)];
+ tags.extend([0x002f, 0x0030, 0x0031, 0x0032].iter().zip([12u16, 10, 52, 80]).map(|(&tag, v)| short(tag, v)));
+ tags.extend(LINEARITY_LIMIT.map(|tag| short(tag, 2000))); // recorded limits are not used
+ let bytes = file(w as u16, h as u16, &tags, &rotate(pack(&px, 12)), &|off| vec![(RAW_OFFSET, Value::Long(vec![off]))]);
+ let r = crate::decode(&bytes).unwrap();
+ let d = samples(&r);
+ assert_eq!(d[30 * w + 40], ((px[30 * w + 38] as u32 + px[30 * w + 42] as u32 + px[28 * w + 40] as u32 + px[32 * w + 40] as u32) / 4) as u16);
+ assert_eq!(d[1], 0);
+ assert!((3800.0..=3900.0).contains(&r.white_at(0)), "{}", r.white_at(0));
+ assert_eq!(r.black.mean(), DEFAULT_BLACK, "no black recorded");
+ assert_eq!(r.crop, Rect::new(6, 10, 70, 40), "crop tags are sensor coordinates; the crop is relative to the active area");
+ assert_eq!((r.metadata.width, r.metadata.height), (Some(70), Some(40)));
+ // the CFA tag describes sample (0, 0) of the data, not the active area's origin
+ assert_eq!(r.cfa.as_ref().unwrap().name(), "GRBG");
+ let mut tags = vec![short(CFA_PATTERN, 2), short(BITS, 12), short(RAW_FORMAT, 5)];
+ tags.extend([0x002f, 0x0030, 0x0031, 0x0032].map(|tag| short(tag, 0))); // some bodies write zeros
+ let bytes = file(w as u16, h as u16, &tags, &rotate(pack(&px, 12)), &|off| vec![(RAW_OFFSET, Value::Long(vec![off]))]);
+ assert_eq!(crate::decode(&bytes).unwrap().crop, Rect::new(0, 0, w - 4, h - 2));
+ }
+
+ /// Format 8 code tables as two cameras write them (16-bit DC-GH6; 14-bit DC-S5M2, whose codes for categories 15
+ /// and 16 start with the 8-bit code of category 14 and are unreachable).
+ const TABLE16: [(u32, u32); 17] = [
+ (10, 1022),
+ (11, 2046),
+ (8, 254),
+ (9, 510),
+ (7, 126),
+ (4, 14),
+ (4, 12),
+ (3, 4),
+ (3, 2),
+ (2, 0),
+ (3, 3),
+ (3, 5),
+ (4, 13),
+ (5, 30),
+ (6, 62),
+ (12, 4094),
+ (12, 4095),
+ ];
+ const TABLE14: [(u32, u32); 17] = [
+ (6, 62),
+ (7, 126),
+ (6, 61),
+ (5, 28),
+ (4, 12),
+ (3, 4),
+ (3, 2),
+ (2, 0),
+ (3, 3),
+ (3, 5),
+ (4, 13),
+ (5, 29),
+ (6, 60),
+ (8, 254),
+ (8, 255),
+ (12, 4094),
+ (12, 4095),
+ ];
+
+ /// Encode a strip (the inverse of [`decode_strip`]): returns the stored bytes and the number of bits.
+ fn encode_strip(px: &[u16], w: usize, h: usize, table: &[(u32, u32); 17], padding_rows: usize) -> (Vec, u64) {
+ let mut bits: Vec = vec![];
+ let mut put = |n: u32, v: u32| bits.extend((0..n).rev().map(|i| v >> i & 1 == 1));
+ let mut first = [0i32; 4];
+ let at = |x: usize, y: usize| if y < h { px[y * w + x] as i32 } else { 1000 };
+ for pair in 0..(h / 2 + padding_rows) {
+ let mut cell = first;
+ for cx in 0..w / 2 {
+ for (k, p) in cell.iter_mut().enumerate() {
+ let v = at(2 * cx + k % 2, 2 * pair + k / 2);
+ let d = v - *p;
+ let c = 32 - d.unsigned_abs().leading_zeros();
+ let (len, code) = table[c as usize];
+ put(len, code);
+ put(c, if d >= 0 { d as u32 } else { (d + (1 << c) - 1) as u32 });
+ *p = v;
+ }
+ if cx == 0 {
+ first = cell;
+ }
+ }
+ }
+ let n = bits.len() as u64;
+ // stored with each byte's bits taken from bit 0 up
+ (bits.chunks(8).map(|b| b.iter().enumerate().fold(0u8, |a, (i, &x)| a | (x as u8) << i)).collect(), n)
+ }
+
+ /// A format-8 file with the given strip widths; `tweak` edits the tags before writing.
+ fn format8_file(w: usize, h: usize, px: &[u16], widths: &[usize], table: &[(u32, u32); 17], tweak: &dyn Fn(&mut Vec<(u16, Value)>)) -> Vec {
+ let mut payload = vec![];
+ let mut strips = vec![]; // (offset in payload, bits, left, width)
+ let mut left = 0;
+ for &sw in widths {
+ let sub: Vec = (0..h).flat_map(|y| px[y * w + left..y * w + left + sw].to_vec()).collect();
+ let (data, bits) = encode_strip(&sub, sw, h, table, 2);
+ strips.push((payload.len(), bits, left, sw));
+ payload.extend(data);
+ payload.resize(payload.len().div_ceil(16) * 16, 0);
+ left += sw;
+ }
+ let u16s = |v: Vec| undefined(&v);
+ let wide = |v: Vec| u16s(std::iter::once(v.len() as u16).chain(v.iter().flat_map(|&x| [x as u16, (x >> 16) as u16])).collect());
+ let narrow = |v: Vec| u16s(std::iter::once(v.len() as u16).chain(v.iter().map(|&x| x as u16)).collect());
+ let mut tags = vec![short(CFA_PATTERN, 1), short(BITS, 14), short(RAW_FORMAT, 8)];
+ tags.extend(BLACK_RGB.map(|tag| short(tag, 512)));
+ tags.push((HUFFMAN_CODES, u16s(std::iter::once(17).chain(table.iter().flat_map(|&(l, c)| [l as u16, c as u16])).collect())));
+ for (tag, v) in FORMAT8_FIXED {
+ tags.push(if v.len() == 1 { short(tag, v[0]) } else { (tag, u16s(v.to_vec())) });
+ }
+ tags.push((STRIP_LEFT, wide(strips.iter().map(|s| s.2 as u64).collect())));
+ tags.push((STRIP_BITS, wide(strips.iter().map(|s| s.1).collect())));
+ tags.push((STRIP_WIDTHS, narrow(strips.iter().map(|s| s.3 as u64).collect())));
+ tags.push((STRIP_HEIGHTS, narrow(vec![h as u64; strips.len()])));
+ tweak(&mut tags);
+ let offsets: Vec = strips.iter().map(|s| s.0).collect();
+ file(w as u16, h as u16, &tags, &payload, &|off| {
+ vec![(RAW_OFFSET, Value::Long(vec![off])), (STRIP_OFFSETS, wide(offsets.iter().map(|&o| off as u64 + o as u64).collect()))]
+ })
+ }
+
+ #[test]
+ fn format8_strips_round_trip() {
+ let (w, h) = (64usize, 30usize);
+ let px = image(w, h, 520, 16000);
+ for (table, widths) in [(&TABLE14, vec![64]), (&TABLE14, vec![30, 34]), (&TABLE16, vec![20, 22, 22])] {
+ let bytes = format8_file(w, h, &px, &widths, table, &|_| {});
+ let r = crate::decode(&bytes).unwrap_or_else(|e| panic!("{widths:?}: {e}"));
+ assert_eq!(samples(&r), px, "{widths:?}");
+ assert_eq!(crate::probe_info(&bytes).unwrap(), r.info());
+ assert_eq!(r.black.mean(), 512.0, "format 8 stores samples at the recorded black level");
+ }
+ }
+
+ fn replace(tags: &mut Vec<(u16, Value)>, tag: u16, v: Value) {
+ tags.retain(|(t, _)| *t != tag);
+ tags.push((tag, v));
+ }
+
+ fn undefined(v: &[u16]) -> Value {
+ Value::Undefined(v.iter().flat_map(|x| x.to_le_bytes()).collect())
+ }
+
+ #[test]
+ fn format8_rejects_what_it_cannot_decode() {
+ let (w, h) = (64usize, 30usize);
+ let px = image(w, h, 520, 16000);
+ let case = |tag: u16, v: Value| crate::decode(&format8_file(w, h, &px, &[32, 32], &TABLE14, &|t| replace(t, tag, v.clone())));
+ // an auxiliary tag with a value no sample has
+ assert!(matches!(case(0x003b, Value::Short(vec![4095])), Err(RawError::Unsupported(_))));
+ // strips that don't tile the width, or extend past the file
+ assert!(matches!(case(STRIP_WIDTHS, undefined(&[2, 32, 30])), Err(RawError::Corrupt(_))));
+ assert!(matches!(case(STRIP_BITS, undefined(&[2, 0, 256, 0, 256])), Err(RawError::Corrupt(_))));
+ // a strip whose stated length ends before its data does
+ assert!(matches!(case(STRIP_BITS, undefined(&[2, 64, 0, 64, 0])), Err(RawError::Corrupt(_))));
+ // code tables: two categories with the same code, a code longer than its length, missing entries
+ let table: Vec = std::iter::once(17).chain(TABLE14.iter().flat_map(|&(l, c)| [l as u16, c as u16])).collect();
+ let mut dup = table.clone();
+ dup[1..3].copy_from_slice(&[2, 0]);
+ assert!(matches!(case(HUFFMAN_CODES, undefined(&dup)), Err(RawError::Corrupt(_))));
+ let mut long = table.clone();
+ long[1..3].copy_from_slice(&[3, 9]);
+ assert!(matches!(case(HUFFMAN_CODES, undefined(&long)), Err(RawError::Corrupt(_))));
+ assert!(matches!(case(HUFFMAN_CODES, undefined(&table[..20])), Err(RawError::Corrupt(_))));
+ // the shadowed 12-bit codes of the 14-bit table are fine
+ assert!(case(HUFFMAN_CODES, undefined(&table)).is_ok());
+ }
+
+ #[test]
+ fn unsupported_and_truncated_files() {
+ let payload = [0u8; 64];
+ for (format, bits) in [(1u16, 12u16), (3, 12), (4, 14), (7, 12), (9, 12)] {
+ assert!(matches!(crate::decode(&rw2(28, 4, bits, format, 128, &payload)), Err(RawError::Unsupported(_))), "{format}/{bits}");
+ }
+ let tags = [short(BITS, 12), short(COMPRESSION, 34826)];
+ let bytes = file(28, 4, &tags, &payload, &|off| vec![(t::STRIP_OFFSETS, Value::Long(vec![off]))]);
assert!(matches!(crate::decode(&bytes), Err(RawError::Unsupported(_))));
- let bytes = rw2(280, 20, 12, 5, &[0; 64]);
- assert!(matches!(crate::decode(&bytes), Err(RawError::Corrupt(_))));
+ for format in [4u16, 5, 6] {
+ let bytes = rw2(280, 20, 12, format, 128, &payload);
+ assert!(matches!(crate::decode(&bytes), Err(RawError::Corrupt(_))), "{format}");
+ assert!(matches!(crate::probe_info(&bytes), Err(RawError::Corrupt(_))), "{format}");
+ }
}
}
diff --git a/crates/raw/tests/corpus.rs b/crates/raw/tests/corpus.rs
index ad187032..6d497b4e 100644
--- a/crates/raw/tests/corpus.rs
+++ b/crates/raw/tests/corpus.rs
@@ -19,9 +19,6 @@ const KNOWN_UNSUPPORTED: &[&str] = &[
"arw-sony-a7m4-lossless-m", // Sony lossless compressed M/S: subsampled (YCbCr) lossless JPEG
"arw-sony-a7m4-lossless-s", // "
"raf-fuji-xt20-compressed", // Fujifilm compressed RAF
- "rw2-panasonic-gh5.", // Panasonic raw format 4 (quantised)
- "rw2-panasonic-gx80", // "
- "rw2-panasonic-g9-b", // "
"orf-olympus-em", // Olympus compressed ORF
"sraw", // Canon sRAW / mRAW
];
@@ -46,11 +43,12 @@ fn corpus_raw_decodes() {
let t0 = Instant::now();
let preview = embedded_preview(&bytes);
let tp = t0.elapsed().as_secs_f64() * 1e3;
- // DNG previews are optional (and some carry only an uncompressed RGB thumbnail); vendor raws always embed a JPEG
+ // DNG previews are optional (and some carry only an uncompressed RGB thumbnail); vendor raws embed a JPEG,
+ // except the Panasonic `.RAW` files of 2005–2007
if let Some(p) = &preview {
assert!(p.starts_with(&[0xff, 0xd8]) && p.ends_with(&[0xff, 0xd9]), "{name}: preview is not a JPEG");
} else {
- assert_eq!(fmt, RawFormat::Dng, "{name}: no embedded preview");
+ assert!(fmt == RawFormat::Dng || name.starts_with("raw-panasonic-"), "{name}: no embedded preview");
}
let preview_kb = preview.as_ref().map_or(0, |p| p.len() / 1024);
let t1 = Instant::now();
@@ -289,3 +287,145 @@ fn corpus_sony_pre2017_colour_metadata() {
}
eprintln!("pre-2017 Sony ARW colour metadata checked on {seen} files");
}
+
+/// The embedded JPEG as linear RGB, reduced to `gw × gh` cells.
+fn jpeg_cells(jpeg: &[u8], gw: usize, gh: usize) -> Vec<[f64; 3]> {
+ use zune_core::{bytestream::ZCursor, colorspace::ColorSpace, options::DecoderOptions};
+ let opts = DecoderOptions::default().jpeg_set_out_colorspace(ColorSpace::RGB);
+ let mut d = zune_jpeg::JpegDecoder::new_with_options(ZCursor::new(jpeg), opts);
+ let px = d.decode().unwrap();
+ let info = d.info().unwrap();
+ let (w, h) = (info.width as usize, info.height as usize);
+ let (mut sum, mut n) = (vec![[0f64; 3]; gw * gh], vec![0f64; gw * gh]);
+ for y in 0..h {
+ for x in 0..w {
+ let i = (y * gh / h) * gw + x * gw / w;
+ for c in 0..3 {
+ let v = px[(y * w + x) * 3 + c] as f64 / 255.0;
+ sum[i][c] += if v <= 0.04045 { v / 12.92 } else { ((v + 0.055) / 1.055).powf(2.4) };
+ }
+ n[i] += 1.0;
+ }
+ }
+ sum.iter().zip(&n).map(|(s, n)| s.map(|v| v / n)).collect()
+}
+
+/// Black-subtracted means of the four sites of the 2×2 cells (anchored at sample (0, 0)) of the active area,
+/// reduced to `gw × gh` cells.
+fn raw_sites(img: &lightcraft_raw::RawImage, gw: usize, gh: usize) -> Vec<[f64; 4]> {
+ let lightcraft_raw::RawData::U16(d) = &img.data else { panic!("float data") };
+ let (w, a, black) = (img.width, img.active_area, img.black.mean() as f64);
+ let (x0, y0) = ((a.x + 1) & !1, (a.y + 1) & !1);
+ let (cw, ch) = ((a.x + a.width - x0) / 2, (a.y + a.height - y0) / 2);
+ let (mut sum, mut n) = (vec![[0f64; 4]; gw * gh], vec![0f64; gw * gh]);
+ for cy in 0..ch {
+ for cx in 0..cw {
+ let i = (cy * gh / ch) * gw + cx * gw / cw;
+ for site in 0..4 {
+ sum[i][site] += d[(y0 + 2 * cy + site / 2) * w + x0 + 2 * cx + site % 2] as f64 - black;
+ }
+ n[i] += 1.0;
+ }
+ }
+ sum.iter().zip(&n).map(|(s, n)| s.map(|v| v / n)).collect()
+}
+
+fn ranks(v: &[f64]) -> Vec {
+ let mut idx: Vec = (0..v.len()).collect();
+ idx.sort_by(|&a, &b| v[a].total_cmp(&v[b]));
+ let mut r = vec![0f64; v.len()];
+ for (rank, &i) in idx.iter().enumerate() {
+ r[i] = rank as f64;
+ }
+ r
+}
+
+/// How well the raw's chromaticities (log r/g, log b/g per cell) follow the JPEG's when the mosaic is read with
+/// `layout` (colour of each 2×2 site at sample (0, 0)).
+fn chroma_agreement(sites: &[[f64; 4]], jpeg: &[[f64; 3]], layout: &[u8]) -> f64 {
+ let (mut x, mut y) = ([vec![], vec![]], [vec![], vec![]]);
+ for (s, j) in sites.iter().zip(jpeg) {
+ let (mut rgb, mut n) = ([0f64; 3], [0f64; 3]);
+ for (site, &c) in layout.iter().enumerate() {
+ rgb[c as usize] += s[site];
+ n[c as usize] += 1.0;
+ }
+ let rgb = [rgb[0] / n[0], rgb[1] / n[1], rgb[2] / n[2]];
+ if rgb.iter().all(|v| *v > 1.0) && j.iter().all(|v| (0.002..0.9).contains(v)) {
+ for (k, c) in [0, 2].into_iter().enumerate() {
+ x[k].push((rgb[c] / rgb[1]).ln());
+ y[k].push((j[c] / j[1]).ln());
+ }
+ }
+ }
+ (correlation(&x[0], &y[0]) + correlation(&x[1], &y[1])) / 2.0
+}
+
+/// Panasonic RW2 / RWL / RAW (`crates/raw/src/vendor/rw2.rs`): one file per raw encoding (formats 2, 4, 5, 6, 7 and
+/// 8 — at 12, 14 and 16 bits — and the 16-bit words of the oldest bodies) decodes to the image of the camera's own
+/// JPEG, with the black levels, crops and colour-filter layouts the module docs derive and no defect markers left.
+#[test]
+fn corpus_panasonic_encodings() {
+ let dir = corpus_root().join("raw");
+ // (file, bits, mean black level, crop width × height, CFA layout)
+ let cases = [
+ ("raw-panasonic-fz50.raw", 12, 15.0, (3648, 2736), "BGGR"),
+ ("raw-panasonic-fz8.raw", 12, 15.0, (3072, 2304), "RGGB"),
+ ("rw2-panasonic-fz1000m2-4x3.rw2", 12, 143.0, (4864, 3648), "GBRG"),
+ ("rw2-panasonic-g9-b.rw2", 12, 143.0, (5184, 3888), "RGGB"),
+ ("rw2-panasonic-g9.rw2", 12, 127.5, (5184, 3888), "RGGB"),
+ ("rw2-panasonic-gh1.rw2", 12, 15.0, (4000, 3000), "GBRG"),
+ ("rw2-panasonic-gh5.rw2", 12, 143.5, (5184, 3888), "RGGB"),
+ ("rw2-panasonic-gh5m2.rw2", 12, 144.5, (5184, 3888), "RGGB"),
+ ("rw2-panasonic-gh5s.rw2", 14, 509.5, (3680, 2760), "RGGB"),
+ ("rw2-panasonic-gh6.rw2", 16, 2048.0, (5776, 4336), "RGGB"),
+ ("rw2-panasonic-gx80.rw2", 12, 143.0, (4592, 3448), "BGGR"),
+ ("rw2-panasonic-s1.rw2", 14, 526.0, (6000, 4000), "RGGB"),
+ ("rw2-panasonic-s5-format7.rw2", 14, 512.0, (6000, 4000), "RGGB"),
+ ("rw2-panasonic-s5m2.rw2", 14, 512.0, (6000, 4000), "RGGB"),
+ ("rw2-panasonic-s9.rw2", 12, 128.0, (6000, 4000), "RGGB"),
+ ("rwl-leica-dlux7.rwl", 12, 143.0, (4736, 3552), "BGGR"),
+ ];
+ let mut seen = 0;
+ for (name, bits, black, (cw, ch), cfa) in cases {
+ let Ok(bytes) = std::fs::read(dir.join(name)) else {
+ eprintln!("skip: {name} absent");
+ continue;
+ };
+ let img = decode(&bytes).unwrap_or_else(|e| panic!("{name}: {e}"));
+ assert_eq!(img.bits, bits, "{name}: bits");
+ assert_eq!(img.black.mean(), black, "{name}: black level");
+ assert_eq!((img.crop.width, img.crop.height), (cw, ch), "{name}: crop");
+ let layout = img.cfa.clone().unwrap_or_else(|| panic!("{name}: no CFA"));
+ assert_eq!(layout.name(), cfa, "{name}: CFA layout");
+ let lightcraft_raw::RawData::U16(d) = &img.data else { panic!("float data") };
+ let a = img.active_area;
+ let zeros =
+ (a.y..a.y + a.height).map(|y| d[y * img.width + a.x..y * img.width + a.x + a.width].iter().filter(|&&v| v == 0).count()).sum::();
+ assert_eq!(zeros, 0, "{name}: defect markers left in the active area");
+ let Some(jpeg) = embedded_preview(&bytes) else {
+ // the 2005–2007 `.RAW` files have no preview; their flat scenes make the green diagonal unambiguous
+ assert_eq!(green_on_main_diagonal(&img), cfa.starts_with('G'), "{name}: mosaic statistics disagree with {cfa}");
+ seen += 1;
+ continue;
+ };
+ // the camera's JPEG shows the whole active area (whatever the crop): the raw's green must rank like it, and
+ // its colours must follow the JPEG's best with the decoded colour-filter layout
+ let (gw, gh) = (24, 18);
+ let (sites, jpeg) = (raw_sites(&img, gw, gh), jpeg_cells(&jpeg, gw, gh));
+ let green: Vec = sites.iter().map(|s| (0..4).filter(|&i| layout.pattern[i] == 1).map(|i| s[i]).sum::()).collect();
+ let jg: Vec = jpeg.iter().map(|j| j[1]).collect();
+ let rho = correlation(&ranks(&green), &ranks(&jg));
+ let agree = |l: &str| chroma_agreement(&sites, &jpeg, &lightcraft_raw::Cfa::bayer(l).unwrap().pattern);
+ let best_other = ["RGGB", "GRBG", "GBRG", "BGGR"].into_iter().filter(|l| *l != cfa).map(agree).fold(f64::MIN, f64::max);
+ let own = agree(cfa);
+ eprintln!(
+ "{name:32} {}x{} {bits}-bit: green rank correlation {rho:.3}, colour agreement {own:.3} (others ≤ {best_other:.3})",
+ img.width, img.height
+ );
+ assert!(rho > 0.9, "{name}: rank correlation with the camera JPEG {rho}");
+ assert!(own > 0.5 && own > best_other, "{name}: colours follow the JPEG better with another layout ({own} vs {best_other})");
+ seen += 1;
+ }
+ eprintln!("Panasonic encodings checked on {seen} files");
+}
diff --git a/crates/raw/tests/dng.rs b/crates/raw/tests/dng.rs
index 5f0aad28..0d38ebb4 100644
--- a/crates/raw/tests/dng.rs
+++ b/crates/raw/tests/dng.rs
@@ -453,6 +453,22 @@ fn colour_transform_from_decoded_dng() {
assert!(o.iter().all(|v| (v - 1.0).abs() < 1e-9));
}
+/// Issue #212: a NEF converted to DNG came out dark and green. NEF white balance is vendor multipliers
+/// (no colour matrix, no `AsShotNeutral`); the DNG must carry them as its as-shot neutral.
+#[test]
+fn vendor_white_balance_survives_dng_conversion() {
+ let mut raw = synthetic(16, 16, Some(Cfa::bayer("RGGB").unwrap()), 1);
+ raw.color = ColorData::default();
+ raw.wb_multipliers = Some([2.25, 1.0, 1.7421875]);
+ let before = lightcraft_raw::color::camera_transform(&raw, lightcraft_raw::color::as_shot_white_xy(&raw));
+ let back = decode(&write_dng(&raw, &DngWriteOptions::default()).unwrap()).unwrap();
+ let after = lightcraft_raw::color::camera_transform(&back, lightcraft_raw::color::as_shot_white_xy(&back));
+ for c in 0..3 {
+ assert!((before.wb[c] - after.wb[c]).abs() < 1e-3, "{:?} vs {:?}", before.wb, after.wb);
+ }
+ assert!((after.wb[0] - 2.25).abs() < 1e-3 && (after.wb[2] - 1.7421875).abs() < 1e-3, "{:?}", after.wb);
+}
+
#[test]
fn rejects_unsupported_and_broken() {
let (w, h) = (8, 8);
diff --git a/crates/raw/tests/vendor_robust.rs b/crates/raw/tests/vendor_robust.rs
index 2db97c64..81a751c8 100644
--- a/crates/raw/tests/vendor_robust.rs
+++ b/crates/raw/tests/vendor_robust.rs
@@ -1,4 +1,4 @@
-//! Malformed vendor raw files (NEF, ARW, PEF, ORF, RW2, CR2, RAF) must decode to an error, never panic.
+//! Malformed vendor raw files (NEF, ARW, PEF, ORF, RW2 of every raw format, CR2, RAF) must decode to an error, never panic.
use lightcraft_tiff::tags as t;
use lightcraft_tiff::{ByteOrder, IfdBuilder, ImageData, TiffWriter, Value};
@@ -68,6 +68,66 @@ fn rw2() -> Vec {
b
}
+/// A Panasonic RW2 of raw format `format` (or, with `format` 0, the oldest 16-bit-word kind): sensor size, borders,
+/// bit depth, black levels, an in-camera crop and `n` bytes of payload at the end, at `RawDataOffset`. Format 8 also
+/// gets two strips covering the payload and a code table.
+fn rw2_of(format: u16, bits: u16, n: usize) -> Vec {
+ let u16s = |v: &[u16]| Value::Undefined(v.iter().flat_map(|x| x.to_le_bytes()).collect());
+ let build = |off: u32| {
+ let mut ifd = IfdBuilder::new();
+ for (tag, v) in [(2, W as u16), (3, H as u16), (4, 2), (5, 4), (6, H as u16), (7, W as u16), (9, 2), (0x0a, bits), (0x1c, 128), (0x1d, 128)] {
+ ifd.set(tag, Value::Short(vec![v]));
+ }
+ for (tag, v) in [(0x2f, 2u16), (0x30, 6), (0x31, H as u16 - 2), (0x32, W as u16 - 2)] {
+ ifd.set(tag, Value::Short(vec![v]));
+ }
+ if format == 0 {
+ ifd.set(0x000b, Value::Short(vec![34828]));
+ ifd.set(t::STRIP_OFFSETS, Value::Long(vec![off]));
+ } else {
+ ifd.set(0x002d, Value::Short(vec![format]));
+ ifd.set(0x0118, Value::Long(vec![off]));
+ }
+ if format == 8 {
+ let table = [
+ (6, 62),
+ (7, 126),
+ (6, 61),
+ (5, 28),
+ (4, 12),
+ (3, 4),
+ (3, 2),
+ (2, 0),
+ (3, 3),
+ (3, 5),
+ (4, 13),
+ (5, 29),
+ (6, 60),
+ (8, 254),
+ (8, 255),
+ (12, 4094),
+ (12, 4095),
+ ];
+ ifd.set(0x0040, u16s(&std::iter::once(17).chain(table.iter().flat_map(|&(l, c)| [l, c])).collect::>()));
+ let half = (n / 2) as u32;
+ let (a, b) = (off, off + half);
+ ifd.set(0x0044, u16s(&[2, a as u16, (a >> 16) as u16, b as u16, (b >> 16) as u16]));
+ ifd.set(0x0045, u16s(&[2, 0, 0, W as u16 / 2, 0]));
+ ifd.set(0x0046, u16s(&[2, (half * 8) as u16, ((half * 8) >> 16) as u16, (half * 8) as u16, ((half * 8) >> 16) as u16]));
+ ifd.set(0x0047, u16s(&[2, W as u16 / 2, W as u16 / 2]));
+ ifd.set(0x0048, u16s(&[2, H as u16, H as u16]));
+ }
+ ifd.set(t::MAKE, Value::Ascii("Panasonic".into()));
+ let mut b = TiffWriter::new(ByteOrder::Little, false).write(&[ifd]).unwrap();
+ b[..4].copy_from_slice(b"IIU\0");
+ b
+ };
+ let off = build(0).len();
+ let mut b = build(off as u32);
+ b.extend((0..n).map(|i| (i * 89 % 253) as u8));
+ b
+}
+
/// A Pentax PEF: the raw image in IFD0, wide enough for the dark-border trim.
fn pef(compression: u16) -> Vec {
let (w, h) = (264u32, 4u32);
@@ -181,6 +241,20 @@ fn samples() -> Vec> {
cr2(),
raf(),
]
+ .into_iter()
+ .chain(rw2_formats())
+ .collect()
+}
+
+/// One RW2 per encoding with a sample decoder of its own (formats 4, 6 at 14 and 12 bits, 8, and 16-bit words).
+fn rw2_formats() -> Vec> {
+ vec![
+ rw2_of(4, 12, (W * H).div_ceil(14) as usize * 16),
+ rw2_of(6, 14, (W * H).div_ceil(11) as usize * 16),
+ rw2_of(6, 12, (W * H).div_ceil(14) as usize * 16),
+ rw2_of(8, 14, (W * H) as usize * 2),
+ rw2_of(0, 12, (W * H) as usize * 2),
+ ]
}
fn exercise(bytes: &[u8]) {
@@ -208,7 +282,7 @@ proptest! {
#![proptest_config(ProptestConfig { cases: 2000, .. ProptestConfig::default() })]
#[test]
- fn mutated_vendor_files_never_panic(kind in 0usize..12, flips in proptest::collection::vec((any::(), any::()), 1..16), cut in any::()) {
+ fn mutated_vendor_files_never_panic(kind in 0usize..17, flips in proptest::collection::vec((any::(), any::()), 1..16), cut in any::()) {
let mut data = samples().swap_remove(kind);
let n = data.len();
for (i, v) in flips {
@@ -218,3 +292,12 @@ proptest! {
exercise(&data[..keep]);
}
}
+
+/// The RW2 samples decode as they are, so the mutations above reach every format's sample decoder.
+#[test]
+fn rw2_samples_decode_unmutated() {
+ for s in &rw2_formats() {
+ let r = lightcraft_raw::decode(s).unwrap();
+ assert_eq!((r.width, r.height), (W as usize, H as usize));
+ }
+}
diff --git a/crates/segment/Cargo.toml b/crates/segment/Cargo.toml
index bd7b0d66..11dff5c7 100644
--- a/crates/segment/Cargo.toml
+++ b/crates/segment/Cargo.toml
@@ -18,10 +18,7 @@ log = { workspace = true }
[target.'cfg(not(target_arch = "wasm32"))'.dependencies]
candle-core = { workspace = true }
candle-nn = { workspace = true }
-rustls = { version = "0.23", default-features = false, features = ["std", "tls12"] }
-rustls-rustcrypto = "0.0.2-alpha"
-webpki-roots = "1"
-sha2 = { version = "0.10", default-features = false, features = ["std"] }
+lightcraft-fetch = { workspace = true }
[target.'cfg(target_os = "macos")'.dependencies]
candle-core = { workspace = true, features = ["metal"] }
diff --git a/crates/segment/src/fetch/mod.rs b/crates/segment/src/fetch/mod.rs
index 92c5b6ea..55c083cc 100644
--- a/crates/segment/src/fetch/mod.rs
+++ b/crates/segment/src/fetch/mod.rs
@@ -1,39 +1,13 @@
-//! Downloading the SAM 3 model, only when the user asked for it: from an ordered list of mirrors
-//! (the next one when a mirror fails), resuming partial files, with connect and stall timeouts,
-//! a size cap, progress and cancellation, and checked (exact size, SHA-256 where pinned) before
-//! each file is moved into the model folder. Runs on whatever thread calls [`download`] (the
-//! engine uses a background thread; the UI thread never waits for the network).
-//!
-//! Files are written to `.part` next to their final place and renamed when verified, so a
-//! model folder never holds a half-written file under a real name, and an interrupted download
-//! resumes from where it stopped. A file that fails its check is deleted, never kept.
+//! Downloading the SAM 3 model, only when the user asked for it. The downloader itself (mirrors,
+//! resuming, checks, the pure-Rust HTTPS client) is the `lightcraft-fetch` crate; this module
+//! says what SAM 3 needs and where it may come from.
//!
//! The weights are never part of LightCraft (SAM License, see docs/ai-masks.md); where they are
//! downloaded from is configured by [`DEFAULT_MIRRORS`] and the user's own list.
-pub mod http;
+pub use lightcraft_fetch::*;
-use std::fs::{File, OpenOptions};
-use std::io::{Read, Write};
-use std::path::{Path, PathBuf};
-use std::sync::atomic::{AtomicBool, Ordering};
-use std::time::Duration;
-
-use sha2::{Digest, Sha256};
-
-use http::{HttpError, Limits, Url};
-
-/// One file of the model.
-#[derive(Clone, Debug, PartialEq, Eq)]
-pub struct FileSpec {
- pub name: &'static str,
- /// Exact size in bytes, when pinned (then the download resumes and is checked against it).
- pub size: Option,
- /// SHA-256 (lower-case hex), when pinned.
- pub sha256: Option<&'static str>,
- /// Largest size accepted when `size` is not pinned.
- pub max: u64,
-}
+use std::path::Path;
/// The `facebook/sam3` checkpoint files [`crate::Sam3::load`] needs.
///
@@ -71,354 +45,27 @@ pub const DEFAULT_MIRRORS: &[&str] = &[];
/// tried before the defaults.
pub const MIRRORS_ENV: &str = "LIGHTCRAFT_SAM3_MIRRORS";
-/// Most mirrors used (a hostile list can't make a download loop forever).
-const MAX_MIRRORS: usize = 16;
-/// Redirects followed per request.
-const MAX_REDIRECTS: usize = 8;
-
-/// The mirrors to try, in order: the environment variable's, then the mirrors file's (one base
-/// URL per line, `#` comments), then [`DEFAULT_MIRRORS`]; duplicates and unusable URLs dropped.
+/// The mirrors to try for SAM 3, in order: the environment variable's, then the mirrors file's
+/// (one base URL per line, `#` comments), then [`DEFAULT_MIRRORS`].
pub fn mirrors(env: Option<&str>, file: Option<&Path>) -> Vec {
- let mut out: Vec = Vec::new();
- let mut add = |s: &str| {
- let s = s.trim().trim_end_matches('/');
- if !s.is_empty() && Url::parse(s).is_ok() && !out.iter().any(|m| m == s) && out.len() < MAX_MIRRORS {
- out.push(s.to_string());
- }
- };
- if let Some(env) = env {
- env.split([',', ' ', '\n', '\t']).for_each(&mut add);
- }
- // a small text file; anything larger isn't a mirror list
- if let Some(text) = file.and_then(|f| std::fs::metadata(f).ok().filter(|m| m.len() < 64 * 1024).and_then(|_| std::fs::read_to_string(f).ok())) {
- text.lines().map(|l| l.split('#').next().unwrap_or_default()).for_each(&mut add);
- }
- DEFAULT_MIRRORS.iter().for_each(|m| add(m));
- out
+ lightcraft_fetch::mirrors(env, file, DEFAULT_MIRRORS)
}
-/// Download settings.
-#[derive(Clone, Debug)]
-pub struct Options {
- /// Longest wait to connect to a mirror.
- pub connect_timeout: Duration,
- /// Longest wait for the next bytes once connected.
- pub stall_timeout: Duration,
- /// Tries per mirror and file (each resumes where the last one stopped).
- pub attempts: u32,
-}
-
-impl Default for Options {
- fn default() -> Self {
- Options { connect_timeout: Duration::from_secs(15), stall_timeout: Duration::from_secs(30), attempts: 3 }
- }
-}
-
-/// Where a download is.
-#[derive(Clone, Debug, Default, PartialEq, Eq)]
-pub struct Progress {
- pub file: String,
- /// Bytes of all files on disk so far, and the total expected (pinned sizes + announced ones).
- pub done: u64,
- pub total: u64,
- /// The mirror in use (without query).
- pub mirror: String,
-}
-
-/// Why a download failed.
-#[derive(Debug, Clone, PartialEq, Eq)]
-pub enum DownloadError {
- /// No mirror is configured.
- NoMirrors,
- Cancelled,
- /// A local file problem (folder not writable, disk full…).
- Disk(String),
- /// Every mirror failed for `file`; the reason for each.
- AllMirrorsFailed {
- file: String,
- errors: Vec,
- },
-}
-
-impl std::fmt::Display for DownloadError {
- fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
- match self {
- DownloadError::NoMirrors => write!(
- f,
- "no download location is configured for the SAM 3 model in this build (set {MIRRORS_ENV}, or put the files in the model folder yourself; see docs/ai-masks.md)"
- ),
- DownloadError::Cancelled => write!(f, "the download was cancelled"),
- DownloadError::Disk(e) => write!(f, "{e}"),
- DownloadError::AllMirrorsFailed { file, errors } => write!(f, "could not download {file}: {}", errors.join("; ")),
- }
- }
-}
-
-/// One mirror's failure for one file; `Retry` resumes on the same mirror.
-enum Fail {
- Retry(String),
- NextMirror(String),
- Cancelled,
- Disk(String),
-}
-
-/// Download `files` into `dir` from `mirrors` (in order; see [`mirrors`]). Files already there
-/// and verified are kept. `progress` is called as bytes arrive (often: throttle in it).
-pub fn download(
- files: &[FileSpec],
- mirrors: &[String],
- dir: &Path,
- opts: &Options,
- cancel: &AtomicBool,
- progress: &mut dyn FnMut(&Progress),
-) -> Result<(), DownloadError> {
- std::fs::create_dir_all(dir).map_err(|e| DownloadError::Disk(format!("can't create {}: {e}", dir.display())))?;
- // what's left to fetch
- let mut todo = Vec::new();
- for f in files {
- if !verified(f, &dir.join(f.name))? {
- todo.push(f);
- }
- }
- if todo.is_empty() {
- return Ok(());
- }
- if mirrors.is_empty() {
- return Err(DownloadError::NoMirrors);
- }
- let total: u64 = files.iter().map(|f| f.size.unwrap_or(0)).sum();
- let mut before: u64 = files.iter().filter(|f| !todo.contains(f)).filter_map(|f| f.size).sum();
- for f in todo {
- let mut errors = Vec::new();
- let mut ok = false;
- 'mirrors: for m in mirrors {
- let url = format!("{m}/{}", f.name);
- for _ in 0..opts.attempts.max(1) {
- if cancel.load(Ordering::Relaxed) {
- return Err(DownloadError::Cancelled);
- }
- let mut report = |n: u64, announced: u64| {
- let mirror = Url::parse(m).map(|u| u.display()).unwrap_or_default();
- progress(&Progress {
- file: f.name.to_string(),
- done: before.saturating_add(n),
- total: total.max(before.saturating_add(announced)),
- mirror,
- })
- };
- match fetch_file(f, &url, dir, opts, cancel, &mut report) {
- Ok(()) => {
- ok = true;
- break 'mirrors;
- }
- Err(Fail::Cancelled) => return Err(DownloadError::Cancelled),
- Err(Fail::Disk(e)) => return Err(DownloadError::Disk(e)),
- Err(Fail::Retry(e)) => {
- log::warn!("SAM 3 download of {url}: {e} (retrying)");
- errors.push(format!("{}: {e}", short(m)));
- }
- Err(Fail::NextMirror(e)) => {
- log::warn!("SAM 3 download of {url}: {e}");
- errors.push(format!("{}: {e}", short(m)));
- continue 'mirrors;
- }
- }
- }
- }
- if !ok {
- return Err(DownloadError::AllMirrorsFailed { file: f.name.to_string(), errors });
- }
- before = before.saturating_add(std::fs::metadata(dir.join(f.name)).map(|m| m.len()).unwrap_or(0));
- }
- Ok(())
-}
-
-/// A mirror for messages (no query string: it may hold a token).
-fn short(m: &str) -> String {
- Url::parse(m).map(|u| u.display()).unwrap_or_else(|_| "mirror".into())
-}
-
-/// Whether `path` is already the right file (exact size and hash when pinned).
-fn verified(f: &FileSpec, path: &Path) -> Result {
- let Ok(meta) = std::fs::metadata(path) else { return Ok(false) };
- if !meta.is_file() || meta.len() == 0 {
- return Ok(false);
- }
- if f.size.is_some_and(|s| s != meta.len()) || meta.len() > f.max {
- return Ok(false);
- }
- match f.sha256 {
- Some(want) => Ok(sha256_file(path).map_err(DownloadError::Disk)? == want),
- None => Ok(true),
- }
-}
-
-fn sha256_file(path: &Path) -> Result {
- let mut file = File::open(path).map_err(|e| format!("{}: {e}", path.display()))?;
- let mut h = Sha256::new();
- let mut buf = vec![0u8; 1 << 20];
- loop {
- let n = file.read(&mut buf).map_err(|e| format!("{}: {e}", path.display()))?;
- if n == 0 {
- break;
- }
- h.update(buf.get(..n).unwrap_or_default());
- }
- Ok(h.finalize().iter().map(|b| format!("{b:02x}")).collect())
-}
-
-fn part_path(dir: &Path, name: &str) -> PathBuf {
- dir.join(format!("{name}.part"))
-}
-
-/// Fetch one file from one mirror into `.part` (resuming it when its size is pinned),
-/// check it and move it into place.
-fn fetch_file(f: &FileSpec, url: &str, dir: &Path, opts: &Options, cancel: &AtomicBool, report: &mut dyn FnMut(u64, u64)) -> Result<(), Fail> {
- let part = part_path(dir, f.name);
- let limit = f.size.unwrap_or(f.max);
- let disk = |e: std::io::Error| Fail::Disk(format!("{}: {e}", part.display()));
- // resume only a pinned file (an unpinned one can't be checked after resuming)
- let mut have = match std::fs::metadata(&part) {
- Ok(m) if f.size.is_some() && m.len() <= limit => m.len(),
- Ok(_) => {
- std::fs::remove_file(&part).map_err(disk)?;
- 0
- }
- Err(_) => 0,
- };
- let fetched = f.size != Some(have) || have == 0;
- if fetched {
- let limits = Limits { connect: opts.connect_timeout, stall: opts.stall_timeout, cancel };
- let mut url = Url::parse(url).map_err(|e| Fail::NextMirror(e.to_string()))?;
- let origin = url.host.clone();
- let mut redirects = 0;
- let mut resp = loop {
- let mut headers = vec![];
- if have > 0 {
- headers.push(("Range", format!("bytes={have}-")));
- }
- // a token for the configured host only (never sent on to a redirect's host)
- if let Ok(t) = std::env::var("LIGHTCRAFT_SAM3_TOKEN")
- && !t.trim().is_empty()
- && url.tls
- && url.host == origin
- {
- headers.push(("Authorization", format!("Bearer {}", t.trim())));
- }
- let r = http::get(&url, &headers, &limits).map_err(http_fail)?;
- match r.status {
- 301 | 302 | 303 | 307 | 308 => {
- redirects += 1;
- if redirects > MAX_REDIRECTS {
- return Err(Fail::NextMirror("too many redirects".into()));
- }
- let loc = r.header("location").ok_or_else(|| Fail::NextMirror("redirect without a location".into()))?;
- url = url.join(loc).map_err(|e| Fail::NextMirror(e.to_string()))?;
- }
- _ => break r,
- }
- };
- let mut file = match resp.status {
- 200 => {
- // the whole file (from the start, also when a range was asked for)
- if resp.content_length().is_some_and(|n| n > limit || f.size.is_some_and(|s| s != n)) {
- return Err(Fail::NextMirror(format!("the server's file has the wrong size ({:?} bytes)", resp.content_length())));
- }
- have = 0;
- File::create(&part).map_err(disk)?
- }
- 206 if have > 0 => {
- let range = resp.header("content-range").unwrap_or_default();
- let (start, total) = parse_content_range(range).ok_or_else(|| Fail::NextMirror(format!("bad Content-Range `{range}`")))?;
- if start != have || total.is_some_and(|t| Some(t) != f.size) {
- return Err(Fail::NextMirror(format!("the server resumed at the wrong place ({range})")));
- }
- OpenOptions::new().append(true).open(&part).map_err(disk)?
- }
- 416 => {
- // the partial file doesn't fit what the server has: start over
- std::fs::remove_file(&part).map_err(disk)?;
- return Err(Fail::Retry("the partial download didn't match; starting over".into()));
- }
- 401 | 403 => return Err(Fail::NextMirror(format!("access denied (HTTP {})", resp.status))),
- 404 | 410 => return Err(Fail::NextMirror(format!("not found (HTTP {})", resp.status))),
- s if s >= 500 => return Err(Fail::Retry(format!("server error (HTTP {s})"))),
- s => return Err(Fail::NextMirror(format!("HTTP {s}"))),
- };
- let announced = f.size.unwrap_or_else(|| resp.content_length().unwrap_or(0));
- report(have, announced);
- let mut buf = vec![0u8; 256 * 1024];
- loop {
- let n = match resp.read(&mut buf, &limits) {
- Ok(n) => n,
- Err(e) => {
- let _ = file.flush();
- if f.size.is_none() {
- drop(file);
- let _ = std::fs::remove_file(&part);
- }
- return Err(http_fail(e));
- }
- };
- if n == 0 {
- break;
- }
- if have.saturating_add(n as u64) > limit {
- drop(file);
- let _ = std::fs::remove_file(&part);
- return Err(Fail::NextMirror(format!("the server sent more than the expected {limit} bytes")));
- }
- file.write_all(buf.get(..n).unwrap_or_default()).map_err(disk)?;
- have = have.saturating_add(n as u64);
- report(have, announced);
- }
- file.sync_all().map_err(disk)?;
- }
- // checks: exact size, hash
- let len = std::fs::metadata(&part).map_err(disk)?.len();
- if f.size.is_some_and(|s| s != len) {
- return Err(Fail::Retry(format!("incomplete ({len} of {} bytes)", f.size.unwrap_or(0))));
- }
- if len == 0 {
- let _ = std::fs::remove_file(&part);
- return Err(Fail::NextMirror("the server sent an empty file".into()));
- }
- if let Some(want) = f.sha256 {
- let got = sha256_file(&part).map_err(Fail::Disk)?;
- if got != want {
- std::fs::remove_file(&part).map_err(disk)?;
- let why = format!("the file is damaged or not the official one (SHA-256 {got})");
- // a damaged partial file from an earlier run isn't this mirror's fault
- return Err(if fetched { Fail::NextMirror(why) } else { Fail::Retry(why) });
- }
- }
- std::fs::rename(&part, dir.join(f.name)).map_err(disk)?;
- Ok(())
-}
-
-fn http_fail(e: HttpError) -> Fail {
- match e {
- HttpError::Cancelled => Fail::Cancelled,
- HttpError::Stalled | HttpError::Io(_) | HttpError::Protocol(_) => Fail::Retry(e.to_string()),
- HttpError::BadUrl(_) | HttpError::Connect(_) | HttpError::Tls(_) => Fail::NextMirror(e.to_string()),
- }
-}
-
-/// `bytes a-b/total` → (a, total) (`*` total → None).
-fn parse_content_range(v: &str) -> Option<(u64, Option)> {
- let v = v.trim().strip_prefix("bytes")?.trim();
- let (range, total) = v.split_once('/')?;
- let (a, b) = range.split_once('-')?;
- let (a, b) = (a.trim().parse::().ok()?, b.trim().parse::().ok()?);
- if b < a {
- return None;
- }
- let total = match total.trim() {
- "*" => None,
- t => Some(t.parse::().ok()?),
- };
- Some((a, total))
+/// What to tell the user when there is no mirror for the SAM 3 model.
+pub fn no_mirrors_message() -> String {
+ format!(
+ "no download location is configured for the SAM 3 model in this build (set {MIRRORS_ENV}, or put the files in the model folder yourself; see docs/ai-masks.md)"
+ )
}
#[cfg(test)]
-mod tests;
+mod tests {
+ use super::*;
+
+ #[test]
+ fn sam3_mirrors_come_from_the_environment_the_file_and_the_defaults() {
+ let m = mirrors(Some("https://a.example/x/"), Some(Path::new("/nonexistent/mirrors.txt")));
+ assert_eq!(m.len(), 1 + DEFAULT_MIRRORS.len());
+ assert!(no_mirrors_message().contains(MIRRORS_ENV));
+ }
+}
diff --git a/crates/ui-egui/src/headless.rs b/crates/ui-egui/src/headless.rs
index 4b8ec39e..67bec906 100644
--- a/crates/ui-egui/src/headless.rs
+++ b/crates/ui-egui/src/headless.rs
@@ -231,18 +231,16 @@ impl Headless {
self.paint()
}
- /// Wait for a fixture widget whose background directory listing may still be loading.
+ /// Tests that browse a folder in the temp directory expect it to be a Local location of its
+ /// own. Where the temp directory lies inside the home folder (Windows), the sidebar lists it
+ /// inside Home's tree instead, so hide Home for the test; elsewhere this does nothing.
#[cfg(test)]
- pub(crate) fn wait_for_widget(&mut self, id: &str, timeout: Duration) -> bool {
- let start = Instant::now();
- while let Some(remaining) = timeout.checked_sub(start.elapsed()) {
- let reply = self.request("ui.widgets", json!({"filter": id}), remaining);
- if reply["result"].as_array().is_some_and(|widgets| widgets.iter().any(|w| w["id"].as_str() == Some(id))) {
- return true;
- }
- std::thread::sleep(Duration::from_millis(1));
+ pub(crate) fn hide_home_above(&mut self, path: &std::path::Path) {
+ let home = std::env::var("HOME").or_else(|_| std::env::var("USERPROFILE")).unwrap_or_default();
+ if !home.is_empty() && lightcraft_catalog::query::folder_within(&path.to_string_lossy(), &home) {
+ let r = self.request("engine.execute", json!({"command": "local.hide", "params": {"path": home}}), Duration::from_secs(10));
+ assert_eq!(r["ok"], true, "{r}");
}
- false
}
/// Send a control-protocol request (see [`crate::control`]) and run frames until it is
@@ -294,6 +292,8 @@ mod tests {
fn demo_grid_snapshot_has_ui_pixels() {
let t0 = Instant::now();
let mut h = demo([1200.0, 760.0]);
+ // settle() can see a quiet spell before the first thumbnails land on a loaded machine
+ h.step_until(SETTLE, |h| h.app.renderer.thumb_textures() > 0);
let img = h.snapshot(SETTLE);
eprintln!("headless snapshot: {:?} in {:?} ({} frames)", img.size, t0.elapsed(), h.frames());
assert_eq!(img.size, [1200, 760]);
@@ -398,6 +398,7 @@ mod tests {
let id = h.app.session.active().unwrap();
let photo = |h: &Headless| h.app.session.catalog.photo(id).unwrap().clone();
let before = photo(&h);
+ h.step_until(SETTLE, |h| h.app.renderer.variant_textures() >= 6);
assert!(h.app.renderer.variant_textures() >= 6, "variant thumbnails rendered: {}", h.app.renderer.variant_textures());
// hover: the loupe shows the look, nothing is committed
let r = h.request("ui.hoverWidget", json!({"id": "profileCell:lc.vivid"}), t);
@@ -900,6 +901,7 @@ mod tests {
};
let ids = |h: &mut Headless| -> Vec { rects(h).into_iter().map(|(id, _)| id).collect() };
h.request("ui.set", json!({"leftPanel": true}), t);
+ h.hide_home_above(&dir);
// Browse Folder… browses the picked folder and keeps it in Local within one frame; a
// request runs frames, so keep it first (no frame sees it browsed but not yet kept)
let r = h.request("engine.execute", json!({"command": "local.addRoot", "params": {"path": path}}), t);
@@ -907,8 +909,11 @@ mod tests {
let r = h.request("engine.execute", json!({"command": "library.browse", "params": {"path": path}}), t);
assert_eq!(r["ok"], true, "{r}");
h.settle(SETTLE);
- assert!(h.wait_for_widget(&format!("source:local:{path}"), t), "the browsed folder is listed: {:?}", ids(&mut h));
- assert!(!ids(&mut h).iter().any(|i| i == "source:local:restoreHidden"));
+ let row = format!("source:local:{path}");
+ assert!(h.step_until(t, |h| h.app.widgets.iter().any(|(w, _)| *w == row)), "the browsed folder is listed: {:?}", ids(&mut h));
+ // the "Show hidden locations" row appears only while something is hidden
+ let hidden_before = !h.app.ui.hidden_locations.is_empty();
+ assert_eq!(ids(&mut h).iter().any(|i| i == "source:local:restoreHidden"), hidden_before);
// hide it while it is still being browsed
let r = h.request("engine.execute", json!({"command": "local.hide", "params": {"path": path}}), t);
assert_eq!(r["ok"], true, "{r}");
@@ -930,7 +935,11 @@ mod tests {
assert_eq!(h.request("ui.clickWidget", json!({"id": "source:local:restoreHidden"}), t)["ok"], true);
h.settle(SETTLE);
assert!(h.app.ui.hidden_locations.is_empty());
- assert!(ids(&mut h).contains(&format!("source:local:{path}")));
+ assert!(!ids(&mut h).iter().any(|i| i == "source:local:restoreHidden"), "nothing left to restore");
+ if !hidden_before {
+ // (with Home hidden for this test, the folder is back inside Home's tree instead)
+ assert!(ids(&mut h).contains(&format!("source:local:{path}")));
+ }
let _ = std::fs::remove_dir_all(&dir);
}
@@ -947,12 +956,10 @@ mod tests {
std::fs::create_dir_all(base.join(d)).unwrap();
}
let s = |p: std::path::PathBuf| p.to_string_lossy().to_string();
- let (photos, day1, day2, other) = (
- s(base.join("Photos")),
- s(base.join("Photos").join("2026").join("20260101")),
- s(base.join("Photos").join("2026").join("20260114")),
- s(base.join("Other")),
- );
+ // joined by component: the sidebar's ids spell paths with the platform's separator
+ let year = base.join("Photos").join("2026");
+ let (photos, day1, day2, other) = (s(base.join("Photos")), s(year.join("20260101")), s(year.join("20260114")), s(base.join("Other")));
+ h.hide_home_above(&base);
let exec = |h: &mut Headless, c: &str, p: Value| h.request("engine.execute", json!({"command": c, "params": p}), t);
let rects = |h: &mut Headless| -> std::collections::HashMap {
let w = h.request("ui.widgets", json!({"filter": "lc-ui-roots-"}), t);
@@ -969,7 +976,8 @@ mod tests {
exec(&mut h, "library.browse", json!({"path": day1}));
h.settle(SETTLE);
h.step();
- assert!(h.wait_for_widget(&format!("source:local:{day2}"), t), "the sibling listing did not arrive");
+ let row = format!("source:local:{day2}");
+ assert!(h.step_until(t, |h| h.app.widgets.iter().any(|(w, _)| *w == row)), "the sibling listing did not arrive");
let r = rects(&mut h);
assert!(r.contains_key(&format!("source:local:{photos}")), "the kept root stays: {r:?}");
assert!(r.contains_key(&format!("source:local:{day2}")), "the sibling stays reachable: {r:?}");
@@ -982,7 +990,8 @@ mod tests {
assert_eq!(exec(&mut h, "library.browse", json!({"path": other}))["ok"], true);
h.settle(SETTLE);
h.step();
- assert!(h.wait_for_widget(&format!("source:local:{other}"), t), "the other location did not arrive");
+ let row = format!("source:local:{other}");
+ assert!(h.step_until(t, |h| h.app.widgets.iter().any(|(w, _)| *w == row)), "the other location did not arrive");
let r = rects(&mut h);
assert!(r.contains_key(&format!("source:local:{photos}")) && r.contains_key(&format!("source:local:{other}")), "{r:?}");
// restart: the kept roots come back with the saved UI state
@@ -1094,6 +1103,7 @@ mod tests {
let r = h.request("engine.execute", json!({"command": "file.addFromDevice", "params": {"path": sub.to_string_lossy()}}), t);
assert_eq!(r["ok"], true, "{r}");
h.settle(SETTLE);
+ h.step_until(SETTLE, |h| matches!(h.app.ui.dialog, Some(crate::state::Dialog::Import { .. })));
let Some(crate::state::Dialog::Import { opts }) = &h.app.ui.dialog else { panic!("no import review") };
assert!(opts.copy);
let _ = std::fs::remove_dir_all(&dir);
@@ -1308,6 +1318,8 @@ mod tests {
let r = h.request("engine.execute", json!({"command": "file.addPhotos", "params": {"paths": [dir.to_string_lossy()]}}), t);
assert_eq!(r["result"]["scanning"], true, "{r}");
h.settle(SETTLE);
+ h.step_until(SETTLE, |h| matches!(h.app.ui.dialog, Some(crate::state::Dialog::Import { .. })));
+ h.step_until(SETTLE, |h| h.app.renderer.textures.keys().filter(|s| matches!(s, crate::render::Slot::Import(_))).count() >= 5);
let Some(crate::state::Dialog::Import { opts }) = &h.app.ui.dialog else { panic!("no import review") };
assert_eq!(opts.candidates.len(), 6);
assert_eq!(opts.candidates.iter().filter(|c| c.duplicate.is_some()).count(), 1);
@@ -1440,6 +1452,7 @@ mod tests {
let r = h.request("engine.execute", json!({"command": "file.addPhotos", "params": {"paths": [src.to_string_lossy()]}}), t);
assert_eq!(r["result"]["scanning"], true, "{r}");
h.settle(SETTLE);
+ h.step_until(SETTLE, |h| matches!(h.app.ui.dialog, Some(crate::state::Dialog::Import { .. })));
let Some(crate::state::Dialog::Import { opts }) = &h.app.ui.dialog else { panic!("no import review") };
assert_eq!(opts.candidates.len(), 10);
for id in ["button:importCopy", "button:importDest"] {
diff --git a/crates/ui-egui/src/tests_masking.rs b/crates/ui-egui/src/tests_masking.rs
index bdf9a852..8e2c8139 100644
--- a/crates/ui-egui/src/tests_masking.rs
+++ b/crates/ui-egui/src/tests_masking.rs
@@ -366,6 +366,7 @@ fn local_folder_tree_expands_and_browses() {
std::fs::create_dir_all(base.join(".hidden")).unwrap();
let mut h = detail("panel.edit");
exec(&mut h, "view.leftPanel", json!({"show": true}));
+ h.hide_home_above(&base);
exec(&mut h, "local.addRoot", json!({"path": base.to_string_lossy()}));
exec(&mut h, "library.browse", json!({"path": base.to_string_lossy()}));
h.settle(SETTLE);
diff --git a/docs/ai-masks.md b/docs/ai-masks.md
index 26d4f717..5a32d690 100644
--- a/docs/ai-masks.md
+++ b/docs/ai-masks.md
@@ -170,7 +170,7 @@ wait and return the result (when built with the `sam` feature; the default CLI h
with 2-D RoPE and windowed attention, the feature pyramids, the SAM 2-style prompt encoder and
two-way mask decoder for clicks, and the CLIP text encoder, DETR encoder/decoder (box
relative-position bias, presence token) and pixel decoder for text. The CLIP tokenizer is a
- small BPE in `tokenizer.rs`. `fetch/` is the downloader (original code).
+ small BPE in `tokenizer.rs`. The downloader (original code) is the `lightcraft-fetch` crate, shared by any model that is downloaded; `fetch/` here says what SAM 3 needs.
- candle is pinned at 0.9.2: later releases make `candle-core` depend on `tokenizers` with the
Oniguruma C library, and the product is pure Rust. Weights are read with positional reads (no
memory map: `unsafe` stays in `lightcraft-sysmem`), only the tensors a path needs, and every
diff --git a/docs/camera-preview-colour.md b/docs/camera-preview-colour.md
index fac08bc7..308287c6 100644
--- a/docs/camera-preview-colour.md
+++ b/docs/camera-preview-colour.md
@@ -1,4 +1,4 @@
-# Sony ARW and Nikon NEF starting look
+# Sony ARW, Nikon NEF and Panasonic RW2 starting look
ARW and NEF decoding supply a Bayer mosaic and camera white-balance multipliers, but currently no measured camera colour matrix. Treating camera RGB as linear sRGB produces a dark, muted default rendering. The loader now estimates a file-local starting look from that file's embedded camera JPEG, only when a usable camera matrix is absent. All output pixels still come from the RAW mosaic; there is no JPEG replacement or uniform saturation boost.
@@ -40,6 +40,12 @@ Measuring this found a decoder bug: 12-bit NEFs store maker note `0x003d` BlackL
On one sample, Exposure +1 EV, Saturation +100, Contrast +50, warm/cool WB, tint, highlights and shadows all changed the output. The same matrix and curve were selected for 400-pixel and full-resolution renders. Headless UI checks wait until the loupe reports `source: render` and no jobs remain; the camera JPEG stand-in does not count as verification.
+### Panasonic RW2
+
+The same fit, gates and relative WB apply to Panasonic RW2, Leica RWL and the older Panasonic RAW files (all decode as `RawFormat::Rw2`; white balance from tags `0x0024`–`0x0026`). Their embedded JPEG (`JpgFromRaw`, 1920 px wide) always shows the whole active area, while the default crop is the aspect ratio set in the camera (4:3 on the 3:2 sensors, 1:1, 16:9, 2.71 on the S1R II…); when the preview matches the active area rather than the crop, the reference is cropped to the default crop before fitting.
+
+Public CC0 samples from raw.pixls.us (178 files from 118 Panasonic and Leica bodies; the four from 2005–2007 carry no preview), rendered at 1200 px and compared with each file's embedded JPEG (mean CIE76 ΔE at 200 px): 140 of 174 fits accepted, from 93 of the 114 bodies with a preview, median ΔE 5.0 (1.2–21.3) with matching mean lightness (median |ΔL*| 0.4); 124 of them use a hue/saturation table. (Fitted with the matrix and tone curve alone, 137 were accepted at median ΔE 6.1; every one of those is as close or closer with the table and chroma curve, by 0.75 on average.) On the 14 corpus files with a preview the neutral fallback scored ΔE 9.6–27 and rendered up to 16 L* too dark; 13 fits were accepted, ΔE 2.8–8.9, lightness within 2 L*. Of the 34 files left on the neutral fallback, 21 fail the gates, almost all compacts and kit zooms (TZ/ZS, FZ2500, D-Lux, V-Lux, GF2, GX80 with its 12–32 mm): their camera JPEG is distortion-corrected, the pixel pairs no longer line up and the held-out RMS stays above 0.10 (GX80: 0.138 → 0.107); two S1 II shots just miss it (0.103, 0.107). The other 13 never reach the gates, mostly nearly colourless scenes (grey walls, concrete: under 5% coloured pixels) or dark ones (a night sky, a near-black frame). Panasonic's embedded distortion data (tag `0x0119`) is not used yet, and `lightcraft-cli calibrate` doesn't collect RW2 files: the misaligned pairs of distortion-corrected JPEGs would distort a pooled profile.
+
This is a per-file camera-look estimate, **not measured spectral calibration or Lightroom parity**. Camera local tone, picture styles, noise reduction and lens warps cannot all be reproduced by a global matrix/curve; same-aspect crops may evade the aspect check. Skin, mixed-light scenes and saturated highlights can still differ from a supplied JPEG. Broader Sony and Nikon model/lighting coverage, chart-based calibration, absolute WB and other manufacturers remain unverified. Rejected fits and other RAW formats retain their existing behaviour.
## Camera profiles (pooled per model)
diff --git a/docs/parity.md b/docs/parity.md
index 90c5703c..956bc0c0 100644
--- a/docs/parity.md
+++ b/docs/parity.md
@@ -59,8 +59,8 @@ Ordered by user impact, then tier, then effort. The checklist above counts featu
that decide whether a photographer can switch (see the honest assessment in [ROADMAP.md](../ROADMAP.md#where-we-stand)).
Take the first one nobody is working on.
-1. **LR-PROF-CAMERACOLOR** (P0): our own camera colour calibration. Sony ARW and Nikon NEF now get a guarded file-local fit to their own embedded JPEG (colour matrix + hue/saturation/value table + tone and chroma curves, relative WB; per-model profiles pooled from many photos via `lightcraft-cli calibrate`; ARW: 7 of 10 public samples accepted, mean ΔE vs the camera JPEG 17–26 → 3–10; NEF: 13 of 13 decodable samples from 6 bodies accepted, ΔE 13–46 → 3–8, one mixed-light scene 13 → 12); measured calibration and fidelity remain missing. Other non-DNG raws and rejected fits still use a neutral matrix. Expand validated preview fitting and use matrices the files carry themselves; never Adobe data.
-2. **LR-IMP-FORMATS** (P0): **CR3** first (every Canon body since ~2018), then compressed RAF / ORF, RW2 v4, Nikon
+1. **LR-PROF-CAMERACOLOR** (P0): our own camera colour calibration. Sony ARW, Nikon NEF and Panasonic RW2 now get a guarded file-local fit to their own embedded JPEG (colour matrix + hue/saturation/value table + tone and chroma curves, relative WB; per-model profiles pooled from many photos via `lightcraft-cli calibrate` for ARW and NEF; ARW: 7 of 10 public samples accepted, mean ΔE vs the camera JPEG 17–26 → 3–10; NEF: 13 of 13 decodable samples from 6 bodies accepted, ΔE 13–46 → 3–8, one mixed-light scene 13 → 12; RW2: 140 of 174 public samples accepted, from 93 of the 114 bodies with a preview, median ΔE 5.0); measured calibration and fidelity remain missing. Other non-DNG raws and rejected fits still use a neutral matrix. Expand validated preview fitting and use matrices the files carry themselves; never Adobe data.
+2. **LR-IMP-FORMATS** (P0): **CR3** first (every Canon body since ~2018), then compressed RAF / ORF, Nikon
"lossy after split" NEF, Canon sRAW; HEIC/AVIF decode. Clean-room, from prose descriptions only (see
`crates/raw/src/vendor/nefc.rs` for how compressed NEF was done). Until decoded, such photos are `preview_only`.
3. **LR-IMP-CAMERA-COVERAGE** (P0): per-model verification; grow the CC0 corpus and fix per-model bugs (like the CR2
@@ -124,8 +124,8 @@ under the UI command that wraps it (a no-op error); the UI command now wins.
| LR-IMP-PROFILES | Import profiles & presets | P1 | ✅ | `cmd:file.importPresets`, `cmd:preset.import`, `cmd:profile.import`, `crates/engine/src/preset_import.rs`, `crates/engine/src/preset_luminar.rs` | presets: .lcpreset, XMP, classic .lrtemplate, photos carrying edits ("DNG presets"), Luminar looks (.lmp, .mplumpack collections; sliders with a counterpart), .zip bundles, folders (folder → group), drag & drop; masks carried over; unmapped settings reported; profiles: .cube 3D LUTs; Adobe profile formats deliberately unsupported |
| LR-IMP-LOCAL | Work on files in place | P0 | ✅ | `cmd:library.browse`, `cmd:photo.addToLibrary`, `cmd:library.import` (mode add), `cmd:local.addRoot`, `cmd:local.hide`, `cmd:local.restoreHidden`, `crates/engine/src/cmd/browse.rs`, `crates/ui-egui/src/panels/left.rs` (`local_section`), `crates/ui-egui/src/panels/grid.rs` (`folder_header`) | Local: browse Pictures / Desktop / Downloads / Home or any folder without adding it (breadcrumb, Include subfolders, Add N to My Photos); browsed photos stay out of All Photos, albums and counts; folders picked with Browse Folder… (or Keep in Local / Add to Local on a folder row) stay listed across restarts, and browsing a folder inside a listed one highlights it inside that tree (opened down to it, siblings reachable) instead of adding a root; edits go to XMP sidecars; importing promotes them; right-click a top-level location → Remove from Local hides the shortcut only (nothing on disk or in the catalog changes), saved with the UI state, and “Show N hidden locations” (or Browse Folder… on it) brings it back; a folder is one location however its path is spelled (`/` or `\`, trailing separator, `.`/`..`, drive-letter case; `folder_key` in `crates/catalog/src/query.rs`); browsed photos nobody changed are forgotten once their folder has not been browsed for 30 days (Settings → Performance → Local folders, 0 = never; checked when the library opens, or `cmd:library.forgetLocal` with `dryRun`; files and sidecars stay, browsing again brings them back; see `crates/catalog/src/local.rs` for what counts as a change) |
| LR-IMP-SIDECAR-SPLIT | Separate XMP sidecar variants | P2 | ⬜ | `cmd:library.xmpPreferences` | sidecar naming option exists (stem/full), no split sidecars |
-| LR-IMP-FORMATS | Supported formats | P0 | 🟡 | `crates/codecs/src/lib.rs`, `crates/raw/src/lib.rs` | JPEG, PNG, TIFF, WebP, JXL, PSD, GIF, BMP; DNG (incl. lossy / Smart Preview DNG), CR2 (colour-filter layout from the file's `CR2CFAPattern` tag, issue #85), ARW (pre-2017 bodies: white balance and black level from enciphered maker-note / `SR2SubIFD` data, issue #148), NEF (uncompressed, lossless and lossy compressed 12/14-bit, `crates/raw/src/vendor/nefc.rs`), RAF, RW2, PEF, ORF. 12-bit NEF black level: maker note `0x003d` is in 14-bit units (D750, D780, D850, D7500, Z 50 checked). CR3 opens from its full-size embedded JPEG with full metadata (Exif, GPS, XMP from the `CMT1`/`CMT2`/`CMT4` boxes, `crates/meta/src/cr3.rs`) but its raw data is not decoded yet. Missing: CR3 raw decoding (CRX), compressed RAF/ORF, RW2 v4, NEFs labelled compressed (34713) but stored uncompressed without a `0x0096` table (Z 6 packed 14-bit, D850 12-bit uncompressed), NEF "lossy after split" (preview only: imported with `preview_only` = the decoder's reason, rendered as a rendered JPEG, flagged by a grid badge and loupe/Edit/Info notices, `previewOnly` in `catalog.query`; Reload clears it once the file decodes), HEIC/AVIF decode |
-| LR-IMP-CAMERA-COVERAGE | Camera coverage, verified per model | P0 | 🟡 | `crates/raw/tests/corpus.rs`, `xtask/src/main.rs` | a supported container is not the same as every camera that writes it decoding correctly: ~40 CC0 corpus files (plus ~60 NEFs checked by hand) against the >1,000 models Lightroom lists; per-model differences still surface (e.g. the CR2 colour-filter layout differed by model until it was read from the file's own tag, issue #85, checked on 35 CR2 bodies; pre-2017 Sony ARWs opened green until their scrambled white balance / black level were read, issue #148, checked on 16 ARWs from 15 Sony bodies). Grow the CC0 corpus per model and test each sample decodes with plausible colour |
+| LR-IMP-FORMATS | Supported formats | P0 | 🟡 | `crates/codecs/src/lib.rs`, `crates/raw/src/lib.rs` | JPEG, PNG, TIFF, WebP, JXL, PSD, GIF, BMP; DNG (incl. lossy / Smart Preview DNG), CR2 (colour-filter layout from the file's `CR2CFAPattern` tag, issue #85), ARW (pre-2017 bodies: white balance and black level from enciphered maker-note / `SR2SubIFD` data, issue #148), NEF (uncompressed, lossless and lossy compressed 12/14-bit, `crates/raw/src/vendor/nefc.rs`), RAF, RW2 / Leica RWL / Panasonic RAW in every raw format (compressed 4 and 6, the prefix-coded strips of format 8 from the GH6 on, packed 2/5/7, the 16-bit words of the 2005–2007 bodies; black levels, in-camera aspect crops and mapped-out defect pixels; 178 CC0 files from 118 bodies checked, `crates/raw/src/vendor/rw2.rs`), PEF, ORF. 12-bit NEF black level: maker note `0x003d` is in 14-bit units (D750, D780, D850, D7500, Z 50 checked). CR3 opens from its full-size embedded JPEG with full metadata (Exif, GPS, XMP from the `CMT1`/`CMT2`/`CMT4` boxes, `crates/meta/src/cr3.rs`) but its raw data is not decoded yet. Missing: CR3 raw decoding (CRX), compressed RAF/ORF, NEFs labelled compressed (34713) but stored uncompressed without a `0x0096` table (Z 6 packed 14-bit, D850 12-bit uncompressed), NEF "lossy after split" (preview only: imported with `preview_only` = the decoder's reason, rendered as a rendered JPEG, flagged by a grid badge and loupe/Edit/Info notices, `previewOnly` in `catalog.query`; Reload clears it once the file decodes), HEIC/AVIF decode |
+| LR-IMP-CAMERA-COVERAGE | Camera coverage, verified per model | P0 | 🟡 | `crates/raw/tests/corpus.rs`, `xtask/src/main.rs` | a supported container is not the same as every camera that writes it decoding correctly: ~55 CC0 corpus files (plus ~60 NEFs checked by hand and 178 Panasonic / Leica RW2, RWL and RAW files from 118 bodies, checked against each file's own JPEG) against the >1,000 models Lightroom lists; per-model differences still surface (e.g. the CR2 colour-filter layout differed by model until it was read from the file's own tag, issue #85, checked on 35 CR2 bodies; pre-2017 Sony ARWs opened green until their scrambled white balance / black level were read, issue #148, checked on 16 ARWs from 15 Sony bodies). Grow the CC0 corpus per model and test each sample decodes with plausible colour |
| LR-IMP-CULL-AT-IMPORT | Culling analysis at import | P2 | 🟡 | `cmd:photo.analyze` | run Assisted Culling on the imported photos (they're selected after an import); not automatic |
| LR-IMP-MOVE | Move on import [Classic] | P1 | ✅ | `cmd:library.import` (`mode` move), `crates/engine/src/import_move.rs`, `crates/engine/src/tests_import_move.rs`, `crates/ui-egui/src/import.rs` | Transfer ▸ Move with Copy's destination, folders (day / month / one folder / custom template), rename template and example destination; XMP sidecars (both namings) move along; each source is removed only after its destination is written (hard link on the same volume, else copied, synced and compared byte for byte) and its catalog record is saved; failed, duplicate, unchecked files keep their sources, taken names get -1, -2…, never overwritten; files already in the destination/library are added in place; sources that can't be removed (read-only card) are kept and reported (`kept`). Undo removes the photos from the library but leaves the moved files at the destination. No Copy as DNG while moving |
| LR-IMP-DNG-CONVERT | Convert to DNG on import [Classic] | P2 | ✅ | `cmd:library.import` (`dng`), `crates/ui-egui/src/import.rs` | copy imports: Raw files ▸ Copy as DNG (lossless; the card is untouched); the raw copy is removed only once its DNG is verified and on disk (issue #106) |
@@ -276,7 +276,7 @@ under the UI command that wraps it (a no-op error); the UI command now wins.
| LR-PROF-ADOBE | Standard raw looks (own equivalents) | P0 | ✅ | `crates/engine/src/presets.rs` (`PROFILES`), `crates/pipeline/src/profiles.rs` | six own looks: Color, Neutral, Vivid, Landscape, Portrait, Monochrome |
| LR-PROF-ADAPTIVE | Adaptive profiles | P2 | ⬜ | | |
| LR-PROF-CAMERA | Camera-matching looks | P2 | ⬜ | | |
-| LR-PROF-CAMERACOLOR | Camera colour calibration (own) | P0 | 🟡 | `crates/raw/src/color.rs`, `crates/raw/src/profile.rs`, `crates/engine/src/camera_preview.rs` | DNG files use the colour matrices and the camera-profile look they carry (`ProfileHueSatMap`, `ProfileLookTable`, `ProfileToneCurve`, read from the file at run time per DNG spec ch. 6 — Lightroom-converted DNGs rendered flat and muted without them, issue #138); Sony ARW and Nikon NEF get a guarded file-local fit to their own embedded JPEG — chromaticity matrix + hue/saturation/value table + tone and chroma curves, relative WB — or, with a local per-model profile pooled from many photos (`lightcraft-cli calibrate`, `crates/engine/src/camera_profiles.rs`), the profile's colour and only their own tone/chroma curves (on 29 held-out ILCE-7M4 photos mean ΔE vs the camera JPEG 3.54 → 3.10), kept with smart previews (see docs/camera-preview-colour.md); ARW: on 10 public raw.pixls.us samples from 8 bodies 7 fits were accepted, mean ΔE vs the camera JPEG 17–26 → 3–10; NEF: 13 of 13 decodable samples (D750, D780, D850, D7500, Z 50 and local D7500 shots) accepted, ΔE 13–46 → 3–8 (issue #150); this is not measured camera calibration; other raws and rejected fits use a neutral fallback (camera RGB ≈ linear sRGB, `matrix_is_fallback`) with as-shot white balance, so colours are muted and not accurate. Needs our own per-camera calibration: matrices the files carry themselves (e.g. Olympus `ColorMatrix`), fitting each camera to its own embedded JPEG, then chart shots. Adobe matrices / DCPs are never used. Biggest image-quality gap today |
+| LR-PROF-CAMERACOLOR | Camera colour calibration (own) | P0 | 🟡 | `crates/raw/src/color.rs`, `crates/raw/src/profile.rs`, `crates/engine/src/camera_preview.rs` | DNG files use the colour matrices and the camera-profile look they carry (`ProfileHueSatMap`, `ProfileLookTable`, `ProfileToneCurve`, read from the file at run time per DNG spec ch. 6 — Lightroom-converted DNGs rendered flat and muted without them, issue #138); Sony ARW, Nikon NEF and Panasonic RW2 get a guarded file-local fit to their own embedded JPEG — chromaticity matrix + hue/saturation/value table + tone and chroma curves, relative WB — or, with a local per-model profile pooled from many photos (`lightcraft-cli calibrate`, ARW and NEF so far, `crates/engine/src/camera_profiles.rs`), the profile's colour and only their own tone/chroma curves (on 29 held-out ILCE-7M4 photos mean ΔE vs the camera JPEG 3.54 → 3.10), kept with smart previews (see docs/camera-preview-colour.md); ARW: on 10 public raw.pixls.us samples from 8 bodies 7 fits were accepted, mean ΔE vs the camera JPEG 17–26 → 3–10; NEF: 13 of 13 decodable samples (D750, D780, D850, D7500, Z 50 and local D7500 shots) accepted, ΔE 13–46 → 3–8 (issue #150); RW2 / RWL: 140 of 174 public samples accepted, from 93 of the 114 bodies with a preview, median ΔE 5.0 with matching lightness (13 of the 14 corpus files with a preview: ΔE 9.6–27 → 2.8–8.9; most rejections are compacts and kit zooms whose camera JPEG is distortion-corrected); this is not measured camera calibration; other raws and rejected fits use a neutral fallback (camera RGB ≈ linear sRGB, `matrix_is_fallback`) with as-shot white balance, so colours are muted and not accurate. Needs our own per-camera calibration: matrices the files carry themselves (e.g. Olympus `ColorMatrix`), fitting each camera to its own embedded JPEG, then chart shots. Adobe matrices / DCPs are never used. Biggest image-quality gap today |
| LR-PROF-CREATIVE | Creative profiles (own) | P2 | ✅ | `cmd:develop.profile`, `crates/pipeline/src/profiles.rs` | 16 own looks in Film / Cinematic / Muted / B&W (tone + point-curve fades, colour grading, mixer / B&W mix); scale with `ctl:profile.amount`; sliders untouched |
| LR-PROF-LEGACY | Legacy profiles | P2 | ⬜ | | |
| LR-PROF-NONRAW | Profiles for non-raw files | P0 | ✅ | `cmd:develop.profile` | same looks apply to JPEG/TIFF |
diff --git a/docs/whats-new.md b/docs/whats-new.md
index 00480132..d55c91d0 100644
--- a/docs/whats-new.md
+++ b/docs/whats-new.md
@@ -153,6 +153,12 @@
now develop from the raw data instead of the camera's embedded JPEG, so a B&W or other picture style set in the
camera no longer gets baked in. Photos already imported as "preview only" switch over on Reload. (Files that
Nikon splits into two differently compressed halves still use the preview for now.)
+- Panasonic and Leica raws (RW2, RWL and the older RAW files, DMC-LX1 to DC-S1R II) develop from the raw data in
+ every format the cameras write. Most bodies opened as preview only before: GH1–GH5, the G, GX, GF, GM, FZ, LX and
+ TZ/ZS series, Leica D-Lux, V-Lux and C-Lux, and the GH6, GH7, G9 II, S5 II and S9 generation. They start from a
+ colour and tone look fitted to the camera's own JPEG, as Nikon and Sony raws do, framed in the aspect ratio set in
+ the camera; mapped-out sensor defects are filled in. `.rwl` and `.raw` files are imported too. Photos already
+ imported as "preview only" switch over on Reload.
- Sony ARWs from before about 2017 (RX100, RX100 II–V, RX10, NEX, SLT, ILCE-6000, A7 / A7 II / A7R II and their
siblings) no longer open bright green (issue #148): their as-shot white balance and black level are read from the
file (Sony stores them only in scrambled maker-note data on these bodies), the few columns of padding at the right
diff --git a/flake.lock b/flake.lock
new file mode 100644
index 00000000..5c94f166
--- /dev/null
+++ b/flake.lock
@@ -0,0 +1,45 @@
+{
+ "nodes": {
+ "craft-fonts": {
+ "flake": false,
+ "locked": {
+ "lastModified": 1791295400,
+ "narHash": "sha256-e+6HpOYoAFTh6mNfGBBw2njnBzcE8W8+Jybv1+dKOCA=",
+ "owner": "storytold",
+ "repo": "craft-fonts",
+ "rev": "abb83316d96aa59c1cf64784289e378fe9fa5695",
+ "type": "github"
+ },
+ "original": {
+ "owner": "storytold",
+ "repo": "craft-fonts",
+ "rev": "abb83316d96aa59c1cf64784289e378fe9fa5695",
+ "type": "github"
+ }
+ },
+ "nixpkgs": {
+ "locked": {
+ "lastModified": 1791260994,
+ "narHash": "sha256-Miqqk/ammqnTUxaoCyvtwoPeLbI1ksFyLxi/CazZpWY=",
+ "owner": "NixOS",
+ "repo": "nixpkgs",
+ "rev": "151fa4e8ddfdd8dd25d945ad94ed54a13de9f6e4",
+ "type": "github"
+ },
+ "original": {
+ "owner": "NixOS",
+ "ref": "nixos-unstable",
+ "repo": "nixpkgs",
+ "type": "github"
+ }
+ },
+ "root": {
+ "inputs": {
+ "craft-fonts": "craft-fonts",
+ "nixpkgs": "nixpkgs"
+ }
+ }
+ },
+ "root": "root",
+ "version": 7
+}
diff --git a/flake.nix b/flake.nix
new file mode 100644
index 00000000..6869abf4
--- /dev/null
+++ b/flake.nix
@@ -0,0 +1,115 @@
+{
+ description = "LightCraft — photo library and non-destructive raw developer, in pure Rust";
+
+ inputs = {
+ # LightCraft needs edition 2024 and rustc ≥ 1.90, so the flake tracks unstable. Consumers on a
+ # release channel can retarget it: `inputs.lightcraft.inputs.nixpkgs.follows = "nixpkgs";`
+ nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
+
+ # Optional build input of the repository: the CJK fonts (BIZ UDPGothic, BIZ UDMincho, Noto Sans CJK SC,
+ # Shippori Mincho) that `crates/engine/build.rs` embeds when `CRAFT_FONTS_DIR` is set. Official
+ # releases always build with them; without them everything works but Japanese text has no
+ # glyphs. Pinned to the revision the release workflow uses — bump deliberately
+ # (`nix flake update craft-fonts`); craftrules standards/fonts.md.
+ craft-fonts = {
+ url = "github:storytold/craft-fonts/abb83316d96aa59c1cf64784289e378fe9fa5695";
+ flake = false;
+ };
+ };
+
+ outputs =
+ {
+ self,
+ nixpkgs,
+ craft-fonts,
+ }:
+ let
+ inherit (nixpkgs) lib;
+
+ # No x86_64-darwin: the pinned nixpkgs (26.11, "nixos-unstable") dropped support for it. On
+ # an Intel Mac, retarget nixpkgs at a release that still has it, or install the .dmg from the
+ # GitHub release.
+ systems = [
+ "x86_64-linux"
+ "aarch64-linux"
+ "aarch64-darwin"
+ ];
+
+ # The AppStream metadata carries a release date. The flake source's own commit date is
+ # reproducible, unlike "now"; a tarball or dirty tree without one falls back to the epoch.
+ buildDate =
+ let
+ d = self.lastModifiedDate or "";
+ in
+ if builtins.stringLength d >= 8 then
+ "${builtins.substring 0 4 d}-${builtins.substring 4 2 d}-${builtins.substring 6 2 d}"
+ else
+ "1970-01-01";
+
+ packageArgs = {
+ inherit craft-fonts buildDate;
+ };
+
+ forAllSystems = f: lib.genAttrs systems (system: f (import nixpkgs { inherit system; }) system);
+ in
+ {
+ packages = forAllSystems (
+ pkgs: _:
+ let
+ lightcraft = pkgs.callPackage ./nix/package.nix packageArgs;
+ in
+ {
+ inherit lightcraft;
+ default = lightcraft;
+ }
+ );
+
+ # `nixpkgs.overlays = [ inputs.lightcraft.overlays.default ]` → `pkgs.lightcraft` everywhere.
+ overlays.default = final: _prev: {
+ lightcraft = final.callPackage ./nix/package.nix packageArgs;
+ };
+
+ devShells = forAllSystems (
+ pkgs: system:
+ let
+ lightcraft = self.packages.${system}.lightcraft;
+ in
+ {
+ default = pkgs.mkShell (
+ {
+ # The package's own native inputs, so `cargo build` in the shell needs no extra setup.
+ inputsFrom = [ lightcraft ];
+ packages = with pkgs; [
+ cargo
+ rustc
+ clippy
+ rustfmt
+ rust-analyzer
+ ];
+ # `cargo run` dlopens the Vulkan loader and libxkbcommon like the packaged binary.
+ LD_LIBRARY_PATH = lib.makeLibraryPath lightcraft.passthru.runtimeLibs;
+ RUST_BACKTRACE = "1";
+ }
+ // lib.optionalAttrs (craft-fonts != null) {
+ # CJK glyphs in `cargo run` too; build.rs reads this as CRAFT_FONTS_DIR.
+ CRAFT_FONTS_DIR = toString craft-fonts;
+ }
+ );
+ }
+ );
+
+ checks = forAllSystems (
+ pkgs: system: {
+ lightcraft = self.packages.${system}.lightcraft;
+
+ # `nix fmt` formats the flake; this keeps it formatted.
+ formatting = pkgs.runCommand "lightcraft-nix-formatting" { nativeBuildInputs = [ pkgs.nixfmt ]; } ''
+ nixfmt --check ${./flake.nix} ${./nix}
+ touch $out
+ '';
+ }
+ );
+
+ formatter = forAllSystems (pkgs: _: pkgs.nixfmt);
+ };
+}
diff --git a/nix/package.nix b/nix/package.nix
new file mode 100644
index 00000000..b364b5c4
--- /dev/null
+++ b/nix/package.nix
@@ -0,0 +1,206 @@
+# LightCraft — the desktop app (`lightcraft`) and the CLI/MCP server (`lightcraft-cli`), built from
+# this repository:
+#
+# nix build # → ./result/bin/lightcraft, ./result/bin/lightcraft-cli
+#
+# Kept in step with the release workflow (.github/workflows/release.yml) and packaging/env.sh:
+# * `Cargo.lock` drives the dependencies (`cargoLock`), so there is no vendorHash to bump — only
+# crates.io, no git sources;
+# * `CRAFT_FONTS_DIR` embeds the CJK fonts from storytold/craft-fonts (the `craft-fonts`
+# flake input); without it everything builds and runs, but Japanese and Chinese text have no glyphs;
+# * the desktop file, hicolor icons and AppStream metadata are the same files the .deb/.rpm ship
+# (packaging/linux/), so `apt` and NixOS users see one identical LightCraft;
+# * `doCheck` runs `cargo test --workspace`, what `cargo xtask ci` runs. `nix build` runs it too;
+# `pkgs.lightcraft.overrideAttrs { doCheck = false; }` skips it for a faster, build-only install.
+#
+# Everything in the product is pure Rust (no C/C++ dependencies), so this needs no build system
+# beyond cargo plus the windowing headers winit's build scripts look for.
+{
+ lib,
+ stdenv,
+ rustPlatform,
+ pkg-config,
+ makeWrapper,
+ # Windowing/GPU libraries: needed at build time by the crates that link them, at run time by the
+ # ones that open them with dlopen() (see `runtimeLibs`), and by the dev shell (flake.nix).
+ wayland,
+ wayland-scanner,
+ libxkbcommon,
+ libx11,
+ libxcb,
+ libxcursor,
+ libxi,
+ libxrandr,
+ vulkan-loader,
+ # Validates the desktop file and the AppStream metadata we install (as CI does).
+ desktop-file-utils,
+ appstream,
+ # Adds the driver search path (/run/opengl-driver/lib) to the binaries' RPATH so wgpu finds the
+ # installed Vulkan driver on NixOS. Optional: harmless on nixpkgs without it.
+ autoAddDriverRunpath ? null,
+ # storytold/craft-fonts checkout (flake input). `null` builds without CJK glyphs.
+ craft-fonts ? null,
+ # Release date for the AppStream metadata (YYYY-MM-DD); the flake derives it from its own commit.
+ buildDate ? "1970-01-01",
+}:
+
+let
+ root = ../.;
+
+ # The version lives in exactly one place: `[workspace.package] version` in the root Cargo.toml.
+ version = (builtins.fromTOML (builtins.readFile (root + "/Cargo.toml"))).workspace.package.version;
+
+ isLinux = stdenv.hostPlatform.isLinux;
+
+ # AppStream id: also the desktop file name, and the icon name in share/icons/hicolor.
+ appId = "ai.storyteller.lightcraft";
+
+ # Libraries the binaries open at run time with dlopen(): nothing links them, so no RPATH points at
+ # them — winit loads libxkbcommon/libxcb, wgpu the Vulkan loader (NixOS patches that loader to
+ # find the drivers in /run/opengl-driver). The wrapper below points LD_LIBRARY_PATH here.
+ runtimeLibs = lib.optionals isLinux [
+ libxkbcommon
+ wayland
+ libx11
+ libxcb
+ libxcursor
+ libxi
+ libxrandr
+ vulkan-loader
+ ];
+
+ # The two native binaries. The wasm app (apps/lightcraft-web) is built by `cargo xtask web`, not
+ # here; xtask is tooling.
+ binaries = [
+ "lightcraft"
+ "lightcraft-cli"
+ ];
+in
+rustPlatform.buildRustPackage {
+ pname = "lightcraft";
+ inherit version;
+
+ src = lib.cleanSourceWith {
+ src = lib.cleanSource root;
+ # The Nix files are not part of the build: editing them must not rebuild any Rust.
+ filter =
+ path: _type:
+ !(builtins.elem (baseNameOf path) [
+ "flake.nix"
+ "flake.lock"
+ "nix"
+ ]);
+ };
+
+ cargoLock.lockFile = root + "/Cargo.lock";
+
+ # Both binaries, as the .deb/.rpm install them (`packaging/linux/package.sh`).
+ cargoBuildFlags = [
+ "--locked"
+ "-p"
+ "lightcraft"
+ "-p"
+ "lightcraft-cli"
+ ];
+
+ # `cargo xtask ci` runs `cargo test --workspace`. Tests that need what the sandbox cannot have
+ # skip themselves with a note: corpus/ (gitignored, `cargo xtask corpus --download`) and the GPU
+ # equivalence tests (no adapter).
+ cargoTestFlags = [ "--workspace" ];
+
+ # One test thread at a time (RUST_TEST_THREADS=1). `lightcraft-catalog`'s `tests_lock` tests share
+ # a process-global resource: `flock` is inherited across `fork`, so a child process that another
+ # test forked keeps the parent's just-released library lock alive until its `execve` closes the
+ # (CLOEXEC) fd — long enough that the sibling test sees a spurious `LockError::InUse`, on ~30 % of
+ # runs on a many-core machine. Serializing the tests keeps `nix build` deterministic without
+ # touching the repository. The underlying window (any `fork` in a process holding the library
+ # lock — the app spawns `xdg-open`/`open`/`explorer` and `sysctl`) is a product-level question:
+ # `LibraryLock::acquire` could retry on `WouldBlock` for a few hundred ms before reporting InUse.
+ dontUseCargoParallelTests = true;
+
+ strictDeps = true;
+
+ nativeBuildInputs = [
+ pkg-config
+ ]
+ ++ lib.optionals isLinux [
+ makeWrapper
+ wayland-scanner
+ desktop-file-utils
+ appstream
+ ]
+ ++ lib.optionals (isLinux && autoAddDriverRunpath != null) [ autoAddDriverRunpath ];
+
+ buildInputs = lib.optionals isLinux [
+ wayland
+ libxkbcommon
+ libx11
+ libxcb
+ libxcursor
+ libxi
+ libxrandr
+ ];
+
+ env = lib.optionalAttrs (craft-fonts != null) {
+ CRAFT_FONTS_DIR = toString craft-fonts;
+ # A malformed font input must fail the build rather than silently ship without glyphs.
+ CRAFT_FONTS_REQUIRED = "1";
+ };
+
+ postInstall = ''
+ # Desktop integration: byte-for-byte the files the .deb/.rpm/.AppImage install.
+ install -Dm644 packaging/linux/${appId}.desktop $out/share/applications/${appId}.desktop
+ install -Dm644 packaging/linux/${appId}.mime.xml $out/share/mime/packages/${appId}.xml
+ cp -r assets/app-icon/hicolor $out/share/icons/
+ install -d $out/share/metainfo
+ substitute packaging/linux/${appId}.metainfo.xml.in \
+ $out/share/metainfo/${appId}.metainfo.xml \
+ --subst-var-by VERSION ${version} \
+ --subst-var-by DATE ${buildDate}
+
+ # Licences for everything embedded in the binaries: Inter, the app icon, each craft font.
+ doc=$out/share/doc/lightcraft
+ install -Dm644 -t "$doc" \
+ README.md LICENSE-MIT LICENSE-APACHE NOTICE \
+ assets/ATTRIBUTION.md assets/fonts/OFL-Inter.txt
+ ''
+ + lib.optionalString (craft-fonts != null) ''
+ for lic in ${toString craft-fonts}/fonts/*/OFL.txt; do
+ install -Dm644 "$lic" "$doc/OFL-$(basename "$(dirname "$lic")").txt"
+ done
+ ''
+ + lib.optionalString isLinux ''
+ desktop-file-validate $out/share/applications/${appId}.desktop
+ XDG_CACHE_HOME=$TMPDIR/.cache appstreamcli validate --no-net \
+ $out/share/metainfo/${appId}.metainfo.xml
+
+ # makeWrapper keeps the binaries working when the libs above are only dlopen'd.
+ for bin in ${lib.concatStringsSep " " (map (b: "$out/bin/" + b) binaries)}; do
+ wrapProgram "$bin" --prefix LD_LIBRARY_PATH : "${lib.makeLibraryPath runtimeLibs}"
+ done
+ '';
+
+ passthru = {
+ # Shared with the dev shell so `cargo run` inside `nix develop` finds the same dlopen'd libs.
+ inherit runtimeLibs;
+ };
+
+ meta = {
+ description = "Photo library and non-destructive raw developer";
+ longDescription = ''
+ LightCraft organises a photo library and develops raw files non-destructively: masks,
+ presets, colour grading, local adjustments and batch export, driven by the same command
+ layer as its UI. It ships with lightcraft-cli, a headless renderer, command runner and MCP
+ server for AI agents.
+ '';
+ homepage = "https://getartcraft.com/apps/lightcraft";
+ license = with lib.licenses; [
+ mit
+ asl20
+ ];
+ sourceProvenance = with lib.sourceTypes; [ fromSource ];
+ platforms = lib.platforms.linux ++ lib.platforms.darwin;
+ # `nix run` and the desktop entry both start the GUI.
+ mainProgram = "lightcraft";
+ };
+}
diff --git a/packaging/linux/ai.storyteller.lightcraft.desktop b/packaging/linux/ai.storyteller.lightcraft.desktop
index 5ea3af99..f827c582 100644
--- a/packaging/linux/ai.storyteller.lightcraft.desktop
+++ b/packaging/linux/ai.storyteller.lightcraft.desktop
@@ -9,4 +9,4 @@ Terminal=false
Categories=Graphics;2DGraphics;RasterGraphics;Photography;
Keywords=photo;raw;camera;library;develop;
StartupWMClass=ai.storyteller.lightcraft
-MimeType=image/jpeg;image/png;image/tiff;image/webp;image/x-adobe-dng;image/x-sony-arw;image/x-canon-cr2;image/x-canon-cr3;image/x-nikon-nef;image/x-fuji-raf;image/x-olympus-orf;image/x-panasonic-rw2;image/x-pentax-pef;
+MimeType=image/jpeg;image/png;image/tiff;image/webp;image/x-adobe-dng;image/x-sony-arw;image/x-canon-cr2;image/x-canon-cr3;image/x-nikon-nef;image/x-fuji-raf;image/x-olympus-orf;image/x-panasonic-rw2;image/x-panasonic-rw;image/x-pentax-pef;
diff --git a/xtask/src/assets.rs b/xtask/src/assets.rs
index 7a932945..1680e08c 100644
--- a/xtask/src/assets.rs
+++ b/xtask/src/assets.rs
@@ -9,7 +9,8 @@ use std::process::Command;
/// Extensions treated as assets (lower-case).
const ASSET_EXT: &[&str] = &[
"png", "jpg", "jpeg", "gif", "webp", "avif", "svg", "ico", "icns", "bmp", "ttf", "otf", "woff", "woff2", "wav", "mp3", "ogg", "flac", "mp4",
- "mov", "tif", "tiff", "dng", "cr2", "cr3", "nef", "arw", "raf", "orf", "rw2", "pef", "heic", "jxl", "psd", "cube", "icc", "icm", "xmp",
+ "mov", "tif", "tiff", "dng", "cr2", "cr3", "nef", "arw", "raf", "orf", "rw2", "rwl", "raw", "pef", "heic", "jxl", "psd", "cube", "icc", "icm",
+ "xmp",
];
/// Adobe-specific formats we never ship (camera/lens profiles, Lightroom templates).
const FORBIDDEN_EXT: &[&str] = &["dcp", "lcp", "lrtemplate", "lrcat", "lrsmcol", "aco", "ase", "abr"];
diff --git a/xtask/src/layers.rs b/xtask/src/layers.rs
index 28c99462..28d52677 100644
--- a/xtask/src/layers.rs
+++ b/xtask/src/layers.rs
@@ -38,6 +38,7 @@ pub const TABLE: &[(&str, Class)] = &[
("raster", Class::Layer(0)),
("tiff", Class::Layer(0)),
("sysmem", Class::Layer(0)),
+ ("fetch", Class::Layer(0)),
("raw", Class::Layer(1)),
("codecs", Class::Layer(1)),
("meta", Class::Layer(1)),
diff --git a/xtask/src/main.rs b/xtask/src/main.rs
index 80eb7a2f..d8c4fc2d 100644
--- a/xtask/src/main.rs
+++ b/xtask/src/main.rs
@@ -360,11 +360,22 @@ const RAW_SAMPLES: &[(&str, &str)] = &[
"raf-fuji-xt20-compressed.raf",
"https://raw.pixls.us/getfile.php/1178/nice/Fujifilm%20-%20X-T20%20-%2014bit%2014bit%20compressed%20%283:2%29.RAF",
),
+ ("raw-panasonic-fz50.raw", "https://raw.pixls.us/getfile.php/2234/nice/Panasonic%20-%20DMC-FZ50%20-%204:3.RAW"),
+ ("raw-panasonic-fz8.raw", "https://raw.pixls.us/getfile.php/2282/nice/Panasonic%20-%20DMC-FZ8%20-%204:3.RAW"),
+ ("rw2-panasonic-fz1000m2-4x3.rw2", "https://raw.pixls.us/getfile.php/4706/nice/Panasonic%20-%20DC-FZ10002%20-%204:3.RW2"),
("rw2-panasonic-g9-b.rw2", "https://raw.pixls.us/getfile.php/2348/nice/Panasonic%20-%20DC-G9%20-%204:3.RW2"),
("rw2-panasonic-g9.rw2", "https://raw.pixls.us/getfile.php/2585/nice/Panasonic%20-%20DC-G9%20-%204:3.RW2"),
+ ("rw2-panasonic-gh1.rw2", "https://raw.pixls.us/getfile.php/1323/nice/Panasonic%20-%20DMC-GH1%20-%204:3.RW2"),
("rw2-panasonic-gh5.rw2", "https://raw.pixls.us/getfile.php/1517/nice/Panasonic%20-%20DC-GH5%20-%204:3.RW2"),
+ ("rw2-panasonic-gh5m2.rw2", "https://raw.pixls.us/getfile.php/5082/nice/Panasonic%20-%20DC-GH5M2%20-%204:3.RW2"),
("rw2-panasonic-gh5s.rw2", "https://raw.pixls.us/getfile.php/2603/nice/Panasonic%20-%20DC-GH5S%20-%204:3.RW2"),
+ ("rw2-panasonic-gh6.rw2", "https://raw.pixls.us/getfile.php/5876/nice/Panasonic%20-%20DC-GH6%20-%204:3.RW2"),
("rw2-panasonic-gx80.rw2", "https://raw.pixls.us/getfile.php/1569/nice/Panasonic%20-%20DMC-GX80%20-%204:3.RW2"),
+ ("rw2-panasonic-s1.rw2", "https://raw.pixls.us/getfile.php/3038/nice/Panasonic%20-%20DC-S1%20-%203:2.RW2"),
+ ("rw2-panasonic-s5-format7.rw2", "https://raw.pixls.us/getfile.php/6339/nice/Panasonic%20-%20DC-S5%20-%203:2.RW2"),
+ ("rw2-panasonic-s5m2.rw2", "https://raw.pixls.us/getfile.php/7790/nice/Panasonic%20-%20DC-S5M2%20-%2014bit%20%283:2%29.RW2"),
+ ("rw2-panasonic-s9.rw2", "https://raw.pixls.us/getfile.php/7702/nice/Panasonic%20-%20DC-S9%20-%203:2.RW2"),
+ ("rwl-leica-dlux7.rwl", "https://raw.pixls.us/getfile.php/4204/nice/Leica%20-%20D-Lux%207%20-%204:3.RWL"),
];
fn cmd_corpus(download: bool) -> Result<(), String> {