shifathosn
/
lightcraft
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge pull request #692 from storytold/claude/integrate-batch-25 

Integrate #573, #498, #602, #601, #560, #555, #480, #590
This commit is contained in:
Brandon Thomas 2026-10-10 05:14:58 -04:00 committed by GitHub
parent 5436951299 977f912df2
commit b70ae1741e
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
89 changed files with 4576 additions and 400 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
CLAUDE.md
View File

@ -55,6 +55,10 @@ of it. Full standard: `../craftrules/standards/never-crash.md`
- **Pure Rust** in the product. No C/C++ dependencies.
- **Layering** (`plan/architecture.md` §3, enforced by `cargo xtask layers`): nothing below L5 depends on egui/eframe/winit/rfd.
- **Everything is a command** (`crates/engine`): id, label, menu path, shortcut, params, enabled(), run(). UI, CLI, control channel and MCP all dispatch by id. Every slider is a `develop` control spec.
- **Long-running work shows in the activity stack** ([`docs/background-tasks.md`](docs/background-tasks.md)): anything
that runs off the UI thread and can take more than a moment holds a `TaskGuard` from `session.activity.start(kind,
label, cancel)` for its whole run. It adopts the cancel flag the job already checks (`Cancel::Flag`), or uses
`Cancel::No` when the job can't stop. No progress window, panel or progress toast of its own; its tests assert the row.
- **Resolution independence:** settings use normalized image coordinates and relative radii; previews and exports must match.
- **Quality gates** before every commit: `cargo xtask ci` (fmt, clippy -D warnings, tests, layers, assets, wasm).
- **Commits:** one task id per commit (`M2.3: local Laplacian highlights/shadows`). Only green states. End messages with the attribution line required by the environment.

1
Cargo.lock generated
View File

@ -3352,6 +3352,7 @@ dependencies = [
"lightcraft-engine",
"lightcraft-faces",
"lightcraft-geom",
"lightcraft-merge",
"lightcraft-meta",
"lightcraft-pipeline",
"lightcraft-preview",

7
NOTICE
View File

@ -15,6 +15,13 @@ list (author, source, license) is in assets/ATTRIBUTION.md.
Inter (assets/fonts/Inter-*.ttf) — SIL Open Font License 1.1. See assets/fonts/OFL-Inter.txt.
Showcase photographs by Ansel Adams, Dorothea Lange and NASA (docs/images/) — public domain. See assets/ATTRIBUTION.md.
crates/raw/src/spectral_table.rs (lightcraft-raw) holds camera colour matrices derived from the
camera spectral sensitivities, training reflectances and model alias list of rawtoaces-data
(https://github.com/AcademySoftwareFoundation/rawtoaces-data, v1.1.0), an Academy Software
Foundation project licensed under the Apache License 2.0, and from the CIE 1931 colour-matching
functions it ships (CIE, CC-BY-4.0). The matrices were fitted by the LightCraft contributors
(2026) with tools/spectral_camera_table.py. See assets/ATTRIBUTION.md.
crates/segment (lightcraft-segment) ports the SAM 3 model code of Hugging Face Transformers
(https://github.com/huggingface/transformers, src/transformers/models/sam3 and sam3_tracker),
Copyright The HuggingFace Team and Meta Platforms, Inc., licensed under the Apache License 2.0.

12
ROADMAP.md
View File

@ -24,7 +24,7 @@ duplicate Local entries #22, black GPU exports on an Intel iGPU #78).
|---|---:|---|---|
| **Feature checklist** | 79.7% | P0 core and P1 nearly complete: import (Add / Copy / Move, templates, devices), library, grid/loupe/compare/survey, every Edit slider, curves, colour grading, masking tools, crop/Upright, heal/clone, presets/profiles, versions/history, sync, export, menus, shortcuts | P0: full-resolution zoom detail; P1: lens-profile database, content-aware fill (patch synthesis), video playback/trim |
| **RAW coverage** (formats people shoot) | ~55% | DNG (all kinds), CR2, CR3 lossless Bayer and version 0x100/0x200 C-RAW (M50/R100/R8 pixel-exact corpus), ARW, NEF (uncompressed + Huffman lossless/lossy), RAF (uncompressed, lossless/lossy compressed Bayer and X-Trans), uncompressed ORF, uncompressed Samsung SRW (NX5 to NX1100, EX1, WB2000), RW2 / RWL / Panasonic RAW (every raw format, checked on 178 files from 118 Panasonic and Leica bodies), PEF; every container's embedded preview (incl. CR3) | **CR3** (every Canon since ~2018), compressed ORF, compressed Samsung SRW (NX1, NX30, NX300, NX500, NX2000, NX3000, NX3300, NX mini), Nikon lossy-after-split, Canon sRAW, HEIC/AVIF. NEFs labelled compressed but stored uncompressed (Z 6 packed 14-bit, D850 12-bit uncompressed). Nikon High Efficiency NEF (HE / HE★: Z 8, Z 9, Z 6III, Z f; JPEG XS-based, preview only, #193). Per-model verification is thin (~55 corpus files vs >1,000 models) |
| **Colour & image quality** | ~55–65% | Pipeline is complete and fast; GPU path CPU-exact within 1/255 | **No measured camera calibration database**: ARW, NEF, RW2, RAF, CR3, CR2 and PEF have guarded camera-JPEG colour estimates, with bundled ILCE-7CR, ILCE-7M4, X-H2S and X-T4 profiles (docs/camera-preview-colour.md); other non-DNG raws and rejected estimates use a neutral matrix. Colour fidelity remains incomplete. No lens-profile database. No measured fidelity against Lightroom (tone, highlights, texture/clarity, NR, sharpening are tuned by eye) |
| **Colour & image quality** | ~55–65% | Pipeline is complete and fast; GPU path CPU-exact within 1/255 | **No measured camera calibration database**: ARW, NEF, RW2, RAF, CR3, CR2 and PEF have guarded camera-JPEG colour estimates, with bundled ILCE-7CR, ILCE-7M4, X-H2S and X-T4 profiles (docs/camera-preview-colour.md); 52 models whose JPEG fit is rejected start from matrices fitted to measured spectral sensitivities; other non-DNG raws and rejected estimates use a neutral matrix. Colour fidelity remains incomplete. No lens-profile database. No measured fidelity against Lightroom (tone, highlights, texture/clarity, NR, sharpening are tuned by eye) |
| **AI & computational** | ~15–20% | Assisted culling (focus, bursts), auto tone, HDR/panorama merge; subject/sky/background masks as classical heuristics, or the photo's own segmentation mattes when it ships them (DNG semantic masks, e.g. iPhone ProRAW's sky matte); non-destructive Bayer RAW denoise with opt-in weights on pure-Rust CPU/GPU; **faces (MVP, [docs/faces.md](docs/faces.md))**: a pure-Rust detector (a 232 KB opt-in download), opt-in recognition models (one-click download or add a file; they run on the shared pure-Rust CPU interpreter; pinned SFace/AuraFace embeddings agree with an independent reference runtime on 40 public crops; one-thread medians 34–36 / 233–252 ms (AuraFace remains expensive)), a background scan that finds and embeds faces, name suggestions, a person's page with look-alike faces to confirm, an Unnamed faces section (look-alikes together, select a group and name it at once) | Real segmentation masks (subject, sky, people, objects, landscape, depth), denoise model policy / linear RGB / quality verification, super resolution, lens blur, generative remove, natural-language search; for faces: clusters with separators between groups, small faces in large photos (the detector looks at 640 px), thresholds calibrated on live faces, writing face regions back to XMP. **Blocked on a model strategy** (licensable weights or our own training; pure-Rust inference is feasible) |
| **Workflow & library** | ~85% (single machine) | Robust catalog (journal + snapshots, background compaction, crash-tested), 85k-photo libraries stay responsive, Local browsing with automatic cleanup, XMP interop, keywords, smart albums, Move import | Opening an 85k library takes 1.7–4.7 s; no cloud sync (out of scope), no tablet companion (#74, roadmap), shared albums, publish services, tethering |
| **Classic modules** | ~30% | Geotagging from GPX track logs, soft proofing (partial), slideshow (basic), paginated contact-sheet PDF export (A4/Letter, layout/captions, current edits, 150 dpi sRGB) | **Map view, Book, full Print module, Slideshow module, Web, publish services**: contact sheets work, custom print packages and printer colour management remain missing |
@ -36,7 +36,7 @@ duplicate Local entries #22, black GPU exports on an Intel iGPU #78).
| User | Readiness | What blocks them |
|---|---:|---|
| JPEG / DNG shooter, single machine | ~85% | Fidelity polish, AI masks |
| Nikon / Sony / Panasonic / older-Canon raw shooter | ~65% | Camera colour fidelity and coverage (ARW, NEF and RW2 preview estimates are only a starting point); Panasonic compacts and kit zooms also miss their embedded distortion correction |
| Nikon / Sony / Panasonic / older-Canon raw shooter | ~65% | Camera colour fidelity and coverage (ARW, NEF and RW2 preview estimates are only a starting point); embedded distortion is supported for Panasonic / Leica RW2 and the Sony ILCE-7RM4A (Bayer ARW, 24–105 / 200–600 tested); other lens metadata needs coverage and validation |
| Fujifilm raw shooter | ~65% | Uncompressed and lossless/lossy compressed RAF decoded; camera colour calibration remains missing. Verified on 13 bodies, including 14/16-bit GFX; see `docs/raf-compression.md` |
| Canon CR3 / Olympus compressed-raw shooter | ~45% | CR3 lossless Bayer and version 0x100/0x200 C-RAW develop from sensor data (M50, R100, R8 verified pixel for pixel); other CRX variants and compressed ORF open as embedded JPEG previews. Camera colour and broader model verification remain |
| Lightroom Classic power user | ~45% | Print, Book, Map, publish, tethering |
@ -49,7 +49,8 @@ Priorities, in order. Each points at tracker rows in [`docs/parity.md`](docs/par
1. **Camera colour calibration of our own** (LR-PROF-CAMERACOLOR, P0): fit each camera to its own embedded JPEG, use
matrices the files carry themselves, then chart shots. Sony ARW, Nikon NEF, Panasonic RW2, Fujifilm RAF, Canon CR3/CR2,
Pentax PEF, Samsung SRW and Olympus ORF have the file-local fit (matrix + tone curve from their own JPEG; CR2 and PEF
since issue #310, not yet checked on the corpus); then validate fidelity.
since issue #310, not yet checked on the corpus); 52 models whose fit is rejected fall back to matrices fitted to
measured spectral sensitivities (rawtoaces-data); then validate fidelity.
2. **Raw formats, clean-room** (LR-IMP-FORMATS, P0): Remaining **unverified CR3 CRX** variants, compressed ORF, NEF
lossy-after-split, sRAW. Decided 2026-10-05: write our own decoders from prose descriptions (never decoder source,
no LGPL dependency); compressed NEF (#86) is the template.
@ -57,7 +58,8 @@ Priorities, in order. Each points at tracker rows in [`docs/parity.md`](docs/par
checked for plausible colour; fix per-model bugs (#85).
4. **Render fidelity suite** (LR-BEHAV-RENDER-FIDELITY, P1): measure our output against Lightroom on the same CC0 raws
(references stay in the local `plan/`), then tune against the numbers.
5. **Lens profiles of our own** (LR-EDIT-OPTICS-PROFILE, P1).
5. **Lens profiles of our own** (LR-EDIT-OPTICS-PROFILE, P1): DNG, Panasonic / Leica RW2 and Sony ILCE-7RM4A Bayer ARW
embedded distortion are supported; broaden validation and decode remaining vignetting / lateral CA metadata.
6. **AI model strategy** (maintainer decision): which permissively licensed models (or our own training) for
segmentation masks and denoise. Bayer denoise inference is implemented in pure Rust with an opt-in GPL model; release policy and independently licensed weights remain open. Unblocks M12 and Enhance. Face detection and recognition run on our own checked Rust engine. Object / Describe masks now
run Meta's SAM 3 in pure Rust (optional, user-consented download under the non-OSI SAM License; CDN mirrors still
@ -190,7 +192,7 @@ packed formats 2, 5 and 7, the 16-bit words of the 2005–2007 bodies; all recov
Not decoded yet — preview only (no permissively licensed description; black-box analysis incomplete):
- **Olympus compressed ORF**, **Canon CR3 unverified CRX variants** (M11.1), **Canon sRAW/mRAW**.
**Camera colour matrices:** ARW, NEF, RW2, RAF, CR3, CR2, PEF, Samsung SRW and Olympus ORF files can use guarded, separate chromaticity and tone estimates from their own embedded JPEG (see `docs/camera-preview-colour.md`); this is a per-file camera-look estimate with relative WB, not measured calibration or absolute-Kelvin WB. Other non-DNG raws and rejected fits use the documented neutral fallback (camera RGB ≈ linear sRGB, flagged
**Camera colour matrices:** ARW, NEF, RW2, RAF, CR3, CR2, PEF, Samsung SRW and Olympus ORF files can use guarded, separate chromaticity and tone estimates from their own embedded JPEG (see `docs/camera-preview-colour.md`); this is a per-file camera-look estimate with relative WB, not measured calibration or absolute-Kelvin WB. 52 models whose fit is rejected (or that have no usable JPEG) start from matrices fitted to their measured spectral sensitivities instead (rawtoaces-data, Apache-2.0; `crates/raw/src/spectral.rs`). Other non-DNG raws and rejected fits use the documented neutral fallback (camera RGB ≈ linear sRGB, flagged
`matrix_is_fallback`) with the file's as-shot white-balance multipliers. Clean sources to evaluate next: manufacturer
matrices stored in the files themselves (Olympus ImageProcessing `ColorMatrix`, Pentax equivalents; Panasonic RW2 files carry none) and our
own chart-based calibration (M11.4). Adobe matrices are never used.

1
assets/ATTRIBUTION.md
View File

@ -14,6 +14,7 @@ covers a whole directory), or if a licence file referenced in the Licence column
| `packaging/macos/dmg/` (`background.svg`, the `background.tiff` rendered from it by `generate.py`, and the `dmg-layout.DS_Store` layout file) | LightCraft macOS DMG window background | @XusBadia | original work for this repository, derived from the LightCraft app icon (`assets/app-icon/lightcraft.svg`, referenced via `<image>`, not copied); text is outlined Inter / JetBrains Mono glyph paths | MIT OR Apache-2.0 | 2026-10-08 | none (original); used for the Finder window of the macOS DMG, `packaging/macos/dmg/generate.py` renders it to `background.tiff` |
| `assets/camera-profiles/ILCE-7CR.json` | Sony ILCE-7CR camera colour profile (matrix + hue/saturation/value table), built into LightCraft | LightCraft contributors; source photographs supplied by @AlexChambers | original work: fitted by `lightcraft-cli calibrate` to 125 usable ILCE-7CR ARWs (of 131, September 2025 to September 2026, Standard creative style) and their in-camera JPEGs; aggregate colour statistics only, no image content; no Adobe or third-party profile data | MIT OR Apache-2.0 | 2026-10-09 | none |
| `assets/camera-profiles/X-H2S.json`, `assets/camera-profiles/X-T4.json` | Fujifilm camera colour profiles (matrix + hue/saturation/value table), built into LightCraft | LightCraft contributors; source photographs supplied by Sebastian Pfluegelmeier | original work: fitted by `lightcraft-cli calibrate` to 201 X-H2S and 545 X-T4 RAFs and their in-camera JPEGs; aggregate colour statistics only, no image content; no Adobe or third-party profile data | MIT OR Apache-2.0 | 2026-10-08 | none |
| `crates/raw/src/spectral_table.rs` | Camera colour matrices (DNG ColorMatrix1/2 and ForwardMatrix1/2 for Standard Illuminant A and D65) for 52 camera models, and the model alias list | Derived by LightCraft contributors from rawtoaces-data (Academy Software Foundation): camera spectral sensitivities measured by Weta Digital (44 cameras), scitech (7) and Jonathon Irons at the National Physical Laboratory (Nikon D5100); training reflectances; CIE 1931 colour-matching functions (CIE, CC-BY-4.0, as shipped in rawtoaces-data) | https://github.com/AcademySoftwareFoundation/rawtoaces-data, tag v1.1.0, commit 5cfa53707b053594b2d0d48e8019fb56e6582d6c | Apache-2.0 (`LICENSE-APACHE`); the colour-matching functions CC-BY-4.0 | 2026-10-08 | generated by `tools/spectral_camera_table.py`: a 3×3 matrix per illuminant fitted to the spectral data (no measurement copied verbatim); aliases copied from `data/aliases.json`; no Adobe or third-party camera matrix |
| `assets/camera-tone/sony-dro-off.json` | Sony global tone curve with the Dynamic Range Optimizer off (32 knots), built into LightCraft for ARWs shot with DRO on | LightCraft contributors; source photographs by raw.pixls.us contributors | original work: tone curve quantile-matched over the pooled pixel pairs of 12 CC0 raw.pixls.us ARWs (DRO off, Standard creative style; ILCE-3500, -5000, -6600, -7CR, -7M3 ×3, -7S, -9M2, ILCA-77M2, SLT-A57, NEX-F3) and their embedded camera JPEGs; the file lists each source URL and SHA-256 and how the curve was derived (see also `docs/camera-preview-colour.md`); aggregate tone statistics only, no image content; no Adobe or third-party profile data | MIT OR Apache-2.0 (sources CC0) | 2026-10-09 | none |
| `crates/scenes/` (generated images) | Procedural demo photographs | LightCraft contributors | original work (generated at runtime, no source imagery) | MIT OR Apache-2.0 | 2026-09-30 | n/a |
| `crates/ui-egui/src/icons.rs` | UI icons drawn as vector paths in code | LightCraft contributors | original work | MIT OR Apache-2.0 | 2026-09-30 | n/a |

391
crates/engine/src/activity.rs Normal file
View File

@ -0,0 +1,391 @@
//! Background tasks in flight (issue #345): imports, exports, preview builds, downloads, the face scan… Each one
//! registers here while it runs, so every frontend can show them in one place (the desktop app's activity stack) and
//! cancel the ones that can be cancelled (`activity.list`, `activity.cancel`).
//!
//! A job holds a [`TaskGuard`] for as long as it runs; dropping it — when the job ends, fails or its thread unwinds
//! from a panic — removes the row, so a dead worker never leaves a stuck bar. Jobs keep the cancel flag they already
//! check between files or photos: the guard adopts it ([`Cancel::Flag`]), so the stack's ✕, `activity.cancel` and the
//! job's own Cancel all set the same atomic. Numbers are worded by the frontend from `done`, `total` and [`Unit`]; a
//! detail set here is a name (a file, a model), never a sentence, so the frontend can translate everything else.
use std::sync::atomic::{AtomicBool, AtomicU8, AtomicU64, Ordering};
use std::sync::{Arc, Mutex, MutexGuard, PoisonError};
use serde::Serialize;
/// The longest label or detail kept, in characters (a file name can be anything).
const MAX_TEXT: usize = 200;
/// The background tasks in flight. Cheap to clone (one `Arc`), so worker threads can hold it.
#[derive(Clone, Default)]
pub struct Activity {
inner: Arc<Inner>,
}
#[derive(Default)]
struct Inner {
/// Oldest first.
tasks: Mutex<Vec<Arc<Entry>>>,
next: AtomicU64,
}
/// How a task can be cancelled.
pub enum Cancel {
/// Not at all (no ✕): the face scan, AI Denoise.
No,
/// With a flag of its own ([`TaskGuard::cancel_flag`]).
Yes,
/// With the job's existing flag.
Flag(Arc<AtomicBool>),
}
/// How the frontend words `done` / `total`: "3 of 25", "12 of 340 MB", "40 %".
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Serialize)]
#[serde(rename_all = "lowercase")]
pub enum Unit {
#[default]
Count,
Bytes,
Percent,
}
impl Unit {
fn from_u8(v: u8) -> Unit {
match v {
1 => Unit::Bytes,
2 => Unit::Percent,
_ => Unit::Count,
}
}
fn to_u8(self) -> u8 {
match self {
Unit::Count => 0,
Unit::Bytes => 1,
Unit::Percent => 2,
}
}
}
/// One task as the frontends see it (`activity.list`).
#[derive(Clone, Debug, PartialEq, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct TaskInfo {
pub id: u64,
/// A stable id: `import`, `export`, `scan`, `previews`, `smartPreviews`, `lightroom`, `merge`, `download`,
/// `faces`, `denoise`, `findMissing`.
pub kind: &'static str,
/// English; the frontend translates it.
pub label: String,
pub done: u64,
/// 0 while the amount of work isn't known yet (an indeterminate bar).
pub total: u64,
pub unit: Unit,
/// The file, photo or model being worked on, if any.
pub detail: String,
pub cancellable: bool,
/// Cancel was asked for; the job stops at its next check.
pub cancelling: bool,
pub age_ms: u64,
}
struct Entry {
id: u64,
kind: &'static str,
label: String,
started: web_time::Instant,
done: AtomicU64,
total: AtomicU64,
unit: AtomicU8,
detail: Mutex<String>,
cancel: Option<Arc<AtomicBool>>,
cancellable: AtomicBool,
}
fn lock<T>(m: &Mutex<T>) -> MutexGuard<'_, T> {
m.lock().unwrap_or_else(PoisonError::into_inner)
}
fn capped(text: &str) -> String {
text.chars().take(MAX_TEXT).collect()
}
impl Entry {
fn is_cancellable(&self) -> bool {
self.cancel.is_some() && self.cancellable.load(Ordering::Relaxed)
}
fn is_cancelled(&self) -> bool {
self.cancel.as_ref().is_some_and(|c| c.load(Ordering::Relaxed))
}
fn progress(&self, done: u64, total: u64) {
self.total.store(total, Ordering::Relaxed);
self.done.store(if total > 0 { done.min(total) } else { done }, Ordering::Relaxed);
}
fn detail(&self, text: &str) {
*lock(&self.detail) = capped(text);
}
fn info(&self) -> TaskInfo {
TaskInfo {
id: self.id,
kind: self.kind,
label: self.label.clone(),
done: self.done.load(Ordering::Relaxed),
total: self.total.load(Ordering::Relaxed),
unit: Unit::from_u8(self.unit.load(Ordering::Relaxed)),
detail: lock(&self.detail).clone(),
cancellable: self.is_cancellable(),
cancelling: self.is_cancelled(),
age_ms: u64::try_from(self.started.elapsed().as_millis()).unwrap_or(u64::MAX),
}
}
}
impl Activity {
/// Register a task; it is listed until the returned guard is dropped. `label` is English (the frontend
/// translates it).
pub fn start(&self, kind: &'static str, label: &str, cancel: Cancel) -> TaskGuard {
let id = self.inner.next.fetch_add(1, Ordering::Relaxed).saturating_add(1);
let cancel = match cancel {
Cancel::No => None,
Cancel::Yes => Some(Arc::new(AtomicBool::new(false))),
Cancel::Flag(flag) => Some(flag),
};
let entry = Arc::new(Entry {
id,
kind,
label: capped(label),
started: web_time::Instant::now(),
done: AtomicU64::new(0),
total: AtomicU64::new(0),
unit: AtomicU8::new(Unit::Count.to_u8()),
detail: Mutex::new(String::new()),
cancellable: AtomicBool::new(cancel.is_some()),
cancel,
});
lock(&self.inner.tasks).push(entry.clone());
TaskGuard { activity: self.clone(), entry }
}
/// Every task in flight, oldest first.
pub fn list(&self) -> Vec<TaskInfo> {
lock(&self.inner.tasks).iter().map(|e| e.info()).collect()
}
/// Ask task `id` to stop (it stops at its next check). An unknown id (the task may have just ended) or a task
/// that can't be cancelled is an error.
pub fn cancel(&self, id: u64) -> Result<(), String> {
let tasks = lock(&self.inner.tasks);
let entry = tasks.iter().find(|e| e.id == id).ok_or_else(|| format!("no task {id}"))?;
match (&entry.cancel, entry.is_cancellable()) {
(Some(flag), true) => {
flag.store(true, Ordering::Relaxed);
Ok(())
}
_ => Err(format!("{} can't be cancelled", entry.label)),
}
}
/// Ask every cancellable task to stop; returns how many were asked.
pub fn cancel_all(&self) -> usize {
let tasks = lock(&self.inner.tasks);
let mut n = 0;
for e in tasks.iter().filter(|e| e.is_cancellable()) {
if let Some(flag) = &e.cancel {
flag.store(true, Ordering::Relaxed);
n += 1;
}
}
n
}
/// Tasks that quitting would cut short: cancellable ones not already stopping.
pub fn running_cancellable(&self) -> Vec<TaskInfo> {
lock(&self.inner.tasks).iter().filter(|e| e.is_cancellable() && !e.is_cancelled()).map(|e| e.info()).collect()
}
}
/// A task's place in the list, held by whoever owns the job; dropping it removes the row.
pub struct TaskGuard {
activity: Activity,
entry: Arc<Entry>,
}
impl TaskGuard {
pub fn id(&self) -> u64 {
self.entry.id
}
/// `done` of `total` (`total` 0 = not known yet). `done` is kept at most `total`.
pub fn progress(&self, done: u64, total: u64) {
self.entry.progress(done, total);
}
/// The file, photo or model being worked on (a name, not a sentence).
pub fn detail(&self, text: &str) {
self.entry.detail(text);
}
pub fn set_unit(&self, unit: Unit) {
self.entry.unit.store(unit.to_u8(), Ordering::Relaxed);
}
/// A step that must not be cut short (the Lightroom import's commit) turns cancelling off for its duration.
pub fn set_cancellable(&self, yes: bool) {
self.entry.cancellable.store(yes, Ordering::Relaxed);
}
pub fn cancel_flag(&self) -> Option<Arc<AtomicBool>> {
self.entry.cancel.clone()
}
pub fn is_cancelled(&self) -> bool {
self.entry.is_cancelled()
}
/// For progress from another thread; it does not keep the row.
pub fn handle(&self) -> TaskHandle {
TaskHandle { entry: self.entry.clone() }
}
}
impl Drop for TaskGuard {
fn drop(&mut self) {
let id = self.entry.id;
lock(&self.activity.inner.tasks).retain(|e| e.id != id);
}
}
impl std::fmt::Debug for TaskGuard {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("TaskGuard").field("id", &self.entry.id).field("kind", &self.entry.kind).finish()
}
}
/// Progress for a task from another thread (see [`TaskGuard::handle`]).
#[derive(Clone)]
pub struct TaskHandle {
entry: Arc<Entry>,
}
impl TaskHandle {
pub fn progress(&self, done: u64, total: u64) {
self.entry.progress(done, total);
}
pub fn detail(&self, text: &str) {
self.entry.detail(text);
}
pub fn is_cancelled(&self) -> bool {
self.entry.is_cancelled()
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
#[test]
fn guard_drop_removes_the_row() {
let a = Activity::default();
let g = a.start("export", "Exporting", Cancel::Yes);
assert_eq!(a.list().len(), 1);
drop(g);
assert!(a.list().is_empty());
}
#[test]
fn guard_drop_on_panic_unwind_removes_the_row() {
let a = Activity::default();
let a2 = a.clone();
let r = std::panic::catch_unwind(std::panic::AssertUnwindSafe(move || {
let _g = a2.start("import", "Importing", Cancel::Yes);
panic!("synthetic worker panic");
}));
assert!(r.is_err());
assert!(a.list().is_empty());
}
#[test]
fn cancel_sets_the_adopted_flag() {
let a = Activity::default();
let flag = Arc::new(AtomicBool::new(false));
let g = a.start("export", "Exporting", Cancel::Flag(flag.clone()));
a.cancel(g.id()).unwrap();
assert!(flag.load(Ordering::Relaxed) && g.is_cancelled());
assert!(a.list()[0].cancelling);
}
#[test]
fn cancel_errors_instead_of_panicking() {
let a = Activity::default();
assert!(a.cancel(42).is_err());
let g = a.start("faces", "Finding faces", Cancel::No);
assert!(a.cancel(g.id()).is_err());
let h = a.start("lightroom", "Importing Lightroom catalog", Cancel::Yes);
h.set_cancellable(false);
assert!(a.cancel(h.id()).is_err());
assert!(!h.is_cancelled());
}
#[test]
fn cancel_all_counts_only_cancellable() {
let a = Activity::default();
let (_x, _y, _z) = (a.start("export", "E", Cancel::Yes), a.start("import", "I", Cancel::Yes), a.start("faces", "F", Cancel::No));
assert_eq!(a.cancel_all(), 2);
assert!(a.running_cancellable().is_empty());
}
#[test]
fn progress_is_clamped_and_detail_capped() {
let a = Activity::default();
let g = a.start("export", "E", Cancel::Yes);
g.progress(12, 10);
g.detail(&"é".repeat(300));
let t = &a.list()[0];
assert_eq!((t.done, t.total), (10, 10));
assert_eq!(t.detail.chars().count(), 200);
g.progress(5, 0); // indeterminate keeps done as given
assert_eq!(a.list()[0].total, 0);
assert_eq!(a.list()[0].unit, Unit::Count);
g.set_unit(Unit::Bytes);
assert_eq!(a.list()[0].unit, Unit::Bytes);
}
#[test]
fn ids_are_unique_and_list_is_oldest_first() {
let a = Activity::default();
let (g1, g2) = (a.start("export", "A", Cancel::Yes), a.start("import", "B", Cancel::Yes));
assert!(g2.id() > g1.id());
assert_eq!(a.list().iter().map(|t| t.id).collect::<Vec<_>>(), vec![g1.id(), g2.id()]);
}
#[test]
fn poisoned_lock_still_works() {
let a = Activity::default();
let a2 = a.clone();
let _ = std::thread::spawn(move || {
let _held = a2.inner.tasks.lock().unwrap();
panic!("synthetic panic while holding the task list");
})
.join();
assert!(a.inner.tasks.is_poisoned());
let _g = a.start("export", "E", Cancel::Yes);
assert_eq!(a.list().len(), 1);
}
#[test]
fn handle_updates_from_another_thread() {
let a = Activity::default();
let g = a.start("previews", "Building previews", Cancel::Yes);
let h = g.handle();
std::thread::spawn(move || h.progress(3, 7)).join().unwrap();
assert_eq!((a.list()[0].done, a.list()[0].total), (3, 7));
}
}

176
crates/engine/src/camera_preview.rs
View File

@ -4,7 +4,7 @@
//! A global matrix can't follow the camera's hue-dependent rendering (the best matrix rendered a
//! lime shirt olive that the camera kept lime): a hue/saturation table fitted to the residuals
//! (applied like a DNG `ProfileHueSatMap`) corrects that when it also improves the held-out pixels.
use lightcraft_color::{D50, D65, Mat3, PROPHOTO, REC2020, bradford, luminance_2020};
use lightcraft_color::{D50, D65, Mat3, PROPHOTO, REC2020, Xy, bradford, luminance_2020};
use lightcraft_pipeline::tone::{CameraTone, ToneMap};
use lightcraft_raster::{
Rgb32f,
@ -12,6 +12,19 @@ use lightcraft_raster::{
};
use lightcraft_raw::{RawFormat, RawImage, color::CameraTransform, profile::HsvTable};
/// The version of the camera-look fit: how a raw's colour matrix, hue/saturation table and tone
/// and chroma curves are chosen. Smart previews record it in their header (`look_version`), because
/// they keep the curve and the colour the fit produced; one written by an older version is rebuilt
/// from its original when that is online, and kept as it is while it is offline.
///
/// **Bump this whenever a change makes the fit give a different result for the same file** (the
/// fit, its gates, the fallbacks, the camera profiles' effect on it). Do not bump it for changes
/// that leave the fitted look alone. `RENDER_CACHE_VERSION` covers renders; this covers proxies.
///
/// 1: the fit as of #499's follow-up; 2: Sony DRO (tone curve lowered to Sony's curve without DRO)
/// and the ILCE-7CR profile (#528, #583, #568, #616).
pub const LOOK_VERSION: u32 = 2;
#[derive(Clone, Debug)]
pub(crate) struct CameraLook {
pub matrix: Mat3,
@ -46,6 +59,43 @@ pub(crate) fn file_local_look(format: RawFormat) -> bool {
)
}
/// The colour a raw starts from, first available first: the file's own colour matrices (DNG); a look fitted to the
/// file's JPEG ([`fit_preview`]: with the camera profile for its model when there is one, else with colour fitted to
/// this photo alone); the model's colour matrices from measured spectral sensitivities (`lightcraft_raw::spectral`);
/// the neutral fallback. The spectral matrices only stand in for the neutral fallback, so a photo whose JPEG fit is
/// accepted renders as without them; when used they are written into `raw.color`. Returns the as-shot white, the
/// camera transform at it and the fitted look.
pub(crate) fn starting_colour(raw: &mut RawImage, bytes: &[u8]) -> (Xy, CameraTransform, Option<CameraLook>) {
starting_colour_with(raw, |raw, t| fit_preview(raw, bytes, t))
}
/// [`starting_colour`] with the JPEG fit passed in (tests pin the order with it).
fn starting_colour_with(
raw: &mut RawImage,
fit: impl FnOnce(&RawImage, &CameraTransform) -> Option<CameraLook>,
) -> (Xy, CameraTransform, Option<CameraLook>) {
let start = |raw: &RawImage| {
let xy = lightcraft_raw::color::as_shot_white_xy(raw);
(xy, lightcraft_raw::color::camera_transform(raw, xy))
};
let (xy, t) = start(raw);
if !t.matrix_is_fallback {
return (xy, t, None);
}
if let Some(look) = fit(raw, &t) {
return (xy, t, Some(look));
}
let Some(camera) = raw.metadata.model.as_deref().and_then(|model| lightcraft_raw::spectral::find(raw.metadata.make.as_deref(), model)) else {
return (xy, t, None);
};
camera.fill(&mut raw.color);
if lightcraft_pipeline::profiling() {
eprintln!("[profile] {:?} camera colour from spectral sensitivities: {} {}", raw.format, camera.make, camera.model);
}
let (xy, t) = start(raw);
(xy, t, None)
}
pub(crate) fn fit_preview(raw: &RawImage, bytes: &[u8], transform: &CameraTransform) -> Option<CameraLook> {
if !transform.matrix_is_fallback || !file_local_look(raw.format) {
return None;
@ -1389,6 +1439,130 @@ mod tests {
assert!(fit_hue_sat(&[], &Mat3::IDENTITY).is_none(), "too few samples");
}
fn raw_of(format: RawFormat, make: &str, model: &str, color: lightcraft_raw::ColorData) -> RawImage {
let area = lightcraft_raw::Rect::new(0, 0, 2, 2);
RawImage {
format,
width: 2,
height: 2,
cpp: 1,
data: lightcraft_raw::RawData::U16(vec![0; 4]),
cfa: None,
bits: 14,
black: Default::default(),
white: vec![16383.0],
active_area: area,
crop: area,
orientation: Default::default(),
color,
wb_multipliers: Some([2.0, 1.0, 1.5]),
linearized: false,
opcodes: Default::default(),
metadata: lightcraft_raw::Metadata { make: Some(make.to_owned()), model: Some(model.to_owned()), ..Default::default() },
}
}
/// Step 1: a file's own colour matrices win over everything, including a spectral row for its model; no JPEG fit
/// is tried.
#[test]
fn a_files_own_matrices_come_first() {
let own = Mat3([[0.6, -0.1, -0.05], [-0.4, 1.3, 0.1], [-0.1, 0.2, 0.7]]);
let color = lightcraft_raw::ColorData { illuminant: [21, 0], color_matrix: [Some(own), None], ..Default::default() };
let mut raw = raw_of(RawFormat::Cr2, "Canon", "Canon EOS 600D", color.clone());
let (_, t, look) = starting_colour_with(&mut raw, |_, _| panic!("no JPEG fit for a raw with its own matrices"));
assert_eq!(raw.color, color);
assert!(!t.matrix_is_fallback && look.is_none());
}
/// Steps 2 and 3 before step 4: an accepted JPEG fit (with or without a camera profile) is kept for a model that
/// has spectral matrices, and the spectral matrices are not written. Photos that had a fitted look before the
/// spectral table render exactly as they did.
#[test]
fn an_accepted_jpeg_fit_comes_before_the_spectral_matrices() {
for (make, model) in [("SONY", "ILCE-7M4"), ("Canon", "Canon EOS 600D")] {
assert!(lightcraft_raw::spectral::find(Some(make), model).is_some(), "{model}");
let mut raw = raw_of(RawFormat::Cr2, make, model, Default::default());
let fitted = dro_test_look(&std::array::from_fn(|i| 0.006 * 1.13f32.powi(i as i32)), &|x| x.sqrt());
let (xy, t, look) = starting_colour_with(&mut raw, |raw, t| {
assert!(t.matrix_is_fallback && !lightcraft_raw::color::has_matrix(&raw.color), "the fit sees the raw without spectral matrices");
Some(fitted.clone())
});
assert_eq!(look.map(|l| l.matrix), Some(fitted.matrix), "{model}");
assert_eq!(raw.color, lightcraft_raw::ColorData::default(), "{model}: spectral matrices left out");
assert!(t.matrix_is_fallback, "{model}");
assert_eq!(xy, lightcraft_raw::color::as_shot_white_xy(&raw));
}
}
/// Step 4: when the JPEG fit fails (here: no camera JPEG at all), a model with a spectral row takes its matrices,
/// whether or not it has a camera profile (the ILCE-7M4 has a bundled one).
#[test]
fn a_failed_jpeg_fit_falls_back_to_the_spectral_matrices() {
assert!(crate::camera_profiles::get("ILCE-7M4").is_some());
let camera = lightcraft_raw::spectral::find(Some("SONY"), "ILCE-7M4").unwrap();
let mut raw = raw_of(RawFormat::Arw, "SONY", "ILCE-7M4", Default::default());
let (_, t, look) = starting_colour(&mut raw, &[]);
assert!(look.is_none());
assert_eq!(raw.color.color_matrix, camera.color_matrix.map(Some));
assert!(!t.matrix_is_fallback);
let camera = lightcraft_raw::spectral::find(Some("Canon"), "Canon EOS 600D").unwrap();
let mut raw = raw_of(RawFormat::Cr2, "Canon", "Canon EOS 600D", Default::default());
let (xy, t, look) = starting_colour_with(&mut raw, |_, _| None);
assert!(look.is_none());
assert_eq!(raw.color.illuminant, [17, 21]);
assert_eq!(raw.color.forward_matrix, camera.forward_matrix.map(Some));
assert!(!t.matrix_is_fallback);
assert_eq!(xy, lightcraft_raw::color::as_shot_white_xy(&raw));
// white-balanced white stays white
let w = t.matrix.apply([1.0; 3]);
assert!(w.iter().all(|v| (v - 1.0).abs() < 1e-4), "{w:?}");
}
/// Step 5: a camera outside the table whose fit fails keeps the neutral fallback.
#[test]
fn a_camera_outside_the_table_keeps_the_fallback() {
let mut raw = raw_of(RawFormat::Cr2, "Canon", "Canon EOS 7D", Default::default());
let (_, t, look) = starting_colour(&mut raw, &[]);
assert!(look.is_none() && t.matrix_is_fallback);
assert_eq!(raw.color, lightcraft_raw::ColorData::default());
// without a model there is no table row
let mut raw = raw_of(RawFormat::Cr2, "Canon", "", Default::default());
raw.metadata.model = None;
assert!(starting_colour(&mut raw, &[]).1.matrix_is_fallback);
}
/// The colour precedence on public raws (each skipped without its corpus file): a photo whose JPEG fit is
/// accepted keeps it and its colour data; one whose fit fails takes its spectral matrices when its model has
/// them; a DNG keeps its own matrices. White balance stays relative to the as-shot look for every raw without
/// matrices of its own.
#[test]
fn corpus_colour_precedence() {
let dir = std::env::var_os("LIGHTCRAFT_CORPUS")
.map(std::path::PathBuf::from)
.unwrap_or_else(|| std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("../../corpus"))
.join("raw");
for name in ["arw-sony-a7m4-14bit.arw", "cr2-canon-5d3.cr2", "nef-nikon-d5100-lossless.nef", "cr2-canon-7d.cr2", "dng-canon-5d3-16bit.dng"] {
let Ok(bytes) = std::fs::read(dir.join(name)) else {
eprintln!("skip: {name} absent");
continue;
};
let mut raw = lightcraft_raw::decode(&bytes).unwrap();
raw.opcodes.list3.retain(|op| !op.is_lens_correction());
let own = lightcraft_raw::color::has_matrix(&raw.color);
let covered = raw.metadata.model.as_deref().and_then(|m| lightcraft_raw::spectral::find(raw.metadata.make.as_deref(), m)).is_some();
let before = raw.color.clone();
let (_, t, fitted) = starting_colour(&mut raw, &bytes);
let spectral = !own && fitted.is_none() && covered;
eprintln!("{name}: own matrices {own}, JPEG fit {}, spectral {spectral}", fitted.is_some());
assert_eq!(raw.color != before, spectral, "{name}");
assert_eq!(t.matrix_is_fallback, !own && !spectral, "{name}");
let (_, info) = crate::files::load_bytes(&bytes, 400).unwrap();
assert_eq!(info.camera_tone.is_some(), fitted.is_some(), "{name}");
assert_eq!(info.relative_wb, !own, "{name}");
}
}
#[test]
fn supported_raws_get_a_file_local_look() {
// every decoded raw without a colour matrix of its own (issue #310: CR2 rendered flat with the fallback)

65
crates/engine/src/cmd/activity.rs Normal file
View File

@ -0,0 +1,65 @@
//! `activity.list` / `activity.cancel`: the background tasks in flight (see [`crate::activity`]).
use serde_json::{Value, json};
use super::{CommandSpec, always, bad, cmd};
pub fn specs() -> Vec<CommandSpec> {
vec![
cmd!(
query "activity.list",
"Background Tasks",
[],
None,
"{} → {tasks: [{id, kind, label, done, total, unit: count|bytes|percent, detail, cancellable, cancelling, ageMs}]} — the long-running tasks in flight (imports, exports, preview builds, downloads, the face scan…), oldest first; total 0 = not known yet",
always,
|s, _| Ok(json!({"tasks": s.activity.list()}))
),
cmd!(
"activity.cancel",
"Cancel Background Task",
[],
None,
"{id} | {all: true} → {cancelled: n} — ask a task (or every cancellable task) to stop; it stops at its next file or photo. An unknown id (it may have just finished) or a task that can't be cancelled is an error",
always,
|s, p| {
const C: &str = "activity.cancel";
if p.get("all").and_then(Value::as_bool) == Some(true) {
return Ok(json!({"cancelled": s.activity.cancel_all()}));
}
let id = p.get("id").and_then(Value::as_u64).ok_or_else(|| bad(C, "id or all"))?;
s.activity.cancel(id).map_err(|e| bad(C, e))?;
Ok(json!({"cancelled": 1}))
}
),
]
}
#[cfg(test)]
mod tests {
use serde_json::json;
use crate::Session;
use crate::activity::Cancel;
#[test]
fn list_and_cancel_through_execute() {
let mut s = Session::new();
let g = s.activity.start("export", "Exporting", Cancel::Yes);
let r = s.execute("activity.list", &json!({})).unwrap();
assert_eq!(r["tasks"][0]["kind"], "export");
assert_eq!(r["tasks"][0]["cancellable"], true);
let r = s.execute("activity.cancel", &json!({"id": g.id()})).unwrap();
assert_eq!(r["cancelled"], 1);
assert!(g.is_cancelled());
}
#[test]
fn cancel_all_and_errors() {
let mut s = Session::new();
let (_a, _b) = (s.activity.start("export", "E", Cancel::Yes), s.activity.start("faces", "F", Cancel::No));
assert_eq!(s.execute("activity.cancel", &json!({"all": true})).unwrap()["cancelled"], 1);
assert!(s.execute("activity.cancel", &json!({"id": 999})).is_err());
assert!(s.execute("activity.cancel", &json!({})).is_err());
}
}

3
crates/engine/src/cmd/denoise.rs
View File

@ -368,7 +368,7 @@ fn download(s: &mut Session, p: &Value) -> Result<Value> {
return Err(bad(C, "that model is already installed"));
}
let host = lightcraft_denoise::known::host(&spec.url).to_string();
s.denoise.downloads.start(spec, &dir).map_err(|e| bad(C, e))?;
s.denoise.downloads.start(spec, &dir, &s.activity, &k.manifest.name).map_err(|e| bad(C, e))?;
Ok(json!({"started": id, "from": host}))
}
@ -614,6 +614,7 @@ fn status(s: &mut Session, p: &Value) -> Result<Value> {
/// `denoise.pump {pace?}`: cheap to call every frame. Takes in what the background job has finished and keeps one going.
fn pump(s: &mut Session, p: &Value) -> Result<Value> {
s.denoise_pump(Pace::parse(p.get("pace").and_then(Value::as_str)));
s.denoise_track();
Ok(json!({
"active": s.denoise.active.is_some(),
"running": s.denoise_running_json(),

8
crates/engine/src/cmd/face_models.rs
View File

@ -356,7 +356,13 @@ fn download(s: &mut Session, p: &Value) -> Result<Value> {
return Err(bad(C, "that model is already installed"));
}
let host = spec.host().to_string();
s.face_downloads.start(spec, &dir).map_err(|e| bad(C, e))?;
let name = known::all()
.into_iter()
.find(|m| m.id == spec.id)
.map(|m| m.name)
.or_else(|| s.face_catalog.entries.iter().find(|e| e.manifest.id == spec.id).map(|e| e.manifest.name.clone()))
.unwrap_or_else(|| spec.id.clone());
s.face_downloads.start(spec, &dir, &s.activity, &name).map_err(|e| bad(C, e))?;
Ok(json!({"started": id, "from": host}))
}

23
crates/engine/src/cmd/face_recognize.rs
View File

@ -515,6 +515,21 @@ mod imp {
}))
}
/// The scan's row in the activity stack: there while `pending` photos are left, its whole the most that were left at
/// once since the scan last finished (which it returns; 0 when none are left). It can't be stopped from there: the
/// scan is switched off in Settings.
fn track(s: &mut Session, pending: u64) -> u64 {
if pending == 0 {
s.faces.task = None;
s.faces.peak = 0;
return 0;
}
s.faces.peak = s.faces.peak.max(pending);
let task = s.faces.task.get_or_insert_with(|| s.activity.start("faces", "Finding faces", crate::activity::Cancel::No));
task.progress(s.faces.peak.saturating_sub(pending), s.faces.peak);
s.faces.peak
}
/// `faces.pump`: cheap to call every frame. Takes in what the background worker has finished and, when it is idle,
/// hands it the next photo with faces to embed. Does nothing (and says so) until recognition is switched on in
/// Settings with a model chosen.
@ -528,14 +543,17 @@ mod imp {
finish_downloads(s);
let enabled = s.face_models_dir.clone().is_some_and(|d| enabled_cached(s, &d));
if !enabled {
track(s, 0);
return Ok(json!({"active": false}));
}
if s.faces.retry_at.is_some_and(|t| Instant::now() < t) {
track(s, 0);
return Ok(json!({"active": false}));
}
let Ok(embedder) = current_with(s, C, false) else {
// the chosen model is gone or broken: look again in a few seconds, not every frame
s.faces.retry_at = Some(Instant::now() + Duration::from_secs(3));
track(s, 0);
return Ok(json!({"active": false}));
};
s.faces.retry_at = None;
@ -573,10 +591,13 @@ mod imp {
break;
}
}
let pending = s.faces.queue.len() + s.faces.in_flight.len();
let peak = track(s, pending as u64);
Ok(json!({
"active": true,
"inFlight": s.faces.in_flight.len(),
"pendingPhotos": s.faces.queue.len() + s.faces.in_flight.len(),
"pendingPhotos": pending,
"peak": peak,
"indexedFaces": s.faces.index.len(),
"searchedPhotos": s.faces.scanned.len(),
"workers": target,

2
crates/engine/src/cmd/masks.rs
View File

@ -464,7 +464,7 @@ pub fn specs() -> Vec<CommandSpec> {
),
));
}
let started = s.segmenter.start_download().map_err(ai_err)?;
let started = s.segmenter.start_download(&s.activity).map_err(ai_err)?;
Ok(json!({"started": started, "installed": s.segmenter.installed(), "downloading": s.segmenter.download_status().running}))
}
),

2
crates/engine/src/cmd/mod.rs
View File

@ -4,6 +4,7 @@
//! delete), `album.*`, `develop.*` (settings), `crop.*`, `mask.*`, `preset.*`, `version.*`,
//! `edit.*` (undo/redo), and queries (`catalog.query`, `photo.inspect`, `develop.get`…).
mod activity;
mod before;
pub(crate) mod browse;
mod color;
@ -154,6 +155,7 @@ pub fn command_specs() -> &'static [CommandSpec] {
v.extend(face_models::specs());
v.extend(face_detect::specs());
v.extend(face_recognize::specs());
v.extend(activity::specs());
v
})
}

151
crates/engine/src/cmd/previews.rs
View File

@ -1,7 +1,7 @@
//! Build previews ahead of time: each photo's grid thumbnail and its loupe view (standard size,
//! or 1:1), rendered into the memory + disk preview cache so browsing and the loupe are instant.
//! Runs on a background thread (the app keeps working; progress / cancel by command), or inline
//! with `wait` (CLI, MCP, tests).
//! Runs on a background thread (the app keeps working; progress / cancel by command, and a row in
//! the activity stack, issue #345), or inline with `wait` (CLI, MCP, tests).
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
@ -9,6 +9,7 @@ use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
use serde_json::{Value, json};
use super::{CommandSpec, always, bad, cmd, str_param};
use crate::activity::{Cancel, TaskGuard};
use crate::media::{RenderJob, THUMB_SIZES};
use crate::{Result, Session};
@ -21,7 +22,8 @@ pub struct PreviewBuild {
pub total: usize,
pub done: AtomicUsize,
pub failed: AtomicUsize,
pub cancel: AtomicBool,
/// Shared with the build's row in the activity stack (its ✕ sets it).
pub cancel: Arc<AtomicBool>,
pub finished: AtomicBool,
/// Damaged smart previews rebuilt (Build Smart Previews only).
pub repaired: AtomicUsize,
@ -62,7 +64,19 @@ fn jobs_for(s: &mut Session, id: lightcraft_catalog::PhotoId, edge: Option<usize
v
}
fn run_all(jobs: Vec<Vec<RenderJob>>, state: &PreviewBuild) {
/// The build's row in the activity stack (`what` as in [`PreviewBuild::what`]).
fn task_for(s: &Session, state: &PreviewBuild) -> TaskGuard {
let (kind, label) = match state.what {
"" => ("previews", "Building previews"),
"smart previews" => ("smartPreviews", "Building smart previews"),
_ => ("smartPreviews", "Discarding smart previews"),
};
let task = s.activity.start(kind, label, Cancel::Flag(state.cancel.clone()));
task.progress(0, state.total as u64);
task
}
fn run_all(jobs: Vec<Vec<RenderJob>>, state: &PreviewBuild, task: &TaskGuard) {
for photo_jobs in jobs {
if state.cancel.load(Ordering::Relaxed) {
break;
@ -71,7 +85,8 @@ fn run_all(jobs: Vec<Vec<RenderJob>>, state: &PreviewBuild) {
if !ok {
state.failed.fetch_add(1, Ordering::Relaxed);
}
state.done.fetch_add(1, Ordering::Relaxed);
let done = state.done.fetch_add(1, Ordering::Relaxed).saturating_add(1);
task.progress(done as u64, state.total as u64);
}
state.finished.store(true, Ordering::Relaxed);
}
@ -91,26 +106,35 @@ fn build(s: &mut Session, p: &Value) -> Result<Value> {
let jobs: Vec<Vec<RenderJob>> = ids.iter().map(|id| jobs_for(s, *id, edge)).filter(|j| !j.is_empty()).collect();
let state = Arc::new(PreviewBuild { total: jobs.len(), ..Default::default() });
s.preview_build = Some(state.clone());
let task = task_for(s, &state);
let wait = p.get("wait").and_then(Value::as_bool).unwrap_or(false) || cfg!(target_arch = "wasm32");
if wait {
run_all(jobs, &state);
run_all(jobs, &state, &task);
return Ok(state.json());
}
#[cfg(not(target_arch = "wasm32"))]
{
// the row stays while the worker runs (it is dropped with the closure, however that ends)
let st = state.clone();
std::thread::Builder::new()
.name("lc-build-previews".into())
.spawn(move || run_all(jobs, &st))
.spawn(move || run_all(jobs, &st, &task))
.map_err(|e| bad(C, format!("could not start: {e}")))?;
}
Ok(state.json())
}
/// One photo's smart preview to build or discard: (photo, its file in the smart previews folder,
/// how to load the original at preview size).
/// how to load the original at preview size, whether the photo takes the per-file camera look —
/// `camera_preview::file_local_look`, [`lightcraft_catalog::Photo::relative_wb`]: only those
/// proxies go stale when the fit changes).
#[cfg(not(target_arch = "wasm32"))]
type SmartJob = (lightcraft_catalog::PhotoId, std::path::PathBuf, Option<crate::media::SourceRef>);
type SmartJob = (lightcraft_catalog::PhotoId, std::path::PathBuf, Option<crate::media::SourceRef>, bool);
/// Held while one proxy is checked and (re)written, so Build Smart Previews and the opening look
/// refresh ([`refresh_stale_smart_previews`]), on their own threads, never write the same file at once.
#[cfg(not(target_arch = "wasm32"))]
static SMART_WRITE: std::sync::Mutex<()> = std::sync::Mutex::new(());
/// What Build / Discard Smart Previews did.
#[cfg(not(target_arch = "wasm32"))]
@ -120,11 +144,29 @@ struct SmartCounts {
built: usize,
/// Of those, damaged ones (cut short by a crash or a full drive) that were rebuilt.
repaired: usize,
/// Of those, proxies written by an older camera-look fit that were rebuilt from their online originals.
refreshed: usize,
/// Of those kept as they were: out of date, but the original can't be read now (offline), so
/// the stored look stays and the next run with the original online rebuilds them.
stale_kept: usize,
removed: usize,
/// `[[id, why]]`
failed: Vec<Value>,
}
/// `source` if it reads the photo's original, else `None`: a cached source decoded from a smart
/// preview (it has no decoder facts), a proxy standing in for an offline original, and the proxy
/// fallback of a file source are not the original.
#[cfg(not(target_arch = "wasm32"))]
fn original_only(source: crate::media::SourceRef) -> Option<crate::media::SourceRef> {
use crate::media::SourceRef;
match source {
SourceRef::File { path, max_edge, loader, denoise, .. } => Some(SourceRef::File { path, max_edge, loader, fallback: None, denoise }),
SourceRef::Loaded(d) if d.info.is_some() => Some(SourceRef::Loaded(d)),
_ => None,
}
}
/// The file-system half of Build / Discard Smart Previews (runs on a worker thread in the app).
/// `custom`: the folder was chosen (on another drive, perhaps): it must be there, it is never
/// recreated. With `state`, progress and cancel.
@ -134,7 +176,7 @@ fn smart_run(
custom: bool,
discard: bool,
jobs: Vec<SmartJob>,
state: Option<&PreviewBuild>,
state: Option<(&PreviewBuild, &TaskGuard)>,
) -> std::result::Result<SmartCounts, String> {
let mut n = SmartCounts::default();
if !discard {
@ -145,21 +187,27 @@ fn smart_run(
std::fs::create_dir_all(dir).map_err(|e| format!("{}: {e}", dir.display()))?;
}
}
for (id, path, source) in jobs {
if state.is_some_and(|st| st.cancel.load(Ordering::Relaxed)) {
for (id, path, source, look) in jobs {
if state.is_some_and(|(st, _)| st.cancel.load(Ordering::Relaxed)) {
break;
}
let _one = SMART_WRITE.lock().unwrap_or_else(std::sync::PoisonError::into_inner);
if discard {
if std::fs::remove_file(&path).is_ok() {
n.removed += 1;
}
} else {
// a complete proxy is kept; a missing or damaged one (cut short by a crash or a full
// drive) is (re)built
// a complete, current proxy is kept; a missing or damaged one (cut short by a crash or a
// full drive) is (re)built, and so is one written by an older camera-look fit (see
// `camera_preview::LOOK_VERSION`) when its original can be read
let damaged = path.exists();
if damaged && crate::smart::is_valid(&path) {
let stale = look && damaged && crate::smart::is_stale(&path);
if damaged && !stale && crate::smart::is_valid(&path) {
n.built += 1;
} else {
// a stale proxy is rebuilt from the original only: never from itself (the file
// fallback or a source decoded from a proxy would stamp the old look as current)
let source = if stale { source.and_then(original_only) } else { source };
// atomic: a failed write leaves no partial proxy that would pass for a built one;
// and synced (issue #134): built so the photo can be edited while its original
// is offline, when it is the only copy — the sync is small next to the decode
@ -171,24 +219,33 @@ fn smart_run(
match r {
Ok(()) => {
n.built += 1;
if damaged {
if stale {
n.refreshed += 1;
} else if damaged {
n.repaired += 1;
if let Some(st) = state {
if let Some((st, _)) = state {
st.repaired.fetch_add(1, Ordering::Relaxed);
}
}
}
Err(e) if stale => {
// offline (or unreadable) original: the proxy is all there is, keep its look
log::info!("smart preview {} keeps its older look: {e}", path.display());
n.built += 1;
n.stale_kept += 1;
}
Err(e) => {
n.failed.push(json!([id.0, e]));
if let Some(st) = state {
if let Some((st, _)) = state {
st.failed.fetch_add(1, Ordering::Relaxed);
}
}
}
}
}
if let Some(st) = state {
st.done.fetch_add(1, Ordering::Relaxed);
if let Some((st, task)) = state {
let done = st.done.fetch_add(1, Ordering::Relaxed).saturating_add(1);
task.progress(done as u64, st.total as u64);
}
}
Ok(n)
@ -215,18 +272,19 @@ fn smart(s: &mut Session, p: &Value) -> Result<Value> {
}
let path = dir.join(crate::smart::file_name(&ph));
let source = (!discard).then(|| s.media.source_ref(&ph, crate::media::SourceLevel::Preview));
jobs.push((id, path, source));
jobs.push((id, path, source, ph.relative_wb()));
}
let custom = s.smart_previews_dir.is_some();
if background {
let what = if discard { "discarding smart previews" } else { "smart previews" };
let state = Arc::new(PreviewBuild { total: jobs.len(), what, ..Default::default() });
s.preview_build = Some(state.clone());
let task = task_for(s, &state);
let st = state.clone();
std::thread::Builder::new()
.name("lc-smart-previews".into())
.spawn(move || {
if let Err(e) = smart_run(&dir, custom, discard, jobs, Some(&st)) {
if let Err(e) = smart_run(&dir, custom, discard, jobs, Some((&st, &task))) {
log::warn!("{C}: {e}");
*st.error.lock().unwrap_or_else(std::sync::PoisonError::into_inner) = Some(e);
}
@ -236,7 +294,54 @@ fn smart(s: &mut Session, p: &Value) -> Result<Value> {
return Ok(state.json());
}
let r = smart_run(&dir, custom, discard, jobs, None).map_err(|e| bad(C, e))?;
Ok(json!({"built": r.built, "repaired": r.repaired, "removed": r.removed, "failed": r.failed}))
Ok(
json!({"built": r.built, "repaired": r.repaired, "refreshed": r.refreshed, "staleKept": r.stale_kept, "removed": r.removed, "failed": r.failed}),
)
}
/// Bring smart previews written by an older camera-look fit up to date, once per fit version: when
/// the smart previews folder has not been checked at this `LOOK_VERSION` (its marker,
/// [`crate::smart::look_checked`]), a worker thread (the UI never waits on the drives) looks at the
/// proxies of the raws that take the per-file camera look, the only ones the fit reaches (JPEG,
/// PNG… proxies are never rebuilt), and rebuilds the stale ones from their originals. One whose
/// original can't be read (offline) is kept as it is, still marked stale in its header: Build
/// Smart Previews rebuilds it once the original is back. Only existing proxies are touched: none
/// is built, and a missing folder is never created. Returns the thread, if started.
#[cfg(not(target_arch = "wasm32"))]
pub(crate) fn refresh_stale_smart_previews(s: &mut Session) -> Option<std::thread::JoinHandle<()>> {
let dir = s.media.smart_dir.clone()?;
if !dir.is_dir() || crate::smart::look_checked(&dir) {
return None;
}
let max_edge = crate::media::SourceLevel::Preview.max_edge();
let photos: Vec<_> = s.catalog.photos().filter(|p| matches!(p.source, lightcraft_catalog::Source::File { .. }) && p.relative_wb()).collect();
let jobs: Vec<SmartJob> =
photos.iter().map(|p| (p.id, dir.join(crate::smart::file_name(p)), Some(s.media.origin_ref(&p.source, max_edge)), true)).collect();
std::thread::Builder::new()
.name("lc-smart-look".into())
.spawn(move || refresh_stale(&dir, jobs))
.map_err(|e| log::warn!("smart previews: could not start the look refresh: {e}"))
.ok()
}
/// The body of [`refresh_stale_smart_previews`]: rebuild the stale proxies among `jobs`.
#[cfg(not(target_arch = "wasm32"))]
fn refresh_stale(dir: &std::path::Path, jobs: Vec<SmartJob>) {
let stale: Vec<SmartJob> = jobs.into_iter().filter(|(_, path, _, _)| crate::smart::is_stale(path)).collect();
if !stale.is_empty() {
// `custom`: the folder is there already, and must not be recreated if it went away since
match smart_run(dir, true, false, stale, None) {
Ok(n) => log::info!("smart previews: {} refreshed to the current camera look, {} kept (original offline)", n.refreshed, n.stale_kept),
Err(e) => {
// not marked: the next opening tries again
log::warn!("smart previews: look refresh: {e}");
return;
}
}
}
if let Err(e) = crate::smart::mark_look_checked(dir) {
log::warn!("smart previews: {e}");
}
}
/// The smart previews folder: where it is, whether it is custom, what it holds, whether it can be

28
crates/engine/src/denoise.rs
View File

@ -890,6 +890,8 @@ pub(crate) struct State {
pub generation: u64,
/// Photos made since the session started.
made: u64,
/// AI Denoise's row in the activity stack, while a photo is being made or waits its turn.
task: Option<crate::activity::TaskGuard>,
}
impl Default for State {
@ -909,6 +911,7 @@ impl Default for State {
failed: HashMap::new(),
generation: 0,
made: 0,
task: None,
}
}
}
@ -1354,6 +1357,31 @@ impl Session {
self.denoise.running.is_some() || (!self.denoise.queue.is_empty() && self.denoise.active.is_some())
}
/// Keep AI Denoise's row in the activity stack up to date (each `denoise.pump`): there while a photo is being made or
/// waits its turn, with the running photo's tiles and its file name; gone when the queue is idle. It has no ✕: what is
/// made follows the photos' Denoise amounts and Settings ▸ AI Denoise.
pub(crate) fn denoise_track(&mut self) {
if !self.denoise_busy() {
self.denoise.task = None;
return;
}
let task = self.denoise.task.get_or_insert_with(|| {
let t = self.activity.start("denoise", "AI Denoise", crate::activity::Cancel::No);
t.set_unit(crate::activity::Unit::Percent);
t
});
match &self.denoise.running {
Some(r) => {
task.progress(r.progress.done.load(Ordering::Relaxed) as u64, r.progress.total.load(Ordering::Relaxed) as u64);
task.detail(self.catalog.photo(r.photo).map_or("", |p| p.file_name.as_str()));
}
None => {
task.progress(0, 0);
task.detail("");
}
}
}
/// Seconds the running job has been going, and its progress.
pub(crate) fn denoise_running_json(&self) -> Value {
match &self.denoise.running {

80
crates/engine/src/face_download.rs
View File

@ -15,6 +15,8 @@ use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{Arc, Mutex, PoisonError};
use std::time::Duration;
use crate::activity::{Activity, Cancel, TaskHandle, Unit};
use lightcraft_faces::known::Download;
/// The folder inside the models folder where a downloaded file waits to be installed. Its name starts with a dot, so it
@ -42,15 +44,20 @@ pub enum State {
struct Shared {
state: Mutex<State>,
cancel: AtomicBool,
cancel: Arc<AtomicBool>,
/// The download's row in the activity stack.
task: Option<TaskHandle>,
}
impl Shared {
fn new(total: u64) -> Self {
Shared { state: Mutex::new(State::Running { bytes: 0, total }), cancel: AtomicBool::new(false) }
Shared { state: Mutex::new(State::Running { bytes: 0, total }), cancel: Arc::new(AtomicBool::new(false)), task: None }
}
fn set(&self, s: State) {
if let (State::Running { bytes, total }, Some(task)) = (&s, &self.task) {
task.progress(*bytes, *total);
}
*self.state.lock().unwrap_or_else(PoisonError::into_inner) = s;
}
@ -68,18 +75,27 @@ pub struct Downloads {
impl Downloads {
/// Start fetching `spec` into `<models dir>/.downloads/`. Different models can download side by side; the same
/// one is started once.
pub fn start(&mut self, spec: Download, models_dir: &Path) -> Result<(), String> {
/// The activity stack shows it (as `name`) until the thread ends; its ✕ stops it like [`Downloads::discard`].
pub fn start(&mut self, spec: Download, models_dir: &Path, activity: &Activity, name: &str) -> Result<(), String> {
if self.jobs.get(spec.id.as_str()).is_some_and(|j| matches!(j.get(), State::Running { .. })) {
return Err("that model is already downloading".into());
}
let staging = models_dir.join(STAGING);
std::fs::create_dir_all(&staging).map_err(|e| format!("could not create the download folder: {e}"))?;
let shared = Arc::new(Shared::new(spec.size_bytes));
let cancel = Arc::new(AtomicBool::new(false));
let task = activity.start("download", "Downloading model", Cancel::Flag(cancel.clone()));
task.set_unit(Unit::Bytes);
task.detail(name);
task.progress(0, spec.size_bytes);
let total = spec.size_bytes;
let shared = Arc::new(Shared { state: Mutex::new(State::Running { bytes: 0, total }), cancel, task: Some(task.handle()) });
let worker = shared.clone();
let id = spec.id.clone();
std::thread::Builder::new()
.name("face-model-download".into())
.spawn(move || {
// the row goes when the thread ends, however it ends
let _task = task;
let outcome = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| run(&spec, spec.base(), &staging, &worker)));
if outcome.is_err() {
worker.set(State::Failed("the download stopped unexpectedly".into()));
@ -267,6 +283,62 @@ mod tests {
base
}
/// A local server that sends `body` a few kilobytes at a time: a download that keeps running for a while.
fn serve_slowly(body: Vec<u8>) -> String {
let l = TcpListener::bind("127.0.0.1:0").unwrap();
let base = format!("http://{}/m", l.local_addr().unwrap());
std::thread::spawn(move || {
for mut s in l.incoming().flatten() {
let mut r = BufReader::new(s.try_clone().unwrap());
loop {
let mut line = String::new();
if r.read_line(&mut line).unwrap_or(0) == 0 || line.trim().is_empty() {
break;
}
}
let _ = s.write_all(format!("HTTP/1.1 200 OK\r\nContent-Length: {}\r\nConnection: close\r\n\r\n", body.len()).as_bytes());
for chunk in body.chunks(4096) {
if s.write_all(chunk).is_err() {
break;
}
std::thread::sleep(Duration::from_millis(20));
}
}
});
base
}
/// Poll until `done` holds (or fail after 10 s).
fn wait(what: &str, done: impl Fn() -> bool) {
let t = std::time::Instant::now();
while !done() {
assert!(t.elapsed() < Duration::from_secs(10), "{what}");
std::thread::sleep(Duration::from_millis(10));
}
}
#[test]
fn a_running_download_shows_in_activity_and_cancels() {
use crate::activity::{Activity, Unit};
let dir = scratch("activity");
let bytes: Vec<u8> = (0..400_000u32).map(|i| (i % 251) as u8).collect();
let mut spec = spec_for(&bytes);
spec.url = format!("{}/{}", serve_slowly(bytes), spec.file_name);
let activity = Activity::default();
let mut d = Downloads::default();
d.start(spec, &dir, &activity, "Test Model").unwrap();
let rows = activity.list();
assert_eq!(rows.len(), 1);
assert_eq!((rows[0].kind, rows[0].label.as_str(), rows[0].detail.as_str()), ("download", "Downloading model", "Test Model"));
assert_eq!((rows[0].unit, rows[0].total), (Unit::Bytes, 400_000));
assert!(rows[0].cancellable);
activity.cancel(rows[0].id).unwrap();
wait("the download never stopped", || !d.running());
assert_eq!(d.jobs["test-model"].get(), State::Cancelled);
wait("the row stayed", || activity.list().is_empty());
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
fn fetches_checks_and_stages_a_file() {
let dir = scratch("ok");

4
crates/engine/src/faces_index.rs
View File

@ -434,6 +434,10 @@ pub(crate) struct FacesState {
pub scanned: Scanned,
/// Photos that could not be searched this session (unreadable files): left alone until the next launch.
pub scan_failed: std::collections::HashSet<u64>,
/// The scan's row in the activity stack, while photos are left.
pub task: Option<crate::activity::TaskGuard>,
/// The most photos left at once since the scan last finished (the row's whole).
pub peak: u64,
}
impl FacesState {

93
crates/engine/src/files.rs
View File

@ -223,6 +223,12 @@ impl CameraColourModel<'_> {
lightcraft_raw::highlight::reconstruct_with(img, self.wb, HIGHLIGHT_CLIP, clipped_white(&self.matrix));
}
/// [`Self::rebuild_highlights`] for a binned image whose clipped channels `mask` tells
/// ([`lightcraft_raw::RawImage::develop_binned_masked`]).
pub fn rebuild_highlights_masked(&self, img: &mut Rgb32f, mask: &[u8]) {
lightcraft_raw::highlight::reconstruct_masked(img, self.wb, HIGHLIGHT_CLIP, clipped_white(&self.matrix), mask);
}
/// White balance, matrix, tables and baseline exposure, in place.
pub fn apply(&self, img: &mut Rgb32f) {
let (wb, m, gain) = (self.wb, self.matrix.to_f32(), self.gain);
@ -319,8 +325,9 @@ pub fn load_vec_pair(bytes: Vec<u8>, max_edge: usize, spec: &DenoiseSpec) -> Res
}
/// The denoised camera RGB of `raw` from its cached product, as the plain development would have it: the default
/// crop, and `factor` × `factor` blocks averaged into one pixel when the plain one was binned. `None` when the product
/// is missing, damaged, made from something else or not the size of this raw's active area.
/// crop, and `factor` × `factor` blocks averaged into one pixel when the plain one was binned (plain means: the
/// plain development's clip mask tells highlight reconstruction which of them held clipped samples). `None` when
/// the product is missing, damaged, made from something else or not the size of this raw's active area.
fn denoised_camera_rgb(raw: &lightcraft_raw::RawImage, spec: &DenoiseSpec, factor: usize) -> Option<Rgb32f> {
use lightcraft_denoise::product::{self, Window};
let a = raw.active_area;
@ -331,7 +338,7 @@ fn denoised_camera_rgb(raw: &lightcraft_raw::RawImage, spec: &DenoiseSpec, facto
if raw.opcodes.list3.is_empty() {
let c = raw.develop_crop(a.width, a.height);
let window = Window { x: c.x, y: c.y, width: c.width, height: c.height };
product::read_window(&spec.product, &spec.key, Some(window), factor, Some(HIGHLIGHT_CLIP)).ok()
product::read_window(&spec.product, &spec.key, Some(window), factor, None).ok()
} else {
// (the plain development is not binned either when the file has opcodes of this kind: `factor` is 1)
let whole = product::read(&spec.product, &spec.key, 1).ok()?;
@ -357,27 +364,29 @@ fn load_bytes_now(
};
// Embedded lens corrections are applied by the pipeline ("Enable Profile Corrections"), not baked in
// (removed before the camera look's binned sensor proxy too, which needs an empty `OpcodeList3`).
let lens = embedded_lens(&raw.info());
let info = raw.info();
let lens = embedded_lens(&info);
raw.opcodes.list3.retain(|op| !op.is_lens_correction());
let xy = lightcraft_raw::color::as_shot_white_xy(&raw);
let t = lightcraft_raw::color::camera_transform(&raw, xy);
// the source's segmentation mattes (DNG semantic masks), read while the preview is fitted
let (camera_look, mattes) = rayon::join(|| crate::camera_preview::fit_preview(&raw, &bytes, &t), || dng_mattes(&bytes, &raw));
// Without colour matrices of its own, white balance is relative to the as-shot look (`Photo::relative_wb`).
let own_matrix = lightcraft_raw::color::has_matrix(&raw.color);
// the source's segmentation mattes (DNG semantic masks), read while the starting colour is fitted
let ((xy, t, camera_look), mattes) = rayon::join(|| crate::camera_preview::starting_colour(&mut raw, &bytes), || dng_mattes(&bytes, &info));
drop(bytes);
// Previews and thumbnails bin the mosaic straight to (about) the size they need; only
// larger levels (exports, 1:1) demosaic the whole sensor.
let t0 = web_time::Instant::now();
let bin = bin_factor(&raw, max_edge);
// (binned: block means, and which colours had a clipped sample in each block)
let binned = match bin {
Some(k) => raw.develop_binned(k, HIGHLIGHT_CLIP).map_err(|e| e.to_string())?,
Some(k) => raw.develop_binned_masked(k, HIGHLIGHT_CLIP).map_err(|e| e.to_string())?,
None => None,
};
let factor = if binned.is_some() { bin.unwrap_or(1) } else { 1 };
let img = match binned {
Some(img) => img,
let (img, clip_mask) = match binned {
Some((img, mask)) => (img, Some(mask)),
None => {
let method = if max_edge <= 600 { lightcraft_raw::Method::Bilinear } else { lightcraft_raw::Method::Ahd };
raw.develop(method).map_err(|e| e.to_string())?
(raw.develop(method).map_err(|e| e.to_string())?, None)
}
};
// the same development from the denoised mosaic, when the photo has been denoised
@ -399,7 +408,10 @@ fn load_bytes_now(
let finish = |mut img: Rgb32f| {
let mut stages = vec![("develop", t0.elapsed())];
stages.push(("transform", t0.elapsed()));
colour.rebuild_highlights(&mut img);
match &clip_mask {
Some(mask) => colour.rebuild_highlights_masked(&mut img, mask),
None => colour.rebuild_highlights(&mut img),
}
stages.push(("highlights", t0.elapsed()));
colour.apply(&mut img);
stages.push(("colour", t0.elapsed()));
@ -424,7 +436,7 @@ fn load_bytes_now(
}
let twin = twin.map(|picture| finish(picture).0);
let (temp, tint) = xy_to_temp_tint(xy);
let relative = crate::camera_preview::file_local_look(raw.format) && t.matrix_is_fallback;
let relative = crate::camera_preview::file_local_look(raw.format) && !own_matrix;
// White balance re-evaluates the file's own colour model (when it has one and no
// file-local look matrix sits on top of it)
let camera_color = (!t.matrix_is_fallback && camera_look.is_none()).then(|| {
@ -472,7 +484,7 @@ fn local_tone(raw: &lightcraft_raw::RawImage) -> Option<Arc<lightcraft_pipeline:
/// The semantic masks of a DNG that AI masks understand, over the developed image (default crop,
/// oriented like it). Person mattes without a confidently selected pixel are left out (the
/// Subject mask then falls back to its heuristic); a sky matte counts even when empty (no sky).
fn dng_mattes(bytes: &[u8], raw: &lightcraft_raw::RawImage) -> Option<Arc<lightcraft_pipeline::masks::Mattes>> {
fn dng_mattes(bytes: &[u8], raw: &lightcraft_raw::RawInfo) -> Option<Arc<lightcraft_pipeline::masks::Mattes>> {
use lightcraft_pipeline::masks::{MatteKind, Mattes};
if raw.format != lightcraft_raw::RawFormat::Dng {
return None;
@ -708,6 +720,37 @@ mod tests {
assert!(probe_bytes("x.jpg", b"not an image").is_err());
}
/// Issue #548: a small preview bins the mosaic by 8; specular points that clip a single green sample
/// of a block on a grey surface must not become green speckles (the block's green used to take the
/// clip level, which highlight reconstruction then kept).
#[test]
fn coarse_binning_keeps_specular_points_neutral() {
let (w, h) = (256usize, 192usize);
// grey after the file's white balance (as-shot neutral 0.5 : 1 : 0.7), brighter to the right
let neutral = [0.5f32, 1.0, 0.7];
let cfa = lightcraft_raw::Cfa::bayer("RGGB").unwrap();
let mut data: Vec<u16> = (0..w * h)
.map(|i| {
let (x, y) = (i % w, i / w);
let v = (0.3 + 0.2 * x as f32 / w as f32) * neutral[cfa.color_at(x, y).min(2) as usize];
(256.0 + v * (16000.0 - 256.0)).round() as u16
})
.collect();
// one clipped green sample (RGGB: even row, odd column) in every other 8 × 8 block
for by in 0..h / 8 {
for bx in (by % 2..w / 8).step_by(2) {
data[(by * 8 + 4) * w + bx * 8 + 5] = 16000;
}
}
let bytes = crate::tests_xmp::synthetic_dng_of(w, h, "RGGB", data, None, Default::default());
let raw = lightcraft_raw::decode(&bytes).unwrap();
assert_eq!(bin_factor(&raw, 32), Some(8));
let (img, _) = load_bytes(&bytes, 32).unwrap();
assert_eq!((img.width, img.height), (32, 24));
let worst = img.data.iter().map(|p| p[1] / ((p[0] + p[2]) / 2.0).max(1e-6)).fold(0.0f32, f32::max);
assert!(worst < 1.12, "green speckles: green {worst:.2}× the other channels");
}
#[test]
fn cached_denoise_uses_the_same_white_balance_profile_and_orientation_as_plain_raw() {
use lightcraft_denoise::product;
@ -1464,6 +1507,26 @@ mod tests {
let plain = probe_bytes("small.tif", &tiff_shell(24, 16, false)).unwrap();
assert_eq!((plain.kind, plain.preview_only, plain.width, plain.height), (MediaKind::Image, None, 24, 16));
assert!(lightcraft_raw::probe(&tiff_shell(24, 16, false)).is_none());
// TIFFs preserving camera Make tags (SONY, NIKON, etc.) are ordinary images, not raws (#281)
for make in ["SONY", "NIKON CORPORATION", "PENTAX", "RICOH", "SAMSUNG"] {
use lightcraft_tiff::{IfdBuilder, ImageData, TiffWriter, Value, tags as t};
let mut ifd0 = IfdBuilder::new();
ifd0.set(t::IMAGE_WIDTH, Value::Long(vec![24]));
ifd0.set(t::IMAGE_LENGTH, Value::Long(vec![16]));
ifd0.set(t::BITS_PER_SAMPLE, Value::Short(vec![8, 8, 8]));
ifd0.set(t::SAMPLES_PER_PIXEL, Value::Short(vec![3]));
ifd0.set(t::PHOTOMETRIC, Value::Short(vec![2]));
ifd0.set(t::COMPRESSION, Value::Short(vec![1]));
ifd0.set(t::MAKE, Value::Ascii(make.into()));
let px: Vec<u8> = (0..24 * 16).flat_map(|i| [(i % 251) as u8, 128, 200]).collect();
ifd0.set_image(ImageData::Strips { rows_per_strip: 16, strips: vec![px] });
let bytes = TiffWriter::default().write(&[ifd0]).unwrap();
let p = probe_bytes("photo.tif", &bytes).unwrap();
assert_eq!((p.kind, p.preview_only, p.width, p.height), (MediaKind::Image, None, 24, 16), "make: {make}");
let (img, src) = load_bytes(&bytes, 24).unwrap();
assert_eq!((img.width, img.height, src.raw), (24, 16, false), "make: {make}");
}
}
/// Recognised raw containers we don't decode (Minolta MRW here: the preview's first byte is

3
crates/engine/src/import.rs
View File

@ -501,7 +501,8 @@ pub struct ScanProgress {
/// Files to probe, set once the folders are expanded.
pub total: std::sync::atomic::AtomicUsize,
pub done: std::sync::atomic::AtomicUsize,
pub cancel: std::sync::atomic::AtomicBool,
/// Shared with the scan's row in the activity stack (its ✕ sets it).
pub cancel: std::sync::Arc<std::sync::atomic::AtomicBool>,
}
/// The result of [`scan_with`]: the candidates, and the probes to keep for the import that follows.

7
crates/engine/src/lib.rs
View File

@ -14,8 +14,9 @@
#[cfg(not(feature = "denoise"))]
extern crate lightcraft_denoise_core as lightcraft_denoise;
pub mod activity;
pub mod availability;
mod camera_preview;
pub(crate) mod camera_preview;
pub mod camera_profiles;
pub mod cmd;
pub mod config;
@ -261,6 +262,9 @@ pub struct Session {
/// The last Synchronize Folder scan, which `folder.synchronize` acts on while it is current
/// (see [`Session::take_folder_changes`]).
pub folder_changes: Option<sync::FolderChanges>,
/// Background tasks in flight (imports, exports, preview builds…), for the activity stack and
/// `activity.list` / `activity.cancel`.
pub activity: activity::Activity,
/// The last (or running) Build Previews.
pub preview_build: Option<std::sync::Arc<cmd::previews::PreviewBuild>>,
/// Develop defaults applied on import (persisted in prefs.json).
@ -345,6 +349,7 @@ impl Session {
before: Default::default(),
import_probes: Default::default(),
folder_changes: None,
activity: Default::default(),
preview_build: None,
import_defaults: import::ImportDefaults::default(),
cache_mb: 0,

5
crates/engine/src/library.rs
View File

@ -419,6 +419,11 @@ impl Session {
lib.forgot_local = Some(plan);
}
}
// smart previews written by an older camera-look fit are brought up to date in the background
#[cfg(not(target_arch = "wasm32"))]
if on_disk {
crate::cmd::previews::refresh_stale_smart_previews(self);
}
self.library_identity = std::sync::Arc::new(());
Ok(())
}

2
crates/engine/src/media.rs
View File

@ -62,7 +62,7 @@ impl SettingsHashes {
}
/// Bump when the pipeline's output changes, to invalidate cached thumbnails.
pub const RENDER_CACHE_VERSION: u64 = 37;
pub const RENDER_CACHE_VERSION: u64 = 42;
/// Thumbnails render at one of these long edges (so window/cell size changes reuse the cache).
pub const THUMB_SIZES: [usize; 4] = [128, 256, 384, 512];

82
crates/engine/src/model_download.rs
View File

@ -15,6 +15,8 @@ use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{Arc, Mutex, PoisonError};
use std::time::Duration;
use crate::activity::{Activity, Cancel, TaskHandle, Unit};
use lightcraft_denoise::known::Download;
/// The folder inside the models folder where a downloaded file waits to be installed. Its name starts with a dot, so it
@ -43,15 +45,20 @@ pub enum State {
struct Shared {
state: Mutex<State>,
cancel: AtomicBool,
cancel: Arc<AtomicBool>,
/// The download's row in the activity stack.
task: Option<TaskHandle>,
}
impl Shared {
fn new(total: u64) -> Self {
Shared { state: Mutex::new(State::Running { bytes: 0, total }), cancel: AtomicBool::new(false) }
Shared { state: Mutex::new(State::Running { bytes: 0, total }), cancel: Arc::new(AtomicBool::new(false)), task: None }
}
fn set(&self, s: State) {
if let (State::Running { bytes, total }, Some(task)) = (&s, &self.task) {
task.progress(*bytes, *total);
}
*self.state.lock().unwrap_or_else(PoisonError::into_inner) = s;
}
@ -69,18 +76,27 @@ pub struct Downloads {
impl Downloads {
/// Start fetching `spec` into `<models dir>/.downloads/`. Different models can download side by side; the same
/// one is started once.
pub fn start(&mut self, spec: Download, models_dir: &Path) -> Result<(), String> {
/// The activity stack shows it (as `name`) until the thread ends; its ✕ stops it like [`Downloads::discard`].
pub fn start(&mut self, spec: Download, models_dir: &Path, activity: &Activity, name: &str) -> Result<(), String> {
if self.jobs.get(&spec.id).is_some_and(|j| matches!(j.get(), State::Running { .. })) {
return Err("that model is already downloading".into());
}
let staging = models_dir.join(STAGING);
std::fs::create_dir_all(&staging).map_err(|e| format!("could not create the download folder: {e}"))?;
let shared = Arc::new(Shared::new(spec.size_bytes));
let cancel = Arc::new(AtomicBool::new(false));
let task = activity.start("download", "Downloading model", Cancel::Flag(cancel.clone()));
task.set_unit(Unit::Bytes);
task.detail(name);
task.progress(0, spec.size_bytes);
let total = spec.size_bytes;
let shared = Arc::new(Shared { state: Mutex::new(State::Running { bytes: 0, total }), cancel, task: Some(task.handle()) });
let worker = shared.clone();
let id = spec.id.clone();
std::thread::Builder::new()
.name("denoise-model-download".into())
.spawn(move || {
// the row goes when the thread ends, however it ends
let _task = task;
let outcome = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| run(&spec, &spec.url, &staging, &worker)));
if outcome.is_err() {
worker.set(State::Failed("the download stopped unexpectedly".into()));
@ -218,7 +234,7 @@ mod tests {
use std::net::TcpListener;
fn scratch(name: &str) -> PathBuf {
let d = std::env::temp_dir().join(format!("lc-face-dl-{name}-{}", std::process::id()));
let d = std::env::temp_dir().join(format!("lc-denoise-dl-{name}-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&d);
std::fs::create_dir_all(&d).unwrap();
d
@ -254,6 +270,62 @@ mod tests {
base
}
/// A local server that sends `body` a few kilobytes at a time: a download that keeps running for a while.
fn serve_slowly(body: Vec<u8>) -> String {
let l = TcpListener::bind("127.0.0.1:0").unwrap();
let base = format!("http://{}/m", l.local_addr().unwrap());
std::thread::spawn(move || {
for mut s in l.incoming().flatten() {
let mut r = BufReader::new(s.try_clone().unwrap());
loop {
let mut line = String::new();
if r.read_line(&mut line).unwrap_or(0) == 0 || line.trim().is_empty() {
break;
}
}
let _ = s.write_all(format!("HTTP/1.1 200 OK\r\nContent-Length: {}\r\nConnection: close\r\n\r\n", body.len()).as_bytes());
for chunk in body.chunks(4096) {
if s.write_all(chunk).is_err() {
break;
}
std::thread::sleep(Duration::from_millis(20));
}
}
});
base
}
/// Poll until `done` holds (or fail after 10 s).
fn wait(what: &str, done: impl Fn() -> bool) {
let t = std::time::Instant::now();
while !done() {
assert!(t.elapsed() < Duration::from_secs(10), "{what}");
std::thread::sleep(Duration::from_millis(10));
}
}
#[test]
fn a_running_download_shows_in_activity_and_cancels() {
use crate::activity::{Activity, Unit};
let dir = scratch("activity");
let bytes: Vec<u8> = (0..400_000u32).map(|i| (i % 251) as u8).collect();
let mut spec = spec_for(&bytes);
spec.url = format!("{}/{}", serve_slowly(bytes), spec.file_name);
let activity = Activity::default();
let mut d = Downloads::default();
d.start(spec, &dir, &activity, "Test Model").unwrap();
let rows = activity.list();
assert_eq!(rows.len(), 1);
assert_eq!((rows[0].kind, rows[0].label.as_str(), rows[0].detail.as_str()), ("download", "Downloading model", "Test Model"));
assert_eq!((rows[0].unit, rows[0].total), (Unit::Bytes, 400_000));
assert!(rows[0].cancellable);
activity.cancel(rows[0].id).unwrap();
wait("the download never stopped", || !d.running());
assert_eq!(d.jobs["test-model"].get(), State::Cancelled);
wait("the row stayed", || activity.list().is_empty());
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
fn fetches_checks_and_stages_a_file() {
let dir = scratch("ok");

23
crates/engine/src/segment/download.rs
View File

@ -8,6 +8,8 @@ use std::sync::{Arc, Mutex};
use lightcraft_segment::fetch::{self, Options, Progress};
use crate::activity::{Activity, Cancel, Unit};
/// What the session sees of a download.
#[derive(Clone, Debug, Default, PartialEq, Eq, serde::Serialize)]
pub struct Status {
@ -31,7 +33,7 @@ struct Text {
#[derive(Default)]
struct Shared {
running: AtomicBool,
cancel: AtomicBool,
cancel: Arc<AtomicBool>,
finished: AtomicBool,
done: AtomicU64,
total: AtomicU64,
@ -92,9 +94,17 @@ impl Downloader {
running
}
/// Start downloading `files` from `mirrors` into `dir` on a background thread. False when
/// one is already running.
pub fn start(&self, files: &'static [fetch::FileSpec], mirrors: Vec<String>, dir: PathBuf, opts: Options) -> Result<bool, String> {
/// Start downloading `files` from `mirrors` into `dir` on a background thread, shown in the activity stack as
/// `name` (its ✕ cancels like [`Downloader::cancel`]). False when one is already running.
pub fn start(
&self,
files: &'static [fetch::FileSpec],
mirrors: Vec<String>,
dir: PathBuf,
opts: Options,
activity: &Activity,
name: &str,
) -> Result<bool, String> {
let s = self.shared.clone();
if s.running.swap(true, Ordering::SeqCst) {
return Ok(false);
@ -106,6 +116,10 @@ impl Downloader {
// nothing else runs while `running` was false
*lock(&s.text) = Text::default();
*lock(&self.last) = Text::default();
let task = activity.start("download", "Downloading model", Cancel::Flag(s.cancel.clone()));
task.set_unit(Unit::Bytes);
task.detail(name);
task.progress(0, s.total.load(Ordering::SeqCst));
let guard = Running(s.clone());
let spawned = std::thread::Builder::new().name("sam3-download".into()).spawn(move || {
let s = guard.0.clone();
@ -113,6 +127,7 @@ impl Downloader {
let mut on_progress = |p: &Progress| {
s.done.store(p.done, Ordering::SeqCst);
s.total.store(p.total, Ordering::SeqCst);
task.progress(p.done, p.total);
if p.file != last_file {
last_file.clone_from(&p.file);
lock(&s.text).file.clone_from(&p.file);

6
crates/engine/src/segment/mod.rs
View File

@ -259,7 +259,7 @@ impl Segmenter {
/// Start downloading the model (on a background thread; see `segment.model.download`).
/// `Ok(false)` when it is installed or downloading already.
pub fn start_download(&self) -> Result<bool, String> {
pub fn start_download(&self, activity: &crate::activity::Activity) -> Result<bool, String> {
if !Self::AVAILABLE {
return Err("AI masks are not available in this build".into());
}
@ -273,11 +273,11 @@ impl Segmenter {
if mirrors.is_empty() {
return Err(lightcraft_segment::fetch::no_mirrors_message());
}
self.download.start(lightcraft_segment::fetch::SAM3_FILES, mirrors, dir, lightcraft_segment::fetch::options())
self.download.start(lightcraft_segment::fetch::SAM3_FILES, mirrors, dir, lightcraft_segment::fetch::options(), activity, "SAM 3")
}
#[cfg(not(feature = "sam"))]
{
let _ = dir;
let _ = (dir, activity);
Ok(false)
}
}

56
crates/engine/src/smart.rs
View File

@ -24,6 +24,10 @@ pub fn file_name(p: &Photo) -> String {
/// The most header [`is_valid`] reads (the JSON line before the JPEG).
const HEADER_MAX: u64 = 4096;
/// The header field that records which version of the camera-look fit (`camera_preview::LOOK_VERSION`)
/// wrote the proxy's pixels (colour matrix and profile are baked into them) and stored tone curve.
const LOOK_FIELD: &str = "look_version";
/// Encode a source image (and the decoder's camera tone curve, if any) as a smart preview.
pub fn encode(img: &Rgb32f, tone: Option<&CameraTone>) -> Result<Vec<u8>, String> {
// scale so all but the brightest 0.05 % fit into 0..1
@ -53,7 +57,7 @@ pub fn encode(img: &Rgb32f, tone: Option<&CameraTone>) -> Result<Vec<u8>, String
)
.map_err(|e| e.to_string())?;
let mut out = MAGIC.to_vec();
let mut head = serde_json::json!({"w": img.width, "h": img.height, "scale": scale});
let mut head = serde_json::json!({"w": img.width, "h": img.height, "scale": scale, LOOK_FIELD: crate::camera_preview::LOOK_VERSION});
if let Some(t) = tone {
head["tone"] = serde_json::to_value(t).map_err(|e| e.to_string())?;
}
@ -187,6 +191,56 @@ pub fn is_valid(path: &Path) -> bool {
f.seek(SeekFrom::End(-2)).is_ok() && f.read_exact(&mut end).is_ok() && end == [0xFF, 0xD9]
}
/// The fit version a header records: a missing, negative, fractional or otherwise malformed value
/// counts as 0 (older than any real version).
fn header_look_version(head: &serde_json::Value) -> u64 {
head.get(LOOK_FIELD).and_then(serde_json::Value::as_u64).unwrap_or(0)
}
/// Whether the complete proxy at `path` was written by an older camera-look fit than this build's
/// (so its stored curve, and the colour baked into its pixels, are out of date). A proxy from a
/// newer build is left alone (rebuilding it here would only undo that build's work), as is one
/// that can't be read: those are [`is_valid`]'s business.
pub fn is_stale(path: &Path) -> bool {
use std::io::Read;
if !is_valid(path) {
return false;
}
let Ok(f) = std::fs::File::open(path) else { return false };
let mut head = Vec::with_capacity(1024);
if f.take(HEADER_MAX).read_to_end(&mut head).is_err() {
return false;
}
let Some(rest) = head.strip_prefix(MAGIC) else { return false };
let Some(nl) = rest.iter().position(|b| *b == b'\n') else { return false };
rest.get(..nl)
.and_then(|h| serde_json::from_slice::<serde_json::Value>(h).ok())
.is_some_and(|h| header_look_version(&h) < u64::from(crate::camera_preview::LOOK_VERSION))
}
/// The marker file in a smart previews folder: the `LOOK_VERSION` its proxies were last checked
/// at ([`look_checked`]).
const LOOK_MARKER: &str = "look-version";
/// Whether the smart previews folder `dir` has been checked for stale proxies at this build's
/// camera-look fit version (or a newer one). A missing or unreadable marker counts as not.
pub fn look_checked(dir: &Path) -> bool {
use std::io::Read;
let Ok(f) = std::fs::File::open(dir.join(LOOK_MARKER)) else { return false };
let mut text = String::new();
if f.take(64).read_to_string(&mut text).is_err() {
return false;
}
text.trim().parse::<u64>().is_ok_and(|v| v >= u64::from(crate::camera_preview::LOOK_VERSION))
}
/// Record that the proxies in `dir` were checked at this build's camera-look fit version.
pub fn mark_look_checked(dir: &Path) -> Result<(), String> {
let path = dir.join(LOOK_MARKER);
lightcraft_catalog::safe_file::write_atomic(&path, format!("{}\n", crate::camera_preview::LOOK_VERSION).as_bytes())
.map_err(|e| format!("{}: {e}", path.display()))
}
/// Load the proxy at `path`.
pub fn load(path: &Path) -> Result<crate::media::DecodedSource, String> {
let b = std::fs::read(path).map_err(|e| format!("{}: {e}", path.display()))?;

36
crates/engine/src/tests.rs
View File

@ -721,6 +721,42 @@ fn build_previews_fills_the_cache() {
assert!(s.execute("library.buildPreviews", &json!({"size": "huge"})).is_err());
}
/// Wait (at most a minute) until no background task is listed.
fn wait_activity_empty(s: &Session) {
let t0 = std::time::Instant::now();
while !s.activity.list().is_empty() {
assert!(t0.elapsed().as_secs() < 60, "tasks still listed: {:?}", s.activity.list());
std::thread::sleep(std::time::Duration::from_millis(20));
}
}
/// Issue #345: a background preview build is a row in the activity stack while it runs.
#[test]
fn preview_build_shows_in_activity_until_it_finishes() {
let mut s = demo();
let ids: Vec<u64> = s.catalog.photos().take(4).map(|p| p.id.0).collect();
let r = s.execute("library.buildPreviews", &json!({"ids": ids, "size": "standard", "edge": 256})).unwrap();
let tasks = s.activity.list();
assert_eq!(tasks.first().map(|t| (t.kind, t.total, t.label.as_str())), Some(("previews", 4, "Building previews")), "{tasks:?} {r}");
assert!(tasks[0].cancellable);
wait_activity_empty(&s);
assert_eq!(s.execute("library.previewProgress", &json!({})).unwrap()["running"], false);
}
/// Issue #345: `activity.cancel` stops a preview build like `library.cancelPreviews`.
#[test]
fn activity_cancel_stops_a_preview_build() {
let mut s = demo();
let ids: Vec<u64> = s.catalog.photos().map(|p| p.id.0).collect();
s.execute("library.buildPreviews", &json!({"ids": ids, "size": "full"})).unwrap();
let id = s.activity.list().first().map(|t| t.id).expect("a previews row");
s.execute("activity.cancel", &json!({"id": id})).unwrap();
wait_activity_empty(&s);
let p = s.execute("library.previewProgress", &json!({})).unwrap();
assert_eq!(p["cancelled"], true, "{p}");
assert!(p["done"].as_u64() < p["total"].as_u64(), "stopped part-way: {p}");
}
/// Colour range: a click samples the colour under it, so the mask selects that colour (white
/// in the mask view) and not a different one; ⇧ adds samples, up to five.
#[test]

32
crates/engine/src/tests_denoise.rs
View File

@ -757,3 +757,35 @@ fn the_denoise_switch_validates_arguments_and_accepts_explicit_targets_without_s
drop(x.s);
let _ = std::fs::remove_dir_all(x.dir);
}
#[test]
fn denoise_row_while_queue_is_busy() {
let mut x = setup("activity", true);
let rows = |s: &Session| s.activity.list().into_iter().filter(|t| t.kind == "denoise").collect::<Vec<_>>();
let r = x.s.execute("denoise.pump", &json!({"pace": "full"})).unwrap();
assert!(rows(&x.s).is_empty(), "nothing to make, no row: {r}");
set_amount(&mut x.s, 100.0);
let mut shown = false;
for _ in 0..600 {
let r = x.s.execute("denoise.pump", &json!({"pace": "full"})).unwrap();
let rows = rows(&x.s);
// (what the pump left running or queued is what the row reflects until the next pump)
if x.s.denoise_busy() {
assert_eq!(rows.len(), 1, "{r}");
assert_eq!((rows[0].label.as_str(), rows[0].cancellable, rows[0].unit), ("AI Denoise", false, crate::activity::Unit::Percent), "{r}");
if !r["running"].is_null() {
assert_eq!(rows[0].detail, "ramp.dng", "{r}");
}
shown = true;
} else {
assert!(rows.is_empty(), "{r}");
}
if r["ready"].as_u64() >= Some(1) {
break;
}
std::thread::sleep(std::time::Duration::from_millis(50));
}
assert!(shown, "the photo never showed a row");
assert!(rows(&x.s).is_empty(), "the row goes once the photo is made");
let _ = std::fs::remove_dir_all(&x.dir);
}

36
crates/engine/src/tests_face_recognize.rs
View File

@ -487,3 +487,39 @@ fn every_scanned_face_is_shown_by_the_same_kind_of_box() {
assert!((v.x0 - tight.x0).abs() < 1e-3 && (v.y0 - tight.y0).abs() < 1e-3, "{v:?}");
let _ = std::fs::remove_dir_all(&d);
}
#[test]
fn face_scan_row_follows_pending_photos() {
let d = temp("activity");
let (mut s, _) = setup(&d, 64);
// the last two demo photos, not the first two the other scan tests use: two sessions scanning the same demo scene
// at once can deadlock (issue #630), and this test is about the row, not that
let mut ids = s.catalog.photos().map(|p| p.id).collect::<Vec<_>>();
ids.sort();
let (a, b) = (ids[ids.len() - 2], ids[ids.len() - 1]);
set_regions(&mut s, a, vec![region(0.1, None), region(0.5, None)]);
set_regions(&mut s, b, vec![region(0.3, None)]);
let faces_rows = |s: &Session| s.activity.list().into_iter().filter(|t| t.kind == "faces").collect::<Vec<_>>();
let deadline = std::time::Instant::now() + std::time::Duration::from_secs(60);
let (mut last, mut shown) = (json!(null), false);
while std::time::Instant::now() < deadline {
last = s.execute("faces.pump", &json!({})).unwrap();
let rows = faces_rows(&s);
if last["pendingPhotos"].as_u64().unwrap_or(0) > 0 {
// while photos are left: one row, not cancellable, its whole the most photos left at once
assert_eq!(rows.len(), 1, "{last}");
assert_eq!((rows[0].label.as_str(), rows[0].cancellable), ("Finding faces", false));
assert_eq!(rows[0].total, last["peak"].as_u64().unwrap(), "{last}");
assert_eq!(rows[0].done, rows[0].total - last["pendingPhotos"].as_u64().unwrap());
shown = true;
}
if last["active"] == true && last["pendingPhotos"] == 0 && last["indexedFaces"] == 3 {
break;
}
std::thread::sleep(std::time::Duration::from_millis(50));
}
assert!(shown, "the scan never showed a row: {last}");
assert_eq!((last["pendingPhotos"].clone(), last["peak"].clone()), (json!(0), json!(0)), "{last}");
assert!(faces_rows(&s).is_empty(), "the row goes when the scan is done");
let _ = std::fs::remove_dir_all(&d);
}

212
crates/engine/src/tests_import.rs
View File

@ -497,6 +497,12 @@ fn damaged_smart_previews_are_rebuilt_and_failed_writes_leave_none() {
};
assert_eq!((r["done"].as_u64(), r["failed"].as_u64(), r["repaired"].as_u64()), (Some(1), Some(0), Some(1)), "{r}");
assert_eq!(std::fs::read(&file).unwrap(), full, "rebuilt whole in the background");
// its row in the activity stack (issue #345) goes with it
let t0 = std::time::Instant::now();
while !s.activity.list().is_empty() && t0.elapsed() < std::time::Duration::from_secs(10) {
std::thread::sleep(std::time::Duration::from_millis(5));
}
assert!(s.activity.list().is_empty(), "{:?}", s.activity.list());
let _ = std::fs::remove_dir_all(&src);
let _ = std::fs::remove_dir_all(&lib);
}
@ -1125,6 +1131,212 @@ fn folder_import_picks_up_undecodable_raw_containers_as_preview_only() {
let _ = std::fs::remove_dir_all(&src);
}
/// Rewrite the JSON header line of the smart preview at `file`.
fn edit_smart_header(file: &Path, edit: impl FnOnce(&mut Value)) {
let bytes = std::fs::read(file).unwrap();
let start = b"LCSP1\n".len();
let nl = start + bytes[start..].iter().position(|b| *b == b'\n').unwrap();
let mut head: Value = serde_json::from_slice(&bytes[start..nl]).unwrap();
edit(&mut head);
let mut out = bytes[..start].to_vec();
out.extend_from_slice(head.to_string().as_bytes());
out.extend_from_slice(&bytes[nl..]);
std::fs::write(file, out).unwrap();
}
fn smart_header(file: &Path) -> Value {
let bytes = std::fs::read(file).unwrap();
let start = b"LCSP1\n".len();
let nl = start + bytes[start..].iter().position(|b| *b == b'\n').unwrap();
serde_json::from_slice(&bytes[start..nl]).unwrap()
}
/// A camera tone curve with a recognisable shape (`k` changes it).
fn test_tone(k: f32) -> lightcraft_pipeline::tone::CameraTone {
lightcraft_pipeline::tone::CameraTone::new(std::array::from_fn(|i| {
let x = 0.004 * 1.17f32.powi(i as i32);
[x, 0.95 * (1.0 - (-k * x).exp())]
}))
.unwrap()
}
/// `plain`, but reporting `tone` as the camera look the (current) fit gives the file; counts its calls.
fn look_loader(
plain: crate::media::FileLoader,
tone: lightcraft_pipeline::tone::CameraTone,
calls: std::sync::Arc<std::sync::atomic::AtomicUsize>,
) -> crate::media::FileLoader {
std::sync::Arc::new(move |path, max_edge| {
calls.fetch_add(1, std::sync::atomic::Ordering::SeqCst);
let (img, mut info) = plain(path, max_edge)?;
info.camera_tone = Some(tone);
Ok((img, info))
})
}
/// The follow-up to the camera tone curve change (#499): a smart preview keeps the look the fit
/// gave it, so one written by an older fit is rebuilt when its original is online, kept while the
/// original is offline, and not rebuilt again once current. Only raws that take the per-file
/// camera look go stale (a PNG's proxy never does), and the opening scan runs once per fit version.
#[test]
fn smart_previews_from_an_older_look_fit_are_refreshed() {
use std::sync::atomic::{AtomicUsize, Ordering};
let src = temp_dir("smartlook-src");
let lib = temp_dir("smartlook-lib");
write_png(&src.join("a.png"), 3);
write_png(&src.join("b.png"), 9);
let mut s = Session::new().with_fs();
s.open_library(&lib, false).unwrap();
s.execute("library.import", &json!({"paths": [src.join("a.png").to_string_lossy(), src.join("b.png").to_string_lossy()]})).unwrap();
let find = |s: &Session, name: &str| s.catalog.photos().find(|p| p.file_name == name).unwrap().id;
let (id, png) = (find(&s, "a.png"), find(&s, "b.png"));
// `a` stands for a raw that takes the per-file camera look (the test loader decodes its pixels)
let mut raw = (**s.catalog.photo(id).unwrap()).clone();
raw.format = "ARW".into();
raw.kind = lightcraft_catalog::MediaKind::Raw;
s.catalog.apply(lightcraft_catalog::Op::RemovePhoto { id }).unwrap();
s.catalog.apply(lightcraft_catalog::Op::AddPhoto { photo: Box::new(raw) }).unwrap();
assert!(s.catalog.photo(id).unwrap().relative_wb() && !s.catalog.photo(png).unwrap().relative_wb());
s.execute("library.select", &json!({"ids": [id.0, png.0], "active": id.0})).unwrap();
let dir = s.media.smart_dir.clone().unwrap();
let file = dir.join(crate::smart::file_name(s.catalog.photo(id).unwrap()));
let png_file = dir.join(crate::smart::file_name(s.catalog.photo(png).unwrap()));
let (old, new) = (test_tone(1.5), test_tone(3.5));
let calls = std::sync::Arc::new(AtomicUsize::new(0));
let plain = s.media.file_loader.clone().unwrap();
s.media.file_loader = Some(look_loader(plain.clone(), old, calls.clone()));
let r = s.execute("library.smartPreviews", &json!({})).unwrap();
assert_eq!((r["built"].as_u64(), r["refreshed"].as_u64()), (Some(2), Some(0)), "{r}");
assert_eq!(smart_header(&file)["look_version"], json!(crate::camera_preview::LOOK_VERSION), "the writer stamps the fit version");
assert!(!crate::smart::is_stale(&file));
// current header: kept as it is, the original is not even read
let before = std::fs::read(&file).unwrap();
calls.store(0, Ordering::SeqCst);
s.media.file_loader = Some(look_loader(plain.clone(), new, calls.clone()));
let r = s.execute("library.smartPreviews", &json!({})).unwrap();
assert_eq!((r["built"].as_u64(), r["refreshed"].as_u64()), (Some(2), Some(0)), "{r}");
assert!(std::fs::read(&file).unwrap() == before, "a current proxy is not rebuilt");
assert_eq!(calls.load(Ordering::SeqCst), 0);
// old headers (no field), originals offline: kept unchanged, and still marked stale
for f in [&file, &png_file] {
edit_smart_header(f, |h| {
h.as_object_mut().unwrap().remove("look_version");
});
}
assert!(crate::smart::is_stale(&file), "no field = version 0");
let old_bytes = std::fs::read(&file).unwrap();
let png_bytes = std::fs::read(&png_file).unwrap();
std::fs::rename(&src, src.with_extension("offline")).unwrap();
s.media.forget(id);
s.media.forget(png);
let r = s.execute("library.smartPreviews", &json!({})).unwrap();
assert_eq!((r["built"].as_u64(), r["refreshed"].as_u64(), r["staleKept"].as_u64()), (Some(2), Some(0), Some(1)), "{r}");
assert!(r["failed"].as_array().unwrap().is_empty(), "{r}");
assert!(std::fs::read(&file).unwrap() == old_bytes, "offline: the stored preview and curve are untouched");
assert!(crate::smart::is_stale(&file), "and it stays marked for the next time");
assert_eq!(crate::smart::decode(&old_bytes).unwrap().1, Some(old), "the old look is still there");
// ... and the opening refresh keeps it too, then marks the folder checked
assert!(!crate::smart::look_checked(&dir));
crate::cmd::previews::refresh_stale_smart_previews(&mut s).unwrap().join().unwrap();
assert!(std::fs::read(&file).unwrap() == old_bytes);
assert!(crate::smart::look_checked(&dir));
// the original comes back: the scan has run at this version, so the next opening leaves it;
// Build Smart Previews rebuilds it with the current look
std::fs::rename(src.with_extension("offline"), &src).unwrap();
calls.store(0, Ordering::SeqCst);
assert!(crate::cmd::previews::refresh_stale_smart_previews(&mut s).is_none(), "checked once per version");
assert!(std::fs::read(&file).unwrap() == old_bytes);
let r = s.execute("library.smartPreviews", &json!({})).unwrap();
assert_eq!((r["refreshed"].as_u64(), calls.load(Ordering::SeqCst)), (Some(1), 1), "{r}");
assert_eq!(smart_header(&file)["look_version"], json!(crate::camera_preview::LOOK_VERSION));
assert_eq!(crate::smart::decode(&std::fs::read(&file).unwrap()).unwrap().1, Some(new), "rebuilt with the current look");
assert!(!crate::smart::is_stale(&file));
assert!(std::fs::read(&png_file).unwrap() == png_bytes, "a PNG's proxy is never rebuilt for the look");
// a folder checked at an older version (or not at all) is scanned once at the next opening
edit_smart_header(&file, |h| h["look_version"] = json!(0));
std::fs::write(dir.join("look-version"), b"0\n").unwrap();
assert!(!crate::smart::look_checked(&dir));
s.media.forget(id);
calls.store(0, Ordering::SeqCst);
crate::cmd::previews::refresh_stale_smart_previews(&mut s).unwrap().join().unwrap();
assert_eq!(calls.load(Ordering::SeqCst), 1, "only the raw's original is read");
assert!(!crate::smart::is_stale(&file));
assert!(std::fs::read(&png_file).unwrap() == png_bytes);
assert!(crate::smart::look_checked(&dir));
// once only
let rebuilt = std::fs::read(&file).unwrap();
assert!(crate::cmd::previews::refresh_stale_smart_previews(&mut s).is_none());
let r = s.execute("library.smartPreviews", &json!({})).unwrap();
assert_eq!((r["refreshed"].as_u64(), calls.load(Ordering::SeqCst)), (Some(0), 1), "{r}");
assert!(std::fs::read(&file).unwrap() == rebuilt);
// Build Smart Previews refreshes a stale one the same way
edit_smart_header(&file, |h| h["look_version"] = json!(0));
s.media.file_loader = Some(plain);
let r = s.execute("library.smartPreviews", &json!({})).unwrap();
assert_eq!(r["refreshed"], 1, "{r}");
assert_eq!(crate::smart::decode(&std::fs::read(&file).unwrap()).unwrap().1, None, "this fit gives no curve: none is kept");
let _ = std::fs::remove_dir_all(&src);
let _ = std::fs::remove_dir_all(&lib);
}
/// The look marker: missing, damaged or older than this build's fit version counts as unchecked.
#[test]
fn smart_preview_look_marker_is_read_safely() {
let dir = temp_dir("smartlook-marker");
assert!(!crate::smart::look_checked(&dir), "no marker");
let cur = crate::camera_preview::LOOK_VERSION;
for (text, checked) in [
(String::new(), false),
("banana".into(), false),
("-1".into(), false),
(format!("{}", cur.saturating_sub(1)), false),
("9".repeat(200), false),
(format!("{cur}\n"), true),
(format!(" {}\n", cur + 1), true),
] {
std::fs::write(dir.join("look-version"), &text).unwrap();
assert_eq!(crate::smart::look_checked(&dir), checked, "{text:?}");
}
std::fs::remove_file(dir.join("look-version")).unwrap();
crate::smart::mark_look_checked(&dir).unwrap();
assert!(crate::smart::look_checked(&dir));
assert!(crate::smart::mark_look_checked(&dir.join("missing")).is_err());
let _ = std::fs::remove_dir_all(&dir);
}
/// A hostile or damaged `look_version` never panics: anything that is not a non-negative integer
/// counts as 0 (rebuilt when the original is online); a version from a newer build is left alone.
#[test]
fn malformed_smart_preview_look_versions_are_handled() {
let dir = temp_dir("smartlook-bad");
let img = lightcraft_scenes::demo_library()[0].render(32, 20);
let file = dir.join("a.lcsp");
std::fs::write(&file, crate::smart::encode(&img, None).unwrap()).unwrap();
let cur = u64::from(crate::camera_preview::LOOK_VERSION);
for bad in [json!("1"), json!(-1), json!(1.5), json!(null), json!([]), json!({}), json!(true), json!(1e30), json!(-0.0)] {
let shown = bad.to_string();
edit_smart_header(&file, |h| h["look_version"] = bad);
assert!(crate::smart::is_valid(&file), "{shown}");
assert!(crate::smart::is_stale(&file), "{shown} counts as version 0");
assert!(crate::smart::decode(&std::fs::read(&file).unwrap()).is_ok(), "{shown}");
}
for fine in [json!(cur), json!(cur + 1), json!(u64::MAX)] {
let shown = fine.to_string();
edit_smart_header(&file, |h| h["look_version"] = fine);
assert!(!crate::smart::is_stale(&file), "{shown}");
}
// not a header at all, or missing
assert!(!crate::smart::is_stale(&dir.join("missing.lcsp")));
std::fs::write(&file, b"LCSP1\n{not json\n\xFF\xD8").unwrap();
assert!(!crate::smart::is_stale(&file));
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
fn a_file_gone_since_the_import_review_is_not_imported() {
let src = temp_dir("gone-since-review");

48
crates/engine/src/tests_segment.rs
View File

@ -230,3 +230,51 @@ fn object_clicks_are_capped() {
assert!(!s.segmenter.busy());
let _ = std::fs::remove_dir_all(&dir);
}
#[cfg(feature = "sam")]
#[test]
fn a_running_model_download_shows_in_activity_and_cancels() {
use std::io::{BufRead, BufReader, Write};
// a local mirror that sends its files slowly, so the download is still running when it is cancelled
let l = std::net::TcpListener::bind("127.0.0.1:0").unwrap();
let base = format!("http://{}/sam3", l.local_addr().unwrap());
std::thread::spawn(move || {
for mut c in l.incoming().flatten() {
let mut r = BufReader::new(c.try_clone().unwrap());
let mut line = String::new();
while r.read_line(&mut line).unwrap_or(0) > 2 {
line.clear();
}
let _ = c.write_all(b"HTTP/1.1 200 OK\r\nContent-Length: 1000000\r\nConnection: close\r\n\r\n");
for _ in 0..250 {
if c.write_all(&[b' '; 4000]).is_err() {
break;
}
std::thread::sleep(Duration::from_millis(20));
}
}
});
let dir = tmp("download-activity");
let mirrors = dir.with_extension("activity-mirrors.txt");
std::fs::write(&mirrors, format!("# test mirror\n{base}\n")).unwrap();
let mut s = Session::with_demo();
s.segmenter.dir = Some(dir.clone());
s.segmenter.mirrors_file = Some(mirrors.clone());
assert_eq!(s.execute("segment.model.download", &json!({"acknowledged": true})).unwrap()["started"], true);
let rows = s.activity.list();
assert_eq!(rows.len(), 1);
assert_eq!((rows[0].kind, rows[0].label.as_str(), rows[0].detail.as_str()), ("download", "Downloading model", "SAM 3"));
assert_eq!(rows[0].unit, crate::activity::Unit::Bytes);
assert!(rows[0].cancellable);
s.execute("activity.cancel", &json!({"id": rows[0].id})).unwrap();
let t = Instant::now();
while s.segmenter.download_status().running || !s.activity.list().is_empty() {
assert!(t.elapsed() < Duration::from_secs(10), "the download never stopped");
std::thread::sleep(Duration::from_millis(10));
}
let e = s.segmenter.download_status().error.unwrap_or_default();
assert!(e.contains("cancelled"), "{e}");
assert!(!s.segmenter.installed());
let _ = std::fs::remove_dir_all(&dir);
let _ = std::fs::remove_file(&mirrors);
}

72
crates/raw/src/binned.rs
View File

@ -8,9 +8,13 @@
//! interpolation, so there are no demosaicing artefacts; the colour planes are offset by less than
//! one output pixel, which is invisible at these scales.
//!
//! Clipping is preserved for [`crate::highlight::reconstruct`]: a channel whose block contains a
//! sample at or above `clip` takes that block's maximum sample of the colour (instead of a mean that
//! would fall just below the clip level and escape reconstruction).
//! Clipping is decided per sample, before averaging. [`RawImage::develop_binned_masked`] keeps every
//! channel the mean of its samples and says, per output pixel, which colours had a sample at or above
//! `clip` (for [`crate::highlight::reconstruct_masked`], which rebuilds them with the mean as a lower
//! bound). A specular point that clips one green sample of the 32 in an 8 × 8 block thus counts as
//! partly clipped without the block's green jumping to the clip level, which would read as a green
//! speckle once reconstructed. [`RawImage::develop_binned`] instead gives such a channel the block's
//! maximum sample of the colour, so that plain value thresholds see it as clipped.
use crate::{Normalized, RawData, RawImage, Result, Rgb32f};
use rayon::prelude::*;
@ -31,6 +35,19 @@ impl RawImage {
/// not oriented). `None` when the data cannot be binned ([`RawImage::can_bin`]) or carries an
/// `OpcodeList3` (whose operations are defined at full resolution); callers then demosaic.
pub fn develop_binned(&self, k: usize, clip: f32) -> Result<Option<Rgb32f>> {
Ok(self.bin(k, clip, true)?.map(|(img, _)| img))
}
/// Like [`RawImage::develop_binned`], with every channel the mean of its block's samples, and the
/// clip mask: per output pixel, bits [`crate::highlight::CLIPPED_R`] / `_G` / `_B` for the colours
/// with a sample at or above `clip` in the block. Hand both to [`crate::highlight::reconstruct_masked`].
pub fn develop_binned_masked(&self, k: usize, clip: f32) -> Result<Option<(Rgb32f, Vec<u8>)>> {
self.bin(k, clip, false)
}
/// The binned image and its clip mask; `keep_max`: a channel with a clipped sample takes the
/// block's maximum sample of the colour instead of the mean.
fn bin(&self, k: usize, clip: f32, keep_max: bool) -> Result<Option<(Rgb32f, Vec<u8>)>> {
self.validate()?;
if !self.can_bin(k) || !self.opcodes.list3.is_empty() {
return Ok(None);
@ -63,7 +80,8 @@ impl RawImage {
.collect();
let wlen = bw * k;
let mut out = Rgb32f::new(bw, bh);
out.data.par_chunks_mut(bw).enumerate().for_each(|(by, row)| {
let mut mask = vec![0u8; bw * bh];
out.data.par_chunks_mut(bw).zip(mask.par_chunks_mut(bw)).enumerate().for_each(|(by, (row, mrow))| {
// the block row's k sample rows, normalised
let mut rows = vec![0f32; k * wlen];
for dy in 0..k {
@ -86,7 +104,7 @@ impl RawImage {
}
}
let py = (c.y + by * k) % ph;
for (bx, px) in row.iter_mut().enumerate() {
for (bx, (px, m)) in row.iter_mut().zip(mrow.iter_mut()).enumerate() {
let (layout, inv) = &layouts[py * pw + (c.x + bx * k) % pw];
let (mut sum, mut max) = ([0f32; 3], [f32::MIN; 3]);
for dy in 0..k {
@ -98,11 +116,13 @@ impl RawImage {
}
}
for ch in 0..3 {
px[ch] = if max[ch] >= clip { max[ch] } else { sum[ch] * inv[ch] };
let clipped = max[ch] >= clip;
px[ch] = if clipped && keep_max { max[ch] } else { sum[ch] * inv[ch] };
*m |= u8::from(clipped) << ch;
}
}
});
Ok(Some(out))
Ok(Some((out, mask)))
}
}
@ -177,6 +197,44 @@ mod tests {
assert!((xb.get(4, 3)[2] - 0.3).abs() < 2e-3);
}
/// Issue #548: specular points that clip one green sample in some 8 × 8 blocks of a grey surface.
/// Taking the block's maximum makes those blocks' green the clip level, which reconstruction then
/// keeps (a green speckle); the masked binning keeps the mean and marks the green as clipped.
#[test]
fn clipping_is_decided_before_averaging() {
let img = Rgb32f::filled(64, 64, [0.3, 0.3, 0.3]);
let mut raw = raw_from(&mosaic_from_rgb(&img, &Cfa::bayer("RGGB").unwrap()), 0.0, 4095.0);
if let RawData::U16(d) = &mut raw.data {
// one green sample (RGGB: odd column of an even row) in every other block
for by in 0..8 {
for bx in (by % 2..8).step_by(2) {
d[(by * 8 + 2) * 64 + bx * 8 + 3] = 4095;
}
}
}
let wb = [1.0f32; 3];
let (mut masked, mask) = raw.develop_binned_masked(8, 0.99).unwrap().unwrap();
assert_eq!(mask.len(), 64);
assert_eq!(mask.iter().filter(|&&m| m == highlight::CLIPPED_G).count(), 32);
assert!(mask.iter().all(|&m| m & (highlight::CLIPPED_R | highlight::CLIPPED_B) == 0));
// the clipped block's green is its mean: (31 · 0.3 + 1) / 32
assert!((masked.get(0, 0)[1] - (31.0 * 0.3 + 1.0) / 32.0).abs() < 1e-3, "{:?}", masked.get(0, 0));
assert_eq!(highlight::reconstruct_masked(&mut masked, wb, 0.99, [1.0; 3], &mask), 32);
let green = |img: &Rgb32f| img.data.iter().map(|p| p[1] / p[0]).fold(0.0f32, f32::max);
// (no more than the mean itself: the clipped sample counts at the clip level)
assert!(green(&masked) < 1.08, "green excess {}", green(&masked));
// the maximum rule: a green speckle at the clip level
let mut max = raw.develop_binned(8, 0.99).unwrap().unwrap();
assert!(max.get(0, 0)[1] >= 0.99);
highlight::reconstruct(&mut max, wb, 0.99);
assert!(green(&max) > 3.0, "{}", green(&max));
// without clipped samples both are the same plain means, and the mask is empty
let clean = raw_from(&mosaic_from_rgb(&img, &Cfa::bayer("RGGB").unwrap()), 0.0, 4095.0);
let (a, m) = clean.develop_binned_masked(8, 0.99).unwrap().unwrap();
assert!(m.iter().all(|&v| v == 0));
assert!(a.data == clean.develop_binned(8, 0.99).unwrap().unwrap().data);
}
#[test]
fn binning_matches_demosaic_downscaled_on_smooth_images() {
let img = Rgb32f::from_fn(64, 48, |x, y| [0.2 + 0.3 * (x as f32 / 64.0), 0.4 - 0.2 * (y as f32 / 48.0), 0.25]);

85
crates/raw/src/highlight.rs
View File

@ -9,6 +9,8 @@
//! there is none within reach. Right past the rim, the rim's colour carries on. Fully clipped pixels become
//! neutral at the brightest plausible level; partly clipped ones fade into it on their way from the first channel
//! clipping to the last, pale ones from the start.
//! - [`reconstruct_masked`]: the same for a binned image, whose clipped channels a mask tells (block means with a
//! clipped sample among them).
//!
//! Why the care: the white-balance gains push clipped channels apart (green clips first, red and blue keep
//! rising), so whatever colour the rebuilt pixels carry is hidden at Neutral only by the tone map's roll-off to
@ -116,7 +118,7 @@ fn sample<const N: usize>(l: &Level<N>, scale: f32, x: usize, y: usize) -> [f32;
/// The half-resolution source level of `img`: every `2 × 2` block's source-weighted chromaticity sums, with
/// blocks near clipped pixels or on strong edges left out. Also returns, per cell, whether a clipped pixel lies
/// in its row within [`MARGIN`] cells (the vertical half of that test is left to the readers).
fn sources(img: &Rgb32f, wb: [f32; 3], clip: f32) -> (Level<3>, Vec<bool>) {
fn sources(img: &Rgb32f, wb: [f32; 3], clip: f32, clipped: &impl Clipped) -> (Level<3>, Vec<bool>) {
let (w, h) = (img.width, img.height);
let (cw, ch) = (w.div_ceil(2), h.div_ceil(2));
// per cell: [red sum, blue sum, weight] densities, luminance, whether it holds a clipped pixel
@ -131,13 +133,15 @@ fn sources(img: &Rgb32f, wb: [f32; 3], clip: f32) -> (Level<3>, Vec<bool>) {
for x in 0..cw {
let (mut a, mut sum, mut n, mut hit) = ([0f32; 3], 0f32, 0usize, false);
let span = 2 * x..(2 * x + 2).min(w);
for p in r0.get(span.clone()).unwrap_or(&[]).iter().chain(r1.get(span).unwrap_or(&[])) {
let (i0, i1) = (2 * y * w + 2 * x, (2 * y + 1) * w + 2 * x);
let pixels = r0.get(span.clone()).unwrap_or(&[]).iter().zip(i0..).chain(r1.get(span).unwrap_or(&[]).iter().zip(i1..));
for (p, i) in pixels {
let q = [p[0] * wb[0], p[1] * wb[1], p[2] * wb[2]];
let s = q[0].max(0.0) + q[1].max(0.0) + q[2].max(0.0);
sum += s;
n += 1;
let top = p[0].max(p[1]).max(p[2]);
if top >= clip {
if clipped(i, p).iter().any(|&c| c) {
hit = true;
continue;
}
@ -191,7 +195,7 @@ fn sources(img: &Rgb32f, wb: [f32; 3], clip: f32) -> (Level<3>, Vec<bool>) {
/// above [`RIM_LO`] of the clip level, weighted (steeply) towards the clip, as densities. Pixels next to clipped
/// ones (`margin`, by half-resolution cell) are left out, like the colour sources: they were demosaiced partly from
/// clipped samples, and lens fringes along clipped content would pass for its colour.
fn rim(img: &Rgb32f, wb: [f32; 3], clip: f32, margin: &(dyn Fn(usize, usize) -> bool + Sync)) -> Level<4> {
fn rim(img: &Rgb32f, wb: [f32; 3], clip: f32, clipped: &impl Clipped, margin: &(dyn Fn(usize, usize) -> bool + Sync)) -> Level<4> {
let (w, h) = (img.width, img.height);
let (cw, ch) = (w.div_ceil(4), h.div_ceil(4));
let mut cells = vec![[0f32; 4]; cw * ch];
@ -202,7 +206,7 @@ fn rim(img: &Rgb32f, wb: [f32; 3], clip: f32, margin: &(dyn Fn(usize, usize) ->
for (i, p) in img.data.get(sy * w + 4 * x..sy * w + (4 * x + 4).min(w)).unwrap_or(&[]).iter().enumerate() {
n += 1;
let top = p[0].max(p[1]).max(p[2]) / clip;
if !(RIM_LO..1.0).contains(&top) || margin((4 * x + i) / 2, sy / 2) {
if !(RIM_LO..1.0).contains(&top) || margin((4 * x + i) / 2, sy / 2) || clipped(sy * w + 4 * x + i, p).iter().any(|&c| c) {
continue;
}
let k = ((top - RIM_LO) / (1.0 - RIM_LO)).powi(4);
@ -341,9 +345,44 @@ pub fn reconstruct(img: &mut Rgb32f, wb: [f32; 3], clip: f32) -> usize {
/// chromaticity of reliable unclipped content nearby, and that of `white` where there is none.
/// Returns the number of pixels that had at least one clipped channel; the others are left exactly as they were.
pub fn reconstruct_with(img: &mut Rgb32f, wb: [f32; 3], clip: f32, white: [f32; 3]) -> usize {
rebuild_clipped(img, wb, clip, white, &|_, p| by_value(p, clip))
}
/// Bits of a [`reconstruct_masked`] mask entry: which channels of the pixel had a clipped sample.
pub const CLIPPED_R: u8 = 1;
pub const CLIPPED_G: u8 = 2;
pub const CLIPPED_B: u8 = 4;
/// [`reconstruct_with`] for a binned image ([`crate::RawImage::develop_binned_masked`]): its values are block
/// means, and `mask` (one entry per pixel, bits [`CLIPPED_R`], [`CLIPPED_G`], [`CLIPPED_B`]) says which colours
/// had a clipped sample in the block. Those channels count as clipped with their mean as the lower bound of their
/// true value, so one clipped specular sample among many unclipped ones neither escapes reconstruction nor
/// (as the block's maximum would) lifts the channel to the clip level, which reads as a coloured speckle (#548).
/// A mask of the wrong length is ignored (clipping is then told by the values alone).
pub fn reconstruct_masked(img: &mut Rgb32f, wb: [f32; 3], clip: f32, white: [f32; 3], mask: &[u8]) -> usize {
if mask.len() != img.data.len() {
return reconstruct_with(img, wb, clip, white);
}
rebuild_clipped(img, wb, clip, white, &|i, p| {
let m = mask.get(i).copied().unwrap_or(0);
let v = by_value(p, clip);
[v[0] || m & CLIPPED_R != 0, v[1] || m & CLIPPED_G != 0, v[2] || m & CLIPPED_B != 0]
})
}
/// Which channels of pixel `i` (values `p`) are clipped.
trait Clipped: Fn(usize, &[f32; 3]) -> [bool; 3] + Sync {}
impl<F: Fn(usize, &[f32; 3]) -> [bool; 3] + Sync> Clipped for F {}
#[inline]
fn by_value(p: &[f32; 3], clip: f32) -> [bool; 3] {
[p[0] >= clip, p[1] >= clip, p[2] >= clip]
}
/// [`reconstruct_with`], with `clipped` telling the clipped channels.
fn rebuild_clipped(img: &mut Rgb32f, wb: [f32; 3], clip: f32, white: [f32; 3], clipped: &impl Clipped) -> usize {
let (w, h) = (img.width, img.height);
let is_clipped = |p: &[f32; 3]| p[0] >= clip || p[1] >= clip || p[2] >= clip;
let count = img.data.par_iter().filter(|p| is_clipped(p)).count();
let count = img.data.par_iter().enumerate().filter(|(i, p)| clipped(*i, p).iter().any(|&c| c)).count();
if count == 0 || w == 0 || h == 0 {
return 0;
}
@ -354,7 +393,7 @@ pub fn reconstruct_with(img: &mut Rgb32f, wb: [f32; 3], clip: f32, white: [f32;
let neutral = [white[0] / total, white[1] / total, white[2] / total];
// A clipped pixel is never a source, so its chromaticity is always the bilinear sample of the
// half-resolution estimate: the pyramid starts there, straight from the image.
let (first, near) = sources(img, wb, clip);
let (first, near) = sources(img, wb, clip, clipped);
let mut levels = vec![first];
while let Some(l) = levels.last().filter(|l| l.w > 1 || l.h > 1) {
let next = pull(l);
@ -392,7 +431,7 @@ pub fn reconstruct_with(img: &mut Rgb32f, wb: [f32; 3], clip: f32, white: [f32;
let Some(field) = levels.first() else { return count };
// the rim (level 0 of its pyramid holds the estimate), if there is one anywhere at all
let rims = {
let mut levels = vec![rim(img, wb, clip, &needed)];
let mut levels = vec![rim(img, wb, clip, clipped, &needed)];
while let Some(l) = levels.last().filter(|l| l.w > 1 || l.h > 1) {
let next = pull(l);
levels.push(next);
@ -405,7 +444,7 @@ pub fn reconstruct_with(img: &mut Rgb32f, wb: [f32; 3], clip: f32, white: [f32;
img.data.par_chunks_mut(w).enumerate().for_each(|(y, row)| {
for (x, px) in row.iter_mut().enumerate() {
let p = *px;
let cl = [p[0] >= clip, p[1] >= clip, p[2] >= clip];
let cl = clipped(y * w + x, &p);
if !cl.iter().any(|&b| b) {
continue;
}
@ -813,4 +852,30 @@ mod tests {
.fold(f64::MAX, f64::min);
eprintln!("reconstruct 24 MP: {ms:.0} ms");
}
/// A mask decides which channels are clipped (their values are lower bounds), whatever their values; the
/// unmarked pixels are left as they were; a mask of another size leaves clipping to the values.
#[test]
fn a_mask_marks_the_clipped_channels() {
let base = Rgb32f::from_fn(32, 32, |x, _| if x >= 16 { [0.2, 0.5, 0.25] } else { [0.5, 0.5, 0.5] });
let mask: Vec<u8> = (0..32 * 32).map(|i| if i % 32 >= 16 { CLIPPED_R } else { 0 }).collect();
let mut img = base.clone();
assert_eq!(reconstruct_masked(&mut img, [1.0; 3], 0.99, [1.0; 3], &mask), 512);
for (p, b) in img.data.iter().zip(&base.data) {
assert!(p.iter().all(|v| v.is_finite()), "{p:?}");
if b[0] == 0.5 {
assert_eq!(p, b);
} else {
// red rebuilt from at least its mean, the others kept
assert!(p[0] >= 0.2 && p[1] == 0.5 && p[2] == 0.25, "{p:?}");
}
}
// red is rebuilt above its mean, towards the neighbours' neutral (as far as the pixel's own blue agrees)
assert!(img.get(24, 16)[0] > 0.3, "{:?}", img.get(24, 16));
// the values alone see nothing clipped here, nor does a mask of another size
let mut plain = base.clone();
assert_eq!(reconstruct(&mut plain, [1.0; 3], 0.99), 0);
assert_eq!(reconstruct_masked(&mut plain, [1.0; 3], 0.99, [1.0; 3], &mask[1..]), 0);
assert!(plain.data == base.data);
}
}

91
crates/raw/src/lib.rs
View File

@ -7,7 +7,8 @@
//! - [`RawImage::normalized`] subtracts black, scales white to 1.0 and crops to the active area (applying DNG
//! `OpcodeList1`/`OpcodeList2`); [`demosaic`] turns CFA data into camera-RGB [`Rgb32f`];
//! [`RawImage::develop`] does all of it plus `OpcodeList3` and the default crop; [`RawImage::develop_binned`]
//! produces the same at 1/k of the size straight from the mosaic (previews, thumbnails).
//! produces the same at 1/k of the size straight from the mosaic (previews, thumbnails;
//! [`RawImage::develop_binned_masked`] with the mask of blocks holding clipped samples, for [`highlight`]).
//! - [`color`] implements the DNG colour model (dual-illuminant interpolation, forward matrices, white balance)
//! and produces camera → linear Rec.2020 D65 matrices; [`profile`] reads and applies a DNG's own profile
//! look tables and tone curve, and [`gaintable`] its gain table map (Apple ProRAW's local tone mapping).
@ -19,7 +20,8 @@
//! packed 2/5/7, the 16-bit words of the oldest bodies), Pentax PEF (uncompressed, Huffman), Olympus ORF (uncompressed: 16-bit words, 12-bit XZ-2 words, 12-bit 16-byte blocks of the E-300/E-330/E-500).
//! [`embedded_preview`] covers these containers' JPEG previews. Variants we can't decode yet (compressed ORF, CR3 unverified marker families / C-RAW configurations) return [`RawError::Unsupported`]; each vendor module documents its sources
//! (public specifications, tag-name documentation, black-box analysis of CC0 samples) and gaps. Non-DNG files carry no
//! colour matrix: [`color`] falls back to a documented neutral model. The decoders never panic on malformed input.
//! colour matrix: [`spectral`] has matrices fitted to measured spectral sensitivities for 52 models, and
//! otherwise [`color`] falls back to a documented neutral model. The decoders never panic on malformed input.
#![forbid(unsafe_code)]
#![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable)]
@ -38,6 +40,7 @@ mod preview;
pub mod profile;
mod saturation;
pub mod semantic;
pub mod spectral;
mod tiffraw;
mod unpack;
mod vendor;
@ -159,23 +162,28 @@ pub fn probe(bytes: &[u8]) -> Option<RawFormat> {
}
let make = t.find(lightcraft_tiff::tags::MAKE).and_then(|e| e.value.as_str()).unwrap_or_default().to_ascii_uppercase();
let has_cfa = has_raw_ifd(&t);
// Pentax's reader takes the layout from the Exif `CFAPattern` when no IFD has a CFA photometric (uncompressed
// and PackBits PEFs), so for the Pentax family that tag counts as raw evidence too
let pentax_raw = has_cfa || t.exif().is_some_and(|e| e.contains(vendor::EXIF_CFA_PATTERN));
// Samsung's raw IFD uses private compressions 32769..=32773 (`vendor::srw`), 32773 being PackBits elsewhere
let samsung_raw = has_cfa || t.all_ifds().iter().any(|i| i.u16(lightcraft_tiff::tags::COMPRESSION).is_some_and(|c| (32769..=32773).contains(&c)));
if make.starts_with("CANON") && t.ifds.len() >= 4 && t.ifds[3].u16(lightcraft_tiff::tags::COMPRESSION) == Some(6) {
return Some(RawFormat::Cr2);
}
if make.starts_with("NIKON") {
return Some(if has_cfa || t.all_ifds().len() > 1 { RawFormat::Nef } else { RawFormat::Nrw });
if make.starts_with("NIKON") && has_cfa {
return Some(if t.all_ifds().len() > 1 { RawFormat::Nef } else { RawFormat::Nrw });
}
if make.starts_with("SONY") {
if make.starts_with("SONY") && has_cfa {
return Some(RawFormat::Arw);
}
if make.starts_with("PENTAX") || make.starts_with("RICOH") {
if (make.starts_with("PENTAX") || make.starts_with("RICOH")) && pentax_raw {
return Some(RawFormat::Pef);
}
// a Samsung-branded body built on a Pentax design writes a Pentax-style maker note (the note's own magic)
if make.starts_with("SAMSUNG") && has_pentax_maker_note(&t, bytes) {
if make.starts_with("SAMSUNG") && pentax_raw && has_pentax_maker_note(&t, bytes) {
return Some(RawFormat::Pef);
}
if make.starts_with("SAMSUNG") {
if make.starts_with("SAMSUNG") && samsung_raw {
return Some(RawFormat::Srw);
}
if dng::is_plain_cfa_tiff(&t, bytes) {
@ -194,11 +202,15 @@ fn has_pentax_maker_note(t: &Tiff, bytes: &[u8]) -> bool {
}
/// Whether some IFD is marked as raw: CFA photometric, or a raw-only compression value (99 is not a
/// registered TIFF compression; Leaf MOS files use it for their tiled 16-bit lossless-JPEG raw).
/// registered TIFF compression; Leaf MOS files use it for their tiled 16-bit lossless-JPEG raw),
/// or vendor-specific raw tags (Sony tone curve, CFA pattern, or Pentax compression 65535).
fn has_raw_ifd(t: &Tiff) -> bool {
t.all_ifds().iter().any(|i| {
i.u16(lightcraft_tiff::tags::PHOTOMETRIC) == Some(lightcraft_tiff::tags::photometric::CFA)
|| i.u16(lightcraft_tiff::tags::COMPRESSION).is_some_and(|c| c == 34713 || c == 32767 || c == 32769 || c == 32770 || c == 99)
|| i.u16(lightcraft_tiff::tags::COMPRESSION).is_some_and(|c| matches!(c, 34713 | 32767 | 32769 | 32770 | 99 | 65535))
|| i.contains(0x7010)
|| i.contains(lightcraft_tiff::tags::CFA_PATTERN_EP)
|| i.contains(lightcraft_tiff::tags::CFA_REPEAT_PATTERN_DIM)
})
}
@ -910,6 +922,25 @@ mod tests {
assert_eq!(probe(&padded(&[rgb_ifd(16, 12), rgb_ifd(400, 300)])), None);
// no dimensions at all
assert_eq!(probe(&padded(&[IfdBuilder::new().with(t::MAKE, Value::Ascii("X".into()))])), None);
// raw evidence without a CFA photometric: Pentax's Exif `CFAPattern` (uncompressed / PackBits PEFs) and
// Samsung's private compressions (here 32773, PackBits elsewhere: NX1, NX500)
let mut pef = rgb_ifd(400, 300);
pef.set(t::MAKE, Value::Ascii("PENTAX Corporation".into()));
let mut exif = exif_size(400, 300);
exif.set(vendor::EXIF_CFA_PATTERN, Value::Undefined(vec![0, 2, 0, 2, 0, 1, 1, 2]));
pef.set_child(t::EXIF_IFD, exif);
assert_eq!(probe(&padded(&[pef])), Some(RawFormat::Pef));
let mut srw = rgb_ifd(400, 300);
srw.set(t::MAKE, Value::Ascii("SAMSUNG".into()));
srw.set(t::COMPRESSION, Value::Short(vec![32773]));
assert_eq!(probe(&padded(&[srw])), Some(RawFormat::Srw));
// camera-authored or exported TIFFs preserving camera Make tags remain ordinary images
for make in ["SONY", "NIKON CORPORATION", "PENTAX", "RICOH", "SAMSUNG"] {
let mut cam = rgb_ifd(400, 300);
cam.set(t::MAKE, Value::Ascii(make.into()));
cam.set_child(t::EXIF_IFD, exif_size(400, 300));
assert_eq!(probe(&padded(&[cam])), None, "{make} TIFF should not be probed as raw");
}
}
// --- containers that are recognised but not decoded, with a preview ---
@ -1077,22 +1108,36 @@ mod tests {
assert_eq!(r.cfa.unwrap().name(), "RGGB");
}
/// A lossless-JPEG scan whose predictor selection value is outside 0 to 7 (the 3FR/FFF files use 8) is
/// not a coding the lossless decoder knows: the file stays an undecodable raw, with that as the reason,
/// and keeps the stand-in preview an unsupported container gets.
/// Hasselblad's own lossless-JPEG coding (selection value 8, as in FFF and most compressed 3FR files) of the
/// same layout decodes: the file is read like the uncompressed one.
#[test]
fn selection_value_8_cfa_tiff_decodes() {
let strip = ljpeg::tests::encode_hasselblad(&cfa_samples(), 8, 6);
let bytes = plain_cfa_tiff(7, strip, None);
assert_eq!(probe(&bytes), Some(RawFormat::CfaTiff));
let r = decode(&bytes).unwrap();
assert_eq!((r.format, r.width, r.height), (RawFormat::CfaTiff, 8, 6));
assert_eq!(r.data, RawData::U16(cfa_samples()));
assert_eq!(r.cfa.unwrap().name(), "RGGB");
assert_eq!(probe_info(&bytes).unwrap().format, RawFormat::CfaTiff);
}
/// A lossless-JPEG scan whose predictor selection value is neither 0 to 7 (T.81) nor 8 (Hasselblad) is not a
/// coding the lossless decoder knows: the file stays an undecodable raw, with that as the reason, and keeps the
/// stand-in preview an unsupported container gets.
#[test]
fn cfa_tiff_with_an_unknown_predictor_stays_undecodable() {
let mut strip = ljpeg::encode(&cfa_samples(), 8, 6, 1, 16, 1, 0);
let sos = strip.windows(2).position(|w| w == [0xff, 0xda]).unwrap();
let ss = sos + 2 + 2 + 1 + 2;
assert_eq!(strip[ss], 1, "selection value byte");
strip[ss] = 8;
assert!(matches!(ljpeg::decode(&strip, 1 << 20), Err(RawError::Unsupported(w)) if w.contains("selection value 8")));
strip[ss] = 9;
assert!(matches!(ljpeg::decode(&strip, 1 << 20), Err(RawError::Unsupported(w)) if w.contains("selection value 9")));
let bytes = plain_cfa_tiff(7, strip, None);
assert_eq!(probe(&bytes), Some(RawFormat::OtherTiff));
assert!(!RawFormat::OtherTiff.is_supported());
let Err(RawError::Unsupported(why)) = decode(&bytes) else { panic!("expected Unsupported") };
assert!(why.contains("lossless JPEG") && why.contains("selection value 8"), "{why}");
assert!(why.contains("lossless JPEG") && why.contains("selection value 9"), "{why}");
}
/// Other codings of a CFA IFD (here Deflate) are not claimed by the plain reader.
@ -1106,17 +1151,23 @@ mod tests {
/// Samsung file stays with the Samsung format.
#[test]
fn samsung_make_with_a_pentax_maker_note_is_pef() {
let with_note = |note: &[u8]| {
let with_note = |note: &[u8], cfa: bool| {
let mut ifd = rgb_ifd(16, 12);
if cfa {
ifd.set(t::PHOTOMETRIC, Value::Short(vec![t::photometric::CFA]));
}
ifd.set(t::MAKE, Value::Ascii("SAMSUNG TECHWIN".into()));
let mut exif = IfdBuilder::new();
exif.set(t::MAKER_NOTE, Value::Undefined(note.to_vec()));
ifd.set_child(t::EXIF_IFD, exif);
write(&[ifd])
};
assert_eq!(probe(&with_note(b"AOC\0MM\0\x01\0\0\0\0\0\0")), Some(RawFormat::Pef));
assert_eq!(probe(&with_note(b"PENTAX \0MM\0\0\0\0\0\0\0")), Some(RawFormat::Pef));
assert_eq!(probe(&with_note(b"STMN100\0\0\0\0\0\0\0\0\0")), Some(RawFormat::Srw));
assert_eq!(probe(&with_note(b"AOC\0MM\0\x01\0\0\0\0\0\0", true)), Some(RawFormat::Pef));
assert_eq!(probe(&with_note(b"PENTAX \0MM\0\0\0\0\0\0\0", true)), Some(RawFormat::Pef));
assert_eq!(probe(&with_note(b"STMN100\0\0\0\0\0\0\0\0\0", true)), Some(RawFormat::Srw));
// without raw evidence (an RGB image from such a body) neither reader claims it (#281)
assert_eq!(probe(&with_note(b"AOC\0MM\0\x01\0\0\0\0\0\0", false)), None);
assert_eq!(probe(&with_note(b"STMN100\0\0\0\0\0\0\0\0\0", false)), None);
}
#[test]

320
crates/raw/src/ljpeg.rs
View File

@ -5,6 +5,8 @@
//! restart intervals, `SSSS = 16` differences.
//! - [`decode`] returns the frame as interleaved samples in raster order (`width × height × components`),
//! which callers re-tile (DNG tiles, CR2 slices).
//! - Selection value 8, which T.81 leaves undefined, is Hasselblad's coding in 3FR and FFF files
//! ([`decode`] reads it; [`decode_hasselblad`] documents the rule and how it was established).
//! - [`encode`] writes a conforming stream with per-component optimal Huffman tables built with the
//! T.81 Annex K.2 procedure — used by tests (bit-exact round trips) and later by DNG export.
@ -81,7 +83,7 @@ impl Huffman {
}
#[inline]
pub fn decode(&self, br: &mut BitReader) -> Result<u8, RawError> {
pub fn decode<B: Bits>(&self, br: &mut B) -> Result<u8, RawError> {
let peek = br.peek(LOOKUP);
let (len, sym) = self.fast[peek as usize];
if len > 0 {
@ -103,6 +105,33 @@ impl Huffman {
}
}
/// A source of entropy-coded bits, first bit most significant.
pub trait Bits {
/// The next `n` (1..=32) bits without consuming them; zeros past the end.
fn peek(&mut self, n: u32) -> u32;
fn consume(&mut self, n: u32);
#[inline]
fn get(&mut self, n: u32) -> u32 {
if n == 0 {
return 0;
}
let v = self.peek(n);
self.consume(n);
v
}
}
impl Bits for BitReader<'_> {
#[inline]
fn peek(&mut self, n: u32) -> u32 {
BitReader::peek(self, n)
}
#[inline]
fn consume(&mut self, n: u32) {
BitReader::consume(self, n)
}
}
/// MSB-first bit reader over JPEG entropy-coded data (handles `FF 00` stuffing; stops at markers, feeding zeros).
pub struct BitReader<'a> {
data: &'a [u8],
@ -190,7 +219,7 @@ impl<'a> BitReader<'a> {
/// Difference value for category `ssss` (T.81 Table H.2 / F.12 EXTEND).
#[inline]
pub fn diff_value(br: &mut BitReader, ssss: u8) -> i32 {
pub fn diff_value<B: Bits>(br: &mut B, ssss: u8) -> i32 {
match ssss {
0 => 0,
16 => 32768,
@ -325,9 +354,9 @@ fn parse_header(d: &[u8]) -> Result<Header, RawError> {
}
h.predictor = seg[1 + 2 * ns];
h.pt = seg[3 + 2 * ns] & 15;
if h.predictor > 7 {
if h.predictor > 8 {
return Err(RawError::Unsupported(format!(
"lossless JPEG scan with predictor selection value {} (T.81 defines 0 to 7)",
"lossless JPEG scan with predictor selection value {} (T.81 defines 0 to 7, Hasselblad uses 8)",
h.predictor
)));
}
@ -347,6 +376,9 @@ fn parse_header(d: &[u8]) -> Result<Header, RawError> {
pub fn frame_info(d: &[u8]) -> Result<(usize, usize, usize, u8), RawError> {
let h = parse_header(d)?;
check_full_resolution(&h)?;
if h.predictor == 8 {
check_hasselblad(&h)?;
}
Ok((h.width, h.height, h.comps.len(), h.precision))
}
@ -357,6 +389,139 @@ fn check_full_resolution(h: &Header) -> Result<(), RawError> {
Ok(())
}
/// The layout of every Hasselblad selection-value-8 scan measured; other layouts are not known to follow the rule.
fn check_hasselblad(h: &Header) -> Result<(), RawError> {
if h.comps.len() != 1 || h.precision != 16 || h.pt != 0 || h.restart != 0 || h.width == 0 || !h.width.is_multiple_of(2) {
return Err(RawError::Unsupported(format!(
"lossless JPEG with selection value 8 and {} components, {} bits, point transform {}, restart interval {}, width {}",
h.comps.len(),
h.precision,
h.pt,
h.restart,
h.width
)));
}
Ok(())
}
/// Bits of Hasselblad's entropy data: 32-bit little-endian words, each read from its most significant bit, with
/// no byte stuffing (every byte is data; `FF` bytes are not markers).
struct WordBits<'a> {
data: &'a [u8],
pos: usize,
buf: u64,
bits: u32,
/// Zero bits fed past the last whole word (consumption reached them once `overrun > bits`).
overrun: u32,
}
impl<'a> WordBits<'a> {
fn new(data: &'a [u8]) -> Self {
Self { data, pos: 0, buf: 0, bits: 0, overrun: 0 }
}
#[inline]
fn fill(&mut self) {
while self.bits <= 32 {
let word = match self.data.get(self.pos..self.pos + 4) {
Some(w) => {
self.pos += 4;
u32::from_le_bytes([w[0], w[1], w[2], w[3]])
}
None => {
self.overrun = self.overrun.saturating_add(32);
0
}
};
self.buf |= (word as u64) << (32 - self.bits);
self.bits += 32;
}
}
}
impl Bits for WordBits<'_> {
#[inline]
fn peek(&mut self, n: u32) -> u32 {
if self.bits < n {
self.fill();
}
(self.buf >> (64 - n)) as u32
}
#[inline]
fn consume(&mut self, n: u32) {
self.buf <<= n;
self.bits -= n;
}
}
/// Hasselblad's lossless JPEG with selection value 8 (3FR and FFF raws).
///
/// The rule was established by measurement on 14 sample files with the `ljpeg-explorer` tool, which tested it
/// against alternative bit orders, stuffing, sample groupings, category-16 treatments and predictors:
/// - the entropy data are 32-bit little-endian words, each read from its most significant bit, without byte
/// stuffing;
/// - each row is coded in pairs of samples: both Huffman categories (`SSSS`, T.81 table) come first, then both
/// differences;
/// - a difference has `SSSS` extra bits, extended as in T.81 Table H.2; after `SSSS = 16` come sixteen bits that
/// take no part, and the difference is -32768 (equal to T.81's +32768 modulo 2^16);
/// - a sample is the previous sample of the same column parity in its row plus the difference, modulo 2^16; before
/// the first pair of every row both are 32768 (half the 16-bit range).
///
/// Provenance: the first hypothesis for this rule was recalled by the AI model that wrote this code from its training
/// data, which includes other raw converters; it was treated as one candidate, corrected where the files disagreed
/// (the byte order of the words) and confirmed by an exhaustive black-box search over 17,280 layout hypotheses on the
/// CC0 sample files. No other converter's source was consulted, and the code is original.
pub fn decode_hasselblad(d: &[u8], max_samples: usize) -> Result<Frame, RawError> {
let h = parse_header(d)?;
if h.predictor != 8 {
return Err(err("not a selection value 8 scan"));
}
hasselblad_frame(d, &h, max_samples)
}
fn hasselblad_frame(d: &[u8], h: &Header, max_samples: usize) -> Result<Frame, RawError> {
check_hasselblad(h)?;
let (w, ht) = (h.width, h.height);
let total = w.checked_mul(ht).ok_or_else(|| err("frame too large"))?;
if total > max_samples {
return Err(RawError::Limit("lossless JPEG frame larger than expected"));
}
let entropy = &d[h.scan_start..];
if (entropy.len() as u64 + 64) * 8 < total as u64 {
return Err(err("entropy data too short for frame"));
}
let table =
h.table_for.first().and_then(|&t| h.tables.get(t)).and_then(Option::as_ref).ok_or_else(|| err("scan uses an undefined Huffman table"))?;
let mut out = vec![0u16; total];
let mut br = WordBits::new(entropy);
let start = 1u32 << (h.precision - 1);
for row in out.chunks_exact_mut(w) {
let mut last = [start, start];
// the width is even (checked), so every sample is in a pair
for pair in row.as_chunks_mut::<2>().0 {
let ssss = [table.decode(&mut br)?, table.decode(&mut br)?];
for (k, s) in ssss.into_iter().enumerate() {
let diff = match s {
0..=15 => diff_value(&mut br, s),
16 => {
br.get(16);
-32768
}
_ => return Err(err("invalid difference category")),
};
let v = (last[k] as i32 + diff) as u32 & 0xffff;
last[k] = v;
pair[k] = v as u16;
}
}
// bits consumed beyond the last whole word
if br.overrun > br.bits {
return Err(err("entropy data exhausted"));
}
}
Ok(Frame { width: w, height: ht, components: 1, precision: h.precision, predictor: 8, point_transform: 0, data: out })
}
fn check_subsampled(h: &Header) -> Result<usize, RawError> {
let vertical = match h.sampling.as_slice() {
[0x22, 0x11, 0x11] => 2,
@ -481,6 +646,9 @@ pub(crate) fn decode_subsampled(d: &[u8], max_samples: usize, prediction: Predic
pub fn decode(d: &[u8], max_samples: usize) -> Result<Frame, RawError> {
let h = parse_header(d)?;
check_full_resolution(&h)?;
if h.predictor == 8 {
return hasselblad_frame(d, &h, max_samples);
}
let (w, ht, nc) = (h.width, h.height, h.comps.len());
if w == 0 || ht == 0 {
return Err(err("zero frame size"));
@ -988,6 +1156,150 @@ pub(crate) mod tests {
let _ = decode(&bad, usize::MAX);
}
// --- selection value 8 (Hasselblad 3FR / FFF) ---
/// SOI, SOF3 (16 bits, `height` x `width`, one component), a DHT and an SOS with selection value 8, then
/// `entropy` and EOI.
fn hasselblad_stream(width: u16, height: u16, counts: &[u8; 16], values: &[u8], entropy: &[u8]) -> Vec<u8> {
let mut out = vec![0xff, 0xd8, 0xff, 0xc3, 0, 11, 16];
out.extend_from_slice(&height.to_be_bytes());
out.extend_from_slice(&width.to_be_bytes());
out.extend_from_slice(&[1, 0, 0x11, 0]);
out.extend_from_slice(&[0xff, 0xc4]);
out.extend_from_slice(&((2 + 17 + values.len()) as u16).to_be_bytes());
out.push(0);
out.extend_from_slice(counts);
out.extend_from_slice(values);
out.extend_from_slice(&[0xff, 0xda, 0, 8, 1, 0, 0, 8, 0, 0]);
out.extend_from_slice(entropy);
out.extend_from_slice(&[0xff, 0xd9]);
out
}
/// Codes: SSSS 0 = `00`, 1 = `01`, 2 = `100`, 15 = `101`, 16 = `110`.
const HAND_COUNTS: [u8; 16] = [0, 2, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0];
const HAND_VALUES: [u8; 5] = [0, 1, 2, 15, 16];
/// A 4 x 2 frame written out bit by bit:
/// - row 0, pair 1: codes `110` `110` (SSSS 16 twice), then 16 bits each (`FFFF`, `7FFF`) that do not count:
/// both samples are 32768 - 32768 = 0;
/// - row 0, pair 2: codes `01` `100`, bits `1` (+1) and `01` (-2): 0 + 1 = 1 and 0 - 2 = 65534 (modulo 2^16);
/// - row 1, pair 1: codes `00` `01`, bit `0` (-1): both predictors start again at 32768, so 32768 and 32767;
/// - row 1, pair 2: codes `100` `00`, bits `11` (+3): 32771 and 32767.
///
/// 58 bits, padded with zeros to two 32-bit words `DBFFFDFF` `FD9450C0`, stored little-endian. The `FF FD`
/// they start with would be a marker to a T.81 reader: this coding has no byte stuffing.
const HAND_WORDS: [u8; 8] = [0xff, 0xfd, 0xff, 0xdb, 0xc0, 0x50, 0x94, 0xfd];
const HAND_SAMPLES: [u16; 8] = [0, 0, 1, 65534, 32768, 32767, 32771, 32767];
#[test]
fn selection_value_8_known_answer() {
let enc = hasselblad_stream(4, 2, &HAND_COUNTS, &HAND_VALUES, &HAND_WORDS);
assert_eq!(frame_info(&enc).unwrap(), (4, 2, 1, 16));
let f = decode(&enc, 8).unwrap();
assert_eq!((f.width, f.height, f.components, f.predictor), (4, 2, 1, 8));
assert_eq!(f.data, HAND_SAMPLES);
assert_eq!(decode_hasselblad(&enc, 8).unwrap().data, HAND_SAMPLES);
assert!(matches!(decode(&enc, 7), Err(RawError::Limit(_))));
}
/// The same bits in plain byte order (not 32-bit little-endian words) are a different stream.
#[test]
fn selection_value_8_reads_little_endian_words() {
let mut plain = HAND_WORDS;
plain[..4].reverse();
plain[4..].reverse();
let enc = hasselblad_stream(4, 2, &HAND_COUNTS, &HAND_VALUES, &plain);
assert_ne!(decode(&enc, 8).map(|f| f.data).ok(), Some(HAND_SAMPLES.to_vec()));
}
#[test]
fn selection_value_8_rejects_bad_streams() {
// one word is not enough for the second row
let short = hasselblad_stream(4, 2, &HAND_COUNTS, &HAND_VALUES, &HAND_WORDS[..4]);
assert!(decode(&short, 8).is_err());
// a category above 16
let enc = hasselblad_stream(4, 2, &HAND_COUNTS, &[0, 1, 2, 15, 17], &HAND_WORDS);
assert!(decode(&enc, 8).is_err());
// layouts never seen with selection value 8 are not guessed at
let mut odd = hasselblad_stream(3, 2, &HAND_COUNTS, &HAND_VALUES, &HAND_WORDS);
assert!(matches!(frame_info(&odd), Err(RawError::Unsupported(_))));
assert!(matches!(decode(&odd, 8), Err(RawError::Unsupported(_))));
odd = hasselblad_stream(4, 2, &HAND_COUNTS, &HAND_VALUES, &HAND_WORDS);
odd[6] = 14; // precision
assert!(matches!(decode(&odd, 8), Err(RawError::Unsupported(_))));
// selection values above 8 stay unknown
let mut nine = hasselblad_stream(4, 2, &HAND_COUNTS, &HAND_VALUES, &HAND_WORDS);
let sos = nine.windows(2).position(|w| w == [0xff, 0xda]).unwrap();
nine[sos + 7] = 9;
assert!(matches!(decode(&nine, 8), Err(RawError::Unsupported(w)) if w.contains("selection value 9")));
// every prefix fails cleanly
let enc = hasselblad_stream(4, 2, &HAND_COUNTS, &HAND_VALUES, &HAND_WORDS);
for n in 0..enc.len() {
let _ = decode(&enc[..n], 8);
}
}
/// Test-only writer of the selection-value-8 coding (the inverse of the rule in [`decode_hasselblad`]), with
/// a 5-bit code for every category. Category-16 differences carry `FFFF` as their ignored bits.
pub(crate) fn encode_hasselblad(data: &[u16], width: usize, height: usize) -> Vec<u8> {
assert!(width.is_multiple_of(2) && data.len() == width * height);
let mut bits: Vec<bool> = vec![];
let put = |bits: &mut Vec<bool>, v: u32, n: u32| {
for i in (0..n).rev() {
bits.push((v >> i) & 1 == 1);
}
};
for row in data.chunks_exact(width) {
let mut last = [32768i32, 32768];
for pair in row.as_chunks::<2>().0 {
let mut d = [0i32; 2];
for k in 0..2 {
let mut v = (pair[k] as i32 - last[k]) & 0xffff;
if v >= 32768 {
v -= 65536;
}
d[k] = v;
last[k] = pair[k] as i32;
}
let s = [ssss_of(d[0]), ssss_of(d[1])];
put(&mut bits, s[0] as u32, 5);
put(&mut bits, s[1] as u32, 5);
for k in 0..2 {
match s[k] {
0 => {}
16 => put(&mut bits, 0xffff, 16),
n => {
let v = if d[k] < 0 { d[k] - 1 } else { d[k] };
put(&mut bits, v as u32 & ((1 << n) - 1), n as u32);
}
}
}
}
}
let mut entropy = vec![];
for word in bits.chunks(32) {
let w = word.iter().enumerate().fold(0u32, |a, (i, &b)| a | ((b as u32) << (31 - i)));
entropy.extend_from_slice(&w.to_le_bytes());
}
let mut counts = [0u8; 16];
counts[4] = 17;
let values: Vec<u8> = (0..=16).collect();
hasselblad_stream(width as u16, height as u16, &counts, &values, &entropy)
}
#[test]
fn selection_value_8_round_trips_through_the_test_writer() {
// extremes force SSSS 16 and wrap-around; noise exercises every other category
let mut data = noise(10 * 7, 16, 9);
data[0] = 0;
data[1] = 65535;
data[2] = 65535;
data[3] = 0;
let enc = encode_hasselblad(&data, 10, 7);
assert_eq!(decode(&enc, 70).unwrap().data, data);
}
#[test]
fn code_lengths_are_limited_to_16() {
// Fibonacci-like frequencies produce very long codes before adjustment

171
crates/raw/src/spectral.rs Normal file
View File

@ -0,0 +1,171 @@
//! Camera colour matrices from measured spectral sensitivities, for raws that carry no colour matrix.
//!
//! [`find`] looks a camera up by its Exif make and model in a table generated by
//! `tools/spectral_camera_table.py` from rawtoaces-data (Academy Software Foundation, Apache-2.0): per model, DNG
//! `ColorMatrix1/2` and `ForwardMatrix1/2` for Standard Illuminant A and D65, fitted to the camera's measured spectral
//! sensitivities. [`Camera::fill`] writes them into a file's [`ColorData`], and [`crate::color`]'s DNG model does the
//! rest. Other names of a covered camera (Rebel T3i for EOS 600D) come from rawtoaces-data's alias list. No number
//! comes from Adobe or from a GPL raw decoder. Which colour a raw starts from is the engine's choice
//! (`docs/camera-preview-colour.md`).
use crate::ColorData;
use lightcraft_color::Mat3;
#[path = "spectral_table.rs"]
mod table;
/// DNG `CalibrationIlluminant` codes of the two matrices: Standard Illuminant A, D65.
const ILLUMINANTS: [u16; 2] = [17, 21];
/// One camera's colour matrices.
#[derive(Clone, Copy, Debug, PartialEq)]
pub struct Camera {
pub make: &'static str,
pub model: &'static str,
/// XYZ → camera under Standard Illuminant A and D65 (`ColorMatrix1/2`).
pub color_matrix: [Mat3; 2],
/// White-balanced camera → XYZ D50 (`ForwardMatrix1/2`).
pub forward_matrix: [Mat3; 2],
}
impl Camera {
/// Give `color` this camera's matrices, when it has no colour matrix of its own. Returns whether it did.
pub fn fill(&self, color: &mut ColorData) -> bool {
if crate::color::has_matrix(color) {
return false;
}
color.illuminant = ILLUMINANTS;
color.color_matrix = self.color_matrix.map(Some);
color.forward_matrix = self.forward_matrix.map(Some);
true
}
}
/// Model names compared without case, the brand, a leading `EOS` or a trailing `DIGITAL`
/// (Exif "Canon EOS 400D DIGITAL" is the table's "EOS 400D").
fn key(brand: &str, model: &str) -> String {
let mut words: Vec<String> = model.split_whitespace().map(str::to_uppercase).collect();
if words.first().is_some_and(|w| *w == brand) {
words.remove(0);
}
if words.first().is_some_and(|w| w == "EOS") {
words.remove(0);
}
if words.len() > 1 && words.last().is_some_and(|w| w == "DIGITAL") {
words.pop();
}
words.join(" ")
}
/// First word of a make, upper case ("NIKON CORPORATION" → "NIKON").
fn brand(make: &str) -> String {
make.split_whitespace().next().unwrap_or_default().to_uppercase()
}
/// The table's camera for a file's Exif make and model, directly or through an alias. Without a make, the model
/// alone must match.
pub fn find(make: Option<&str>, model: &str) -> Option<&'static Camera> {
let make = make.map(brand).filter(|b| !b.is_empty());
let same = |row_make: &str, row_model: &str| {
let row_brand = brand(row_make);
let wanted = key(&row_brand, model);
make.as_ref().is_none_or(|m| *m == row_brand) && !wanted.is_empty() && key(&row_brand, row_model) == wanted
};
match table::ALIASES.iter().find(|(m, alias, _)| same(m, alias)) {
Some((m, _, target)) => table::CAMERAS.iter().find(|c| brand(c.make) == brand(m) && c.model == *target),
None => table::CAMERAS.iter().find(|c| same(c.make, c.model)),
}
}
/// Every camera in the table.
pub fn cameras() -> &'static [Camera] {
table::CAMERAS
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn exif_names_find_their_rows() {
for (make, model, row) in [
("Canon", "Canon EOS 5D Mark III", "EOS 5D Mark III"),
("Canon", "Canon EOS 600D", "EOS 600D"),
("Canon", "Canon EOS-1Ds Mark III", "EOS-1Ds Mark III"),
("Canon", "Canon EOS DIGITAL REBEL XTi", "Digital Rebel XTi"),
("Canon", "Canon EOS R5m2", "EOS R5m2"),
("Canon", "Canon PowerShot S90", "PowerShot S90"),
("NIKON CORPORATION", "NIKON D850", "D850"),
("NIKON CORPORATION", "NIKON Z f", "Z f"),
("SONY", "ILCE-7M3", "ILCE-7M3"),
("FUJIFILM", "X-T4", "X-T4"),
("Panasonic", "DC-GX9", "DC-GX9"),
] {
assert_eq!(find(Some(make), model).map(|c| c.model), Some(row), "{make} {model}");
}
assert_eq!(find(None, "NIKON D850").map(|c| c.model), Some("D850"));
assert_eq!(find(None, "ILCE-7M3").map(|c| c.model), Some("ILCE-7M3"));
}
#[test]
fn aliases_reach_the_camera_they_name() {
for (make, model, row) in [
("Canon", "Canon EOS 400D DIGITAL", "Digital Rebel XTi"),
("Canon", "Canon EOS Kiss Digital X", "Digital Rebel XTi"),
("Canon", "Canon EOS REBEL T3i", "EOS 600D"),
("Canon", "Canon EOS Kiss X5", "EOS 600D"),
("Canon", "Canon EOS R5 Mark II", "EOS R5m2"),
("NIKON CORPORATION", "NIKON D800", "D800E"),
("NIKON CORPORATION", "NIKON Z 6_2", "Z f"),
("SONY", "ILCE-7RM4A", "ILCE-7RM4"),
("FUJIFILM", "GFX100S", "GFX 100"),
] {
assert_eq!(find(Some(make), model).map(|c| c.model), Some(row), "{make} {model}");
}
}
#[test]
fn other_cameras_and_other_makes_find_nothing() {
assert_eq!(find(Some("Canon"), "Canon EOS 7D"), None);
assert_eq!(find(Some("SONY"), "ILCE-7M2"), None);
assert_eq!(find(Some("OLYMPUS IMAGING CORP."), "E-M5"), None);
// the model of another make
assert_eq!(find(Some("Canon"), "D850"), None);
assert_eq!(find(Some("SONY"), "X-T4"), None);
assert_eq!(find(None, ""), None);
assert_eq!(find(Some(""), ""), None);
}
#[test]
fn every_row_is_a_usable_dng_profile() {
assert_eq!(cameras().len(), 52);
for c in cameras() {
assert_eq!(find(Some(c.make), c.model), Some(c), "{} {}", c.make, c.model);
for m in c.color_matrix.iter().chain(&c.forward_matrix) {
assert!(m.0.iter().flatten().all(|v| v.is_finite() && v.abs() < 4.0), "{}", c.model);
assert!(m.inverse().is_some(), "{}", c.model);
}
// a forward matrix maps white-balanced white to D50 (DNG spec, chapter 6)
for fm in &c.forward_matrix {
let w = fm.apply([1.0; 3]);
let d50 = lightcraft_color::D50.to_xyz();
assert!((0..3).all(|i| (w[i] - d50[i]).abs() < 1e-4), "{} {w:?}", c.model);
}
}
}
#[test]
fn fill_keeps_a_files_own_matrices() {
let c = find(Some("Canon"), "Canon EOS 600D").unwrap();
let mut color = ColorData::default();
assert!(c.fill(&mut color));
assert_eq!(color.illuminant, [17, 21]);
assert_eq!(color.color_matrix, [Some(c.color_matrix[0]), Some(c.color_matrix[1])]);
assert_eq!(color.forward_matrix, [Some(c.forward_matrix[0]), Some(c.forward_matrix[1])]);
let own = Mat3([[0.5, 0.0, 0.0], [0.0, 1.0, 0.0], [0.0, 0.0, 0.7]]);
let mut color = ColorData { illuminant: [21, 0], color_matrix: [Some(own), None], ..Default::default() };
let before = color.clone();
assert!(!c.fill(&mut color));
assert_eq!(color, before);
}
}

349
crates/raw/src/spectral_table.rs Normal file
View File

@ -0,0 +1,349 @@
// Generated by tools/spectral_camera_table.py from rawtoaces-data (https://github.com/AcademySoftwareFoundation/rawtoaces-data, commit 5cfa53707b053594b2d0d48e8019fb56e6582d6c).
// Derived from the camera spectral sensitivities of rawtoaces-data (Academy Software Foundation), licensed under
// the Apache License 2.0 (LICENSE-APACHE); see NOTICE and assets/ATTRIBUTION.md. Do not edit by hand.
use super::Camera;
use lightcraft_color::Mat3;
#[rustfmt::skip]
pub(super) const CAMERAS: &[Camera] = &[
Camera {
make: "ARRI",
model: "D21",
color_matrix: [Mat3([[0.933065, -0.03997, 0.042177], [-0.321253, 0.916394, 0.250116], [-0.047929, 0.258088, 0.770888]]), Mat3([[0.841118, 0.11685, -0.05716], [-0.416513, 1.06087, 0.169491], [-0.121062, 0.361236, 0.692345]])],
forward_matrix: [Mat3([[0.913988, -0.102353, 0.152661], [0.315506, 0.849769, -0.165276], [-0.134707, -0.698389, 1.6582]]), Mat3([[1.01361, -0.108757, 0.059444], [0.405694, 0.811177, -0.216871], [-0.027612, -0.326433, 1.17915]])],
},
Camera {
make: "Arashi Vision",
model: "Insta360 X5",
color_matrix: [Mat3([[1.0899, -0.436896, 0.023525], [-0.482767, 1.39879, 0.369626], [-0.056659, 0.169421, 0.72407]]), Mat3([[0.700283, -0.158934, -0.077847], [-0.543955, 1.30291, 0.196622], [-0.148964, 0.25165, 0.516509]])],
forward_matrix: [Mat3([[0.710808, 0.160362, 0.093126], [0.231838, 0.891164, -0.123002], [0.020454, -0.498248, 1.3029]]), Mat3([[0.706099, 0.209581, 0.048616], [0.287186, 0.8971, -0.184286], [0.044158, -0.279999, 1.06095]])],
},
Camera {
make: "Canon",
model: "Digital Rebel XTi",
color_matrix: [Mat3([[0.984721, -0.317618, -0.03823], [-0.536205, 1.41139, 0.499196], [-0.019493, 0.0898, 0.993219]]), Mat3([[0.654378, -0.152128, -0.077165], [-0.504141, 1.17532, 0.279049], [-0.071499, 0.132862, 0.672873]])],
forward_matrix: [Mat3([[0.735947, 0.130604, 0.097745], [0.271991, 0.870186, -0.142177], [-0.013289, -0.229681, 1.06808]]), Mat3([[0.693623, 0.247958, 0.022715], [0.295094, 0.978904, -0.273998], [0.00951, -0.117049, 0.932644]])],
},
Camera {
make: "Canon",
model: "EOS-1D X Mark II",
color_matrix: [Mat3([[0.841821, -0.109388, -0.029826], [-0.469467, 1.34341, 0.484215], [-0.074495, 0.204633, 0.843881]]), Mat3([[0.559829, 0.012974, -0.076841], [-0.474068, 1.17573, 0.252424], [-0.137973, 0.263217, 0.557436]])],
forward_matrix: [Mat3([[0.867342, -0.069209, 0.166163], [0.282942, 0.879178, -0.162119], [0.042308, -0.560464, 1.34326]]), Mat3([[0.866644, -0.0379, 0.135551], [0.340507, 0.913959, -0.254466], [0.035971, -0.324047, 1.11318]])],
},
Camera {
make: "Canon",
model: "EOS-1Ds Mark III",
color_matrix: [Mat3([[0.933464, -0.245919, -0.026807], [-0.571496, 1.49286, 0.379177], [-0.048229, 0.100118, 0.927528]]), Mat3([[0.623037, -0.073582, -0.085177], [-0.54689, 1.28786, 0.213006], [-0.105037, 0.156191, 0.659429]])],
forward_matrix: [Mat3([[0.785005, 0.07525, 0.10404], [0.285685, 0.799015, -0.084701], [0.044611, -0.231741, 1.01223]]), Mat3([[0.764142, 0.106772, 0.093381], [0.318478, 0.835999, -0.154477], [0.03106, -0.128764, 0.922809]])],
},
Camera {
make: "Canon",
model: "EOS-1Ds Mark II",
color_matrix: [Mat3([[0.945891, -0.245875, -0.023592], [-0.58518, 1.49586, 0.412994], [-0.049647, 0.124624, 1.04996]]), Mat3([[0.633076, -0.080316, -0.073882], [-0.552992, 1.27833, 0.227086], [-0.10768, 0.180208, 0.730732]])],
forward_matrix: [Mat3([[0.78531, 0.073316, 0.10567], [0.29637, 0.804671, -0.101041], [0.021036, -0.254656, 1.05872]]), Mat3([[0.780605, 0.119757, 0.063934], [0.332784, 0.851961, -0.184745], [0.021321, -0.137163, 0.940946]])],
},
Camera {
make: "Canon",
model: "EOS 100D",
color_matrix: [Mat3([[0.823379, -0.153284, -0.011827], [-0.451822, 1.34611, 0.422153], [-0.063154, 0.17914, 0.866791]]), Mat3([[0.537713, -0.001778, -0.071702], [-0.479729, 1.20511, 0.230384], [-0.135862, 0.245077, 0.608091]])],
forward_matrix: [Mat3([[0.839709, -0.013024, 0.137611], [0.260264, 0.869968, -0.130232], [0.040567, -0.473274, 1.25781]]), Mat3([[0.852444, -0.005311, 0.117164], [0.33135, 0.880249, -0.211599], [0.038232, -0.260159, 1.04703]])],
},
Camera {
make: "Canon",
model: "EOS 200D",
color_matrix: [Mat3([[0.882798, -0.11491, -0.016841], [-0.443662, 1.31235, 0.491834], [-0.062151, 0.206192, 0.798346]]), Mat3([[0.570466, 0.029473, -0.079371], [-0.461712, 1.15876, 0.257222], [-0.118587, 0.24972, 0.538517]])],
forward_matrix: [Mat3([[0.867592, -0.074026, 0.17073], [0.27684, 0.904905, -0.181745], [0.007997, -0.615366, 1.43247]]), Mat3([[0.884048, -0.067045, 0.147293], [0.347584, 0.917256, -0.26484], [0.021454, -0.322825, 1.12647]])],
},
Camera {
make: "Canon",
model: "EOS 200D II",
color_matrix: [Mat3([[0.820814, -0.08112, -0.025232], [-0.449049, 1.31524, 0.500348], [-0.063522, 0.20839, 0.807016]]), Mat3([[0.534782, 0.039491, -0.075304], [-0.461016, 1.15287, 0.26203], [-0.117011, 0.248633, 0.539767]])],
forward_matrix: [Mat3([[0.879874, -0.096427, 0.180849], [0.283378, 0.900453, -0.183831], [0.006798, -0.61219, 1.4305]]), Mat3([[0.897644, -0.089573, 0.156225], [0.354938, 0.914467, -0.269405], [0.020307, -0.323555, 1.12835]])],
},
Camera {
make: "Canon",
model: "EOS 5DS",
color_matrix: [Mat3([[0.784071, -0.14278, -0.018109], [-0.493528, 1.36672, 0.49299], [-0.067845, 0.178082, 0.792129]]), Mat3([[0.501948, -0.005161, -0.069927], [-0.500698, 1.19504, 0.25793], [-0.125608, 0.228657, 0.525639]])],
forward_matrix: [Mat3([[0.838414, -0.019978, 0.145859], [0.285256, 0.874752, -0.160008], [0.038705, -0.507361, 1.29376]]), Mat3([[0.840997, -0.003166, 0.126464], [0.342942, 0.904091, -0.247033], [0.034419, -0.289165, 1.07985]])],
},
Camera {
make: "Canon",
model: "EOS 5D",
color_matrix: [Mat3([[0.961758, -0.26807, -0.006664], [-0.565407, 1.47679, 0.405541], [-0.067833, 0.12912, 0.951998]]), Mat3([[0.616895, -0.064661, -0.085183], [-0.554321, 1.28625, 0.22097], [-0.137616, 0.197613, 0.656267]])],
forward_matrix: [Mat3([[0.79062, 0.078103, 0.095573], [0.287453, 0.812941, -0.100394], [0.068474, -0.279714, 1.03635]]), Mat3([[0.77434, 0.091394, 0.098562], [0.324069, 0.841224, -0.165293], [0.044245, -0.169979, 0.950838]])],
},
Camera {
make: "Canon",
model: "EOS 5D Mark III",
color_matrix: [Mat3([[0.798227, -0.156522, -0.024001], [-0.479014, 1.37953, 0.412187], [-0.067815, 0.178094, 0.795854]]), Mat3([[0.536462, -0.020759, -0.074508], [-0.493492, 1.22724, 0.222071], [-0.136505, 0.242956, 0.547031]])],
forward_matrix: [Mat3([[0.829015, -0.007278, 0.142558], [0.266702, 0.857593, -0.124295], [0.048011, -0.488746, 1.26584]]), Mat3([[0.818929, 0.023372, 0.121994], [0.319742, 0.879191, -0.198933], [0.040465, -0.281513, 1.06615]])],
},
Camera {
make: "Canon",
model: "EOS 5D Mark II",
color_matrix: [Mat3([[0.765281, -0.126067, -0.028016], [-0.539891, 1.44802, 0.407637], [-0.058008, 0.128461, 0.861097]]), Mat3([[0.508976, -0.002163, -0.072663], [-0.524812, 1.25489, 0.224018], [-0.11344, 0.183978, 0.599963]])],
forward_matrix: [Mat3([[0.84396, -0.009524, 0.12986], [0.296711, 0.799464, -0.096175], [0.052873, -0.321603, 1.09383]]), Mat3([[0.8407, 0.001511, 0.122085], [0.343565, 0.82753, -0.171095], [0.035331, -0.182192, 0.971966]])],
},
Camera {
make: "Canon",
model: "EOS 5D Mark IV",
color_matrix: [Mat3([[0.763394, -0.085035, -0.016567], [-0.442907, 1.31347, 0.486351], [-0.080725, 0.247335, 0.884784]]), Mat3([[0.49672, 0.034424, -0.068025], [-0.461551, 1.16055, 0.255439], [-0.154521, 0.307914, 0.579555]])],
forward_matrix: [Mat3([[0.880649, -0.094084, 0.177732], [0.27666, 0.906137, -0.182797], [0.024429, -0.659213, 1.45989]]), Mat3([[0.898336, -0.087734, 0.153694], [0.347611, 0.924078, -0.271689], [0.035769, -0.378779, 1.16812]])],
},
Camera {
make: "Canon",
model: "EOS 600D",
color_matrix: [Mat3([[0.800214, -0.150575, -0.010842], [-0.462694, 1.36091, 0.41413], [-0.054076, 0.17327, 0.853835]]), Mat3([[0.528239, -0.007001, -0.067784], [-0.485979, 1.21691, 0.225002], [-0.13144, 0.244539, 0.592949]])],
forward_matrix: [Mat3([[0.836952, -0.011114, 0.138458], [0.267123, 0.859868, -0.12699], [0.017395, -0.463764, 1.27147]]), Mat3([[0.849461, 0.005947, 0.108888], [0.332065, 0.875561, -0.207627], [0.033552, -0.262503, 1.05405]])],
},
Camera {
make: "Canon",
model: "EOS M",
color_matrix: [Mat3([[0.858133, -0.188655, -0.015316], [-0.477476, 1.37657, 0.415737], [-0.066593, 0.173487, 0.870331]]), Mat3([[0.559337, -0.024286, -0.076489], [-0.498439, 1.22646, 0.227107], [-0.13663, 0.237788, 0.612213]])],
forward_matrix: [Mat3([[0.818185, 0.015661, 0.130449], [0.263684, 0.85776, -0.121444], [0.046738, -0.442459, 1.22083]]), Mat3([[0.820106, 0.031315, 0.112875], [0.32549, 0.874009, -0.199499], [0.03847, -0.243276, 1.02991]])],
},
Camera {
make: "Canon",
model: "EOS R10",
color_matrix: [Mat3([[0.900395, -0.120204, -0.023257], [-0.450784, 1.32079, 0.490112], [-0.060508, 0.207987, 0.757971]]), Mat3([[0.586274, 0.022629, -0.081629], [-0.465757, 1.16444, 0.255537], [-0.114546, 0.247787, 0.505639]])],
forward_matrix: [Mat3([[0.860416, -0.07424, 0.178119], [0.279245, 0.90614, -0.185385], [-0.005252, -0.64772, 1.47808]]), Mat3([[0.874599, -0.057155, 0.146852], [0.346428, 0.920831, -0.267259], [0.018292, -0.341711, 1.14852]])],
},
Camera {
make: "Canon",
model: "EOS R5",
color_matrix: [Mat3([[0.906792, -0.181028, -0.039982], [-0.497007, 1.37995, 0.46653], [-0.056558, 0.135904, 0.790883]]), Mat3([[0.599617, -0.03146, -0.086837], [-0.500836, 1.20715, 0.246929], [-0.122485, 0.204473, 0.529195]])],
forward_matrix: [Mat3([[0.825544, 0.000874, 0.137878], [0.27921, 0.851443, -0.130653], [0.053855, -0.393303, 1.16455]]), Mat3([[0.80269, 0.040879, 0.120727], [0.321664, 0.899552, -0.221215], [0.039913, -0.245426, 1.03062]])],
},
Camera {
make: "Canon",
model: "EOS R5m2",
color_matrix: [Mat3([[0.925997, -0.201878, -0.03247], [-0.514263, 1.39694, 0.472057], [-0.067213, 0.159605, 0.932813]]), Mat3([[0.603973, -0.037874, -0.08705], [-0.520173, 1.22523, 0.247204], [-0.147012, 0.240988, 0.619754]])],
forward_matrix: [Mat3([[0.816853, 0.016859, 0.130583], [0.284699, 0.847793, -0.132492], [0.049678, -0.384186, 1.15961]]), Mat3([[0.796684, 0.049994, 0.117618], [0.326624, 0.8894, -0.216024], [0.040655, -0.243309, 1.02776]])],
},
Camera {
make: "Canon",
model: "EOS R6",
color_matrix: [Mat3([[0.887235, -0.176902, -0.018642], [-0.462429, 1.33965, 0.473061], [-0.070097, 0.19962, 0.862666]]), Mat3([[0.571385, -0.014759, -0.079163], [-0.485367, 1.18708, 0.251862], [-0.145333, 0.265796, 0.580543]])],
forward_matrix: [Mat3([[0.830203, -0.011614, 0.145706], [0.268315, 0.891706, -0.16002], [0.03391, -0.533608, 1.3248]]), Mat3([[0.829826, 0.011021, 0.123449], [0.329107, 0.920067, -0.249174], [0.037653, -0.307101, 1.09455]])],
},
Camera {
make: "Canon",
model: "EOS R6m2",
color_matrix: [Mat3([[0.904651, -0.185945, -0.032292], [-0.490303, 1.37329, 0.46456], [-0.076482, 0.19622, 0.939437]]), Mat3([[0.595351, -0.032062, -0.086354], [-0.499687, 1.20659, 0.246445], [-0.151602, 0.266204, 0.633694]])],
forward_matrix: [Mat3([[0.822515, -0.0028, 0.14458], [0.275534, 0.86495, -0.140483], [0.04532, -0.468236, 1.24802]]), Mat3([[0.80207, 0.038555, 0.123671], [0.322259, 0.905003, -0.227262], [0.03728, -0.271382, 1.05921]])],
},
Camera {
make: "Canon",
model: "EOS RP",
color_matrix: [Mat3([[0.843269, -0.158656, -0.026198], [-0.497933, 1.37758, 0.476056], [-0.070882, 0.179269, 0.839769]]), Mat3([[0.548578, -0.014251, -0.077893], [-0.504763, 1.20691, 0.250579], [-0.139008, 0.243991, 0.560854]])],
forward_matrix: [Mat3([[0.836529, -0.017241, 0.145008], [0.285362, 0.860694, -0.146057], [0.049163, -0.481077, 1.25702]]), Mat3([[0.825922, 0.011858, 0.126516], [0.334357, 0.898585, -0.232942], [0.038621, -0.283664, 1.07015]])],
},
Camera {
make: "Canon",
model: "EOS R",
color_matrix: [Mat3([[0.732886, -0.080769, -0.010303], [-0.454757, 1.32829, 0.481287], [-0.083062, 0.251827, 0.895746]]), Mat3([[0.473635, 0.038459, -0.063761], [-0.474141, 1.17757, 0.250797], [-0.157456, 0.313293, 0.589901]])],
forward_matrix: [Mat3([[0.884185, -0.093978, 0.174089], [0.284166, 0.893986, -0.178152], [0.023242, -0.654769, 1.45663]]), Mat3([[0.908841, -0.095987, 0.151442], [0.356864, 0.902145, -0.259009], [0.034581, -0.371735, 1.16226]])],
},
Camera {
make: "Canon",
model: "PowerShot S90",
color_matrix: [Mat3([[1.37542, -0.592063, 0.037432], [-0.219062, 1.12456, 0.326212], [-0.021707, 0.09788, 0.58587]]), Mat3([[0.90732, -0.244506, -0.087152], [-0.314903, 1.06475, 0.215416], [-0.11223, 0.179775, 0.439924]])],
forward_matrix: [Mat3([[0.67794, 0.203129, 0.083227], [0.105177, 1.01192, -0.1171], [0.055684, -0.464622, 1.23404]]), Mat3([[0.655349, 0.321862, -0.012915], [0.180096, 1.10102, -0.281111], [0.064028, -0.26917, 1.03025]])],
},
Camera {
make: "Fujifilm",
model: "GFX 100",
color_matrix: [Mat3([[1.37208, -0.633889, 0.081365], [-0.430548, 1.32101, 0.427017], [-0.039908, 0.114944, 0.782345]]), Mat3([[0.834968, -0.233255, -0.07589], [-0.475844, 1.18948, 0.241344], [-0.126239, 0.194812, 0.541708]])],
forward_matrix: [Mat3([[0.673446, 0.25104, 0.039809], [0.200798, 0.950696, -0.151494], [0.031013, -0.340478, 1.13457]]), Mat3([[0.679917, 0.304542, -0.020163], [0.261808, 1.00662, -0.268426], [0.044444, -0.214012, 0.994673]])],
},
Camera {
make: "Fujifilm",
model: "X-T3",
color_matrix: [Mat3([[1.29933, -0.598524, 0.075033], [-0.403354, 1.28265, 0.450876], [-0.037817, 0.112326, 0.810909]]), Mat3([[0.785701, -0.221043, -0.071522], [-0.456611, 1.15627, 0.255052], [-0.124341, 0.191243, 0.562458]])],
forward_matrix: [Mat3([[0.671025, 0.255703, 0.037568], [0.190434, 0.97306, -0.163493], [0.03182, -0.334086, 1.12737]]), Mat3([[0.676828, 0.316449, -0.028982], [0.256753, 1.03712, -0.293871], [0.04319, -0.207191, 0.989106]])],
},
Camera {
make: "Fujifilm",
model: "X-T4",
color_matrix: [Mat3([[1.27406, -0.587718, 0.074639], [-0.371617, 1.24808, 0.450058], [-0.038746, 0.113215, 0.780014]]), Mat3([[0.771728, -0.215706, -0.070731], [-0.439625, 1.14183, 0.253491], [-0.125229, 0.192512, 0.542394]])],
forward_matrix: [Mat3([[0.668116, 0.258486, 0.037694], [0.175163, 0.992049, -0.167213], [0.040145, -0.355105, 1.14006]]), Mat3([[0.675162, 0.317733, -0.028599], [0.248581, 1.05006, -0.298641], [0.04702, -0.219918, 0.998002]])],
},
Camera {
make: "Hasselblad",
model: "L1D-20c",
color_matrix: [Mat3([[0.697908, -0.225515, 0.018865], [-0.403519, 1.29608, 0.413637], [-0.048508, 0.167167, 0.74727]]), Mat3([[0.466971, -0.076456, -0.040872], [-0.471572, 1.21777, 0.21163], [-0.133006, 0.246905, 0.516655]])],
forward_matrix: [Mat3([[0.754419, 0.110524, 0.099352], [0.21061, 0.943846, -0.154456], [0.016488, -0.523808, 1.33242]]), Mat3([[0.780313, 0.168783, 0.0152], [0.295267, 0.942833, -0.2381], [0.041, -0.301041, 1.08515]])],
},
Camera {
make: "Hasselblad",
model: "L2D-20c",
color_matrix: [Mat3([[0.787283, -0.249155, 0.044752], [-0.415734, 1.29586, 0.451949], [-0.060535, 0.144901, 0.709626]]), Mat3([[0.506449, -0.071884, -0.038243], [-0.435817, 1.14056, 0.251334], [-0.129637, 0.209508, 0.474062]])],
forward_matrix: [Mat3([[0.776836, 0.120396, 0.067064], [0.218196, 0.951802, -0.169998], [0.071832, -0.4598, 1.21307]]), Mat3([[0.807201, 0.167509, -0.010414], [0.291699, 1.01528, -0.306978], [0.062694, -0.296911, 1.05932]])],
},
Camera {
make: "Nikon",
model: "D200",
color_matrix: [Mat3([[1.37569, -0.560621, 0.01677], [-0.606594, 1.52996, 0.38327], [-0.024329, 0.069398, 0.99135]]), Mat3([[0.898945, -0.249527, -0.086342], [-0.568281, 1.29839, 0.222009], [-0.098369, 0.139456, 0.687094]])],
forward_matrix: [Mat3([[0.708545, 0.18748, 0.068271], [0.274935, 0.818921, -0.093855], [0.012224, -0.169279, 0.98216]]), Mat3([[0.685079, 0.274473, 0.004744], [0.294469, 0.898619, -0.193087], [0.026107, -0.10149, 0.900488]])],
},
Camera {
make: "Nikon",
model: "D3300",
color_matrix: [Mat3([[1.01047, -0.365965, 0.032301], [-0.473579, 1.3513, 0.474736], [-0.075188, 0.164522, 1.02012]]), Mat3([[0.644589, -0.136221, -0.056505], [-0.47509, 1.16499, 0.26318], [-0.15352, 0.237111, 0.681108]])],
forward_matrix: [Mat3([[0.738138, 0.162349, 0.063809], [0.234836, 0.925124, -0.15996], [0.059989, -0.351996, 1.11711]]), Mat3([[0.742449, 0.23561, -0.013763], [0.290358, 1.00587, -0.29623], [0.04507, -0.217376, 0.99741]])],
},
Camera {
make: "Nikon",
model: "D5100",
color_matrix: [Mat3([[1.34772, -0.48474, 0.012024], [-0.466856, 1.29899, 0.44042], [-0.076248, 0.161652, 0.981853]]), Mat3([[0.929942, -0.213704, -0.081538], [-0.481178, 1.17186, 0.262186], [-0.14942, 0.233631, 0.699016]])],
forward_matrix: [Mat3([[0.732398, 0.159354, 0.072543], [0.240376, 0.912366, -0.152741], [0.06111, -0.349037, 1.11303]]), Mat3([[0.727572, 0.254229, -0.017505], [0.287762, 1.00467, -0.292432], [0.040205, -0.206009, 0.990908]])],
},
Camera {
make: "Nikon",
model: "D5300",
color_matrix: [Mat3([[0.943297, -0.302977, 0.012055], [-0.489398, 1.36679, 0.480036], [-0.079912, 0.172098, 1.02217]]), Mat3([[0.613661, -0.114638, -0.052652], [-0.475392, 1.16153, 0.266621], [-0.147109, 0.232899, 0.680179]])],
forward_matrix: [Mat3([[0.75961, 0.127618, 0.077067], [0.24786, 0.911605, -0.159465], [0.062139, -0.362458, 1.12542]]), Mat3([[0.765013, 0.208344, -0.009061], [0.301196, 0.998195, -0.299391], [0.042143, -0.217084, 1.00005]])],
},
Camera {
make: "Nikon",
model: "D7000",
color_matrix: [Mat3([[1.0579, -0.370378, 0.024968], [-0.50908, 1.38998, 0.475636], [-0.075079, 0.158561, 1.00866]]), Mat3([[0.675661, -0.143226, -0.053077], [-0.488456, 1.16985, 0.270384], [-0.137868, 0.21334, 0.681191]])],
forward_matrix: [Mat3([[0.744976, 0.150244, 0.069076], [0.251735, 0.900835, -0.15257], [0.058406, -0.336086, 1.10278]]), Mat3([[0.755556, 0.237153, -0.028413], [0.304411, 0.998446, -0.302857], [0.039148, -0.192875, 0.978832]])],
},
Camera {
make: "Nikon",
model: "D700",
color_matrix: [Mat3([[1.25876, -0.491229, 0.017162], [-0.619342, 1.54791, 0.372191], [-0.050936, 0.098145, 1.01122]]), Mat3([[0.835285, -0.217528, -0.074624], [-0.583051, 1.32613, 0.209425], [-0.121368, 0.168895, 0.708639]])],
forward_matrix: [Mat3([[0.722062, 0.17127, 0.070964], [0.279306, 0.809472, -0.088779], [0.042476, -0.200528, 0.983157]]), Mat3([[0.707886, 0.252255, 0.004155], [0.304416, 0.875438, -0.179854], [0.032977, -0.118624, 0.910752]])],
},
Camera {
make: "Nikon",
model: "D70",
color_matrix: [Mat3([[0.95752, -0.326533, -0.000865], [-0.654123, 1.5628, 0.437699], [-0.01723, 0.05324, 0.899754]]), Mat3([[0.621434, -0.137207, -0.067353], [-0.59081, 1.28888, 0.250406], [-0.066923, 0.098053, 0.633702]])],
forward_matrix: [Mat3([[0.74958, 0.134109, 0.080606], [0.313007, 0.787492, -0.100498], [0.005453, -0.15183, 0.971482]]), Mat3([[0.722318, 0.215673, 0.026305], [0.326263, 0.879351, -0.205614], [0.017282, -0.077471, 0.885294]])],
},
Camera {
make: "Nikon",
model: "D800E",
color_matrix: [Mat3([[0.981723, -0.320813, 0.025417], [-0.500428, 1.37669, 0.486261], [-0.07946, 0.169419, 1.04548]]), Mat3([[0.629114, -0.117451, -0.046422], [-0.483155, 1.16337, 0.27171], [-0.147278, 0.230496, 0.696276]])],
forward_matrix: [Mat3([[0.760723, 0.133423, 0.07015], [0.25382, 0.906695, -0.160515], [0.059962, -0.34837, 1.11351]]), Mat3([[0.780492, 0.212338, -0.028534], [0.311949, 0.998057, -0.310006], [0.041771, -0.208304, 0.991637]])],
},
Camera {
make: "Nikon",
model: "D810",
color_matrix: [Mat3([[0.983274, -0.318832, 0.030267], [-0.51358, 1.39278, 0.481653], [-0.076666, 0.176366, 1.07197]]), Mat3([[0.625688, -0.110113, -0.045912], [-0.497828, 1.18171, 0.267671], [-0.149244, 0.239807, 0.71615]])],
forward_matrix: [Mat3([[0.763991, 0.129679, 0.070625], [0.262526, 0.897486, -0.160012], [0.045849, -0.35404, 1.13329]]), Mat3([[0.791151, 0.196615, -0.023471], [0.322368, 0.977225, -0.299593], [0.038389, -0.209045, 0.995761]])],
},
Camera {
make: "Nikon",
model: "D850",
color_matrix: [Mat3([[1.17359, -0.461087, -0.011934], [-0.607581, 1.52285, 0.4063], [-0.096127, 0.196063, 1.00005]]), Mat3([[0.773836, -0.196466, -0.097677], [-0.609077, 1.35526, 0.205388], [-0.190117, 0.290252, 0.676129]])],
forward_matrix: [Mat3([[0.713588, 0.158895, 0.091813], [0.273041, 0.838318, -0.111359], [0.055293, -0.367196, 1.13701]]), Mat3([[0.669707, 0.227388, 0.0672], [0.291703, 0.871502, -0.163205], [0.042827, -0.229884, 1.01216]])],
},
Camera {
make: "Nikon",
model: "Z f",
color_matrix: [Mat3([[1.16951, -0.415422, 0.033906], [-0.516721, 1.41065, 0.441119], [-0.080466, 0.180954, 1.07672]]), Mat3([[0.749962, -0.151755, -0.063228], [-0.504317, 1.20913, 0.248149], [-0.162888, 0.256933, 0.729432]])],
forward_matrix: [Mat3([[0.744024, 0.151119, 0.069153], [0.253649, 0.888649, -0.142297], [0.049891, -0.351314, 1.12653]]), Mat3([[0.756117, 0.216534, -0.008355], [0.304737, 0.961046, -0.265784], [0.041615, -0.212408, 0.995897]])],
},
Camera {
make: "Panasonic",
model: "DC-GX9",
color_matrix: [Mat3([[0.820911, -0.254983, 0.035979], [-0.372553, 1.25333, 0.438192], [-0.054294, 0.146795, 0.747319]]), Mat3([[0.534195, -0.07728, -0.038219], [-0.400358, 1.10647, 0.251688], [-0.12982, 0.216241, 0.50214]])],
forward_matrix: [Mat3([[0.772645, 0.116685, 0.074966], [0.196365, 0.972505, -0.16887], [0.06062, -0.463162, 1.22765]]), Mat3([[0.805529, 0.178696, -0.01993], [0.275218, 1.04725, -0.322465], [0.060973, -0.29745, 1.06158]])],
},
Camera {
make: "Sony",
model: "DSC-RX100M4",
color_matrix: [Mat3([[0.783848, -0.284096, 0.031033], [-0.411895, 1.30629, 0.410807], [-0.047532, 0.165091, 0.725564]]), Mat3([[0.507376, -0.094837, -0.042497], [-0.488786, 1.23246, 0.213166], [-0.137082, 0.247475, 0.503834]])],
forward_matrix: [Mat3([[0.733625, 0.143965, 0.086705], [0.210364, 0.945779, -0.156144], [0.012886, -0.528963, 1.34118]]), Mat3([[0.767183, 0.193058, 0.004054], [0.296288, 0.94455, -0.240838], [0.043397, -0.304922, 1.08663]])],
},
Camera {
make: "Sony",
model: "ILCE-6400",
color_matrix: [Mat3([[0.686447, -0.204525, 0.026311], [-0.43013, 1.31387, 0.445779], [-0.061279, 0.132669, 0.791917]]), Mat3([[0.44557, -0.064452, -0.027734], [-0.437513, 1.13079, 0.261784], [-0.122323, 0.190304, 0.552427]])],
forward_matrix: [Mat3([[0.782555, 0.106676, 0.075064], [0.224677, 0.924591, -0.149268], [0.078407, -0.377222, 1.12392]]), Mat3([[0.821757, 0.17479, -0.032251], [0.30346, 1.00862, -0.31208], [0.052261, -0.224433, 0.997277]])],
},
Camera {
make: "Sony",
model: "ILCE-7CM2",
color_matrix: [Mat3([[0.758033, -0.221503, 0.036455], [-0.492549, 1.36195, 0.503357], [-0.065665, 0.148645, 0.906293]]), Mat3([[0.482991, -0.070139, -0.027429], [-0.478804, 1.14702, 0.282926], [-0.119226, 0.194381, 0.616628]])],
forward_matrix: [Mat3([[0.788283, 0.10759, 0.068422], [0.261923, 0.909809, -0.171732], [0.053026, -0.362705, 1.13478]]), Mat3([[0.83437, 0.173346, -0.043421], [0.336938, 0.996333, -0.333271], [0.037073, -0.203664, 0.991696]])],
},
Camera {
make: "Sony",
model: "ILCE-7M3",
color_matrix: [Mat3([[0.759047, -0.22787, 0.042531], [-0.502563, 1.37732, 0.491076], [-0.066877, 0.150947, 0.900094]]), Mat3([[0.482867, -0.070899, -0.024574], [-0.489531, 1.16667, 0.274244], [-0.123988, 0.201218, 0.610148]])],
forward_matrix: [Mat3([[0.787201, 0.113043, 0.064052], [0.265374, 0.902211, -0.167585], [0.052744, -0.364541, 1.1369]]), Mat3([[0.840001, 0.174345, -0.05005], [0.340881, 0.981246, -0.322127], [0.039232, -0.209817, 0.995689]])],
},
Camera {
make: "Sony",
model: "ILCE-7M4",
color_matrix: [Mat3([[0.763892, -0.227658, 0.033049], [-0.512687, 1.38412, 0.503225], [-0.063774, 0.141899, 0.91715]]), Mat3([[0.485445, -0.074907, -0.026973], [-0.491937, 1.15934, 0.283071], [-0.115422, 0.185993, 0.622862]])],
forward_matrix: [Mat3([[0.784064, 0.112194, 0.068037], [0.2698, 0.896428, -0.166229], [0.050008, -0.338826, 1.11392]]), Mat3([[0.830062, 0.181159, -0.046925], [0.341401, 0.985843, -0.327244], [0.034749, -0.188826, 0.979182]])],
},
Camera {
make: "Sony",
model: "ILCE-7RM2",
color_matrix: [Mat3([[0.775308, -0.25358, 0.026979], [-0.524545, 1.40154, 0.490876], [-0.060608, 0.132591, 0.920838]]), Mat3([[0.492753, -0.089554, -0.036484], [-0.505928, 1.18284, 0.273705], [-0.116121, 0.182519, 0.626106]])],
forward_matrix: [Mat3([[0.76369, 0.133034, 0.067571], [0.267238, 0.888036, -0.155274], [0.048003, -0.313698, 1.0908]]), Mat3([[0.787687, 0.201628, -0.025019], [0.326839, 0.971594, -0.298433], [0.034173, -0.177854, 0.968786]])],
},
Camera {
make: "Sony",
model: "ILCE-7RM3",
color_matrix: [Mat3([[0.742519, -0.22996, 0.032883], [-0.500197, 1.37662, 0.485744], [-0.063378, 0.142467, 0.898759]]), Mat3([[0.473388, -0.075142, -0.031025], [-0.487453, 1.16732, 0.271829], [-0.119917, 0.192619, 0.610537]])],
forward_matrix: [Mat3([[0.776802, 0.119664, 0.06783], [0.260471, 0.900308, -0.160779], [0.051869, -0.347156, 1.12039]]), Mat3([[0.813753, 0.181916, -0.031373], [0.32902, 0.979656, -0.308676], [0.037764, -0.198487, 0.985828]])],
},
Camera {
make: "Sony",
model: "ILCE-7RM4",
color_matrix: [Mat3([[0.732028, -0.220808, 0.020118], [-0.501296, 1.37502, 0.49365], [-0.061615, 0.13838, 0.885461]]), Mat3([[0.472151, -0.07908, -0.031322], [-0.483014, 1.15528, 0.279019], [-0.111192, 0.18072, 0.604588]])],
forward_matrix: [Mat3([[0.775623, 0.114594, 0.074079], [0.260863, 0.901608, -0.162471], [0.050735, -0.342927, 1.1173]]), Mat3([[0.806038, 0.193021, -0.034763], [0.32686, 0.992356, -0.319216], [0.033831, -0.189102, 0.980375]])],
},
Camera {
make: "Sony",
model: "ILCE-7SM2",
color_matrix: [Mat3([[0.615775, -0.137144, 0.06099], [-0.408209, 1.28402, 0.462018], [-0.084205, 0.196387, 0.85663]]), Mat3([[0.395982, -0.016098, -0.005517], [-0.420931, 1.11058, 0.265879], [-0.162165, 0.267589, 0.576945]])],
forward_matrix: [Mat3([[0.854583, 0.041834, 0.067878], [0.234125, 0.948414, -0.182539], [0.097857, -0.530195, 1.25744]]), Mat3([[0.948357, 0.092307, -0.076369], [0.33668, 1.02973, -0.366408], [0.073834, -0.330955, 1.08223]])],
},
Camera {
make: "Sony",
model: "ILCE-7SM3",
color_matrix: [Mat3([[0.749715, -0.228299, 0.030188], [-0.512062, 1.38442, 0.500457], [-0.072776, 0.169469, 0.933545]]), Mat3([[0.476058, -0.07383, -0.031403], [-0.495874, 1.16904, 0.277606], [-0.134473, 0.222664, 0.625349]])],
forward_matrix: [Mat3([[0.779691, 0.113899, 0.070706], [0.268632, 0.905927, -0.174559], [0.04639, -0.390929, 1.16964]]), Mat3([[0.816745, 0.179015, -0.031464], [0.334824, 0.988848, -0.323672], [0.037938, -0.229363, 1.01653]])],
},
Camera {
make: "Sony",
model: "ILCE-9",
color_matrix: [Mat3([[0.747492, -0.216385, 0.049086], [-0.499768, 1.37142, 0.499037], [-0.071324, 0.164705, 0.951496]]), Mat3([[0.473639, -0.063703, -0.016557], [-0.488372, 1.1591, 0.280181], [-0.129154, 0.213406, 0.651]])],
forward_matrix: [Mat3([[0.796409, 0.106147, 0.061739], [0.26822, 0.906732, -0.174952], [0.050574, -0.378333, 1.15286]]), Mat3([[0.868551, 0.167059, -0.071314], [0.354491, 0.985971, -0.340462], [0.037779, -0.211149, 0.998474]])],
},
];
/// `(make, model as sold, table model)`: other names of a camera in [`CAMERAS`], or of a camera with the same sensor.
#[rustfmt::skip]
pub(super) const ALIASES: &[(&str, &str, &str)] = &[
("Canon", "EOS 400D", "Digital Rebel XTi"), // same model, alternative name
("Canon", "Kiss Digital X", "Digital Rebel XTi"), // same model, alternative name
("Canon", "EOS Rebel SL1", "EOS 100D"), // same model, alternative name
("Canon", "Kiss X7", "EOS 100D"), // same model, alternative name
("Canon", "EOS Rebel SL2", "EOS 200D"), // same model, alternative name
("Canon", "Kiss X9", "EOS 200D"), // same model, alternative name
("Canon", "EOS 250D", "EOS 200D II"), // same model, alternative name
("Canon", "EOS Rebel SL3", "EOS 200D II"), // same model, alternative name
("Canon", "Kiss X10", "EOS 200D II"), // same model, alternative name
("Canon", "EOS Rebel T3i", "EOS 600D"), // same model, alternative name
("Canon", "Kiss X5", "EOS 600D"), // same model, alternative name
("Canon", "EOS R5 Mark II", "EOS R5m2"), // same model, alternative name
("Canon", "EOS 5DS R", "EOS 5DS"), // different model, same sensor
("Canon", "EOS 6D Mark II", "EOS RP"), // different model, same sensor
("Canon", "EOS R5 C", "EOS R5"), // different model, same sensor
("Canon", "EOS R50", "EOS R10"), // different model, same sensor
("Canon", "EOS R8", "EOS R6m2"), // different model, same sensor
("Nikon", "D800", "D800E"), // different model, same sensor
("Nikon", "Z 6_2", "Z f"), // different model, same sensor
("Sony", "ILCE-7RM5", "ILCE-7RM4"), // different model, same sensor
("Sony", "ILCE-7RM4A", "ILCE-7RM4"), // updated model, same sensor
("Sony", "ILCE-7RM3A", "ILCE-7RM3"), // updated model, same sensor
("Fujifilm", "GFX100S", "GFX 100"), // different model, same sensor
];

177
crates/raw/src/vendor/arw.rs vendored
View File

@ -32,6 +32,10 @@
//! comes from the data. The crop and white balance are NOT addressed here: the file has no maker-note `FullImageSize`
//! and no plain WB tag (they live in its `SR2SubIFD`), so it opens uncropped with unit multipliers.
//!
//! Packed 12-bit ARW (DSLR-A900; Compression 32767 but BitsPerSample 12 and a strip of exactly width × height × 1.5
//! bytes): plain 12-bit samples, two per three bytes, least-significant-bit first ([`unpack_row12`]); linear, black 128
//! (512 on the 14-bit scale), CFA from the file's own pattern tag, crop centred on the frame (see [`default_crop`]).
//!
//! Also: uncompressed 16-bit ARW, and lossless-compressed ARW (Compression 7, ILCE-7M4 and later): LJ92 tiles whose
//! frames hold one 2×2 CFA cell per four-component sample ([`read_quad_tiles`]). Other lossless-JPEG layouts go
//! through the generic TIFF path.
@ -203,6 +207,25 @@ fn tag2010_wb(model: &str, block: &[u8], order: lightcraft_tiff::ByteOrder) -> O
((0.2..=8.0).contains(&r) && (0.2..=8.0).contains(&b)).then_some([r, 1.0, b])
}
/// Whether a 12-bit, single-strip, Sony-compressed (32767) image is stored as plain packed 12-bit samples: the strip
/// is exactly 1.5 bytes per pixel (width even), so every row is `1.5 · width` bytes with no padding. On the one
/// DSLR-A900 sample this is the whole strip (36 917 760 = 6080 × 4048 × 1.5) and it ends at the end of the file.
fn is_packed12(info: &ImageInfo, strip_count: usize, strip_len: u64) -> bool {
let (w, h) = (info.width as u64, info.height as u64);
info.bits() == 12 && strip_count == 1 && w % 2 == 0 && w > 0 && h > 0 && strip_len == w * h * 3 / 2
}
/// Unpack plain 12-bit samples stored least-significant-bit first: two pixels per three bytes, the first being
/// `b0 | (b1 & 0xf) << 8` and the second `b1 >> 4 | b2 << 4` (measured: same-colour neighbours differ by ~16 codes
/// in this order against ~200 in the most-significant-first order). `row` is `1.5 · out.len()` bytes.
pub(crate) fn unpack_row12(row: &[u8], out: &mut [u16]) {
for (i, b) in row.windows(3).step_by(3).enumerate() {
let Some(px) = out.get_mut(2 * i..2 * i + 2) else { break };
px[0] = u16::from(b[0]) | (u16::from(b[1] & 0xf) << 8);
px[1] = u16::from(b[1] >> 4) | (u16::from(b[2]) << 4);
}
}
/// White-balance gains `[R/G, 1, B/G]` from `WB_RGGBLevels` (R, G, G, B); `None` unless there are four levels
/// with a positive green and plausible ratios.
fn rggb_gains(levels: &[f64]) -> Option<[f32; 3]> {
@ -462,18 +485,30 @@ fn word16_order(strip: &[u8], bits: u32, file: lightcraft_tiff::ByteOrder) -> li
if n > 0 && over_other == 0 && over_file * 100 > n { other } else { file }
}
/// The image area for files without Sony's crop tags (`0x74c7/0x74c8`, written since about 2017): the DNG-style
/// default crop when the raw IFD has one, else the image size the camera records (see [`recorded_size_crop`]).
fn default_crop(raw: &Ifd, mn: Option<&makernote::MakerNote>, exif_size: Option<(u64, u64)>, w: usize, h: usize) -> Rect {
let full = Rect::new(0, 0, w, h);
/// The raw IFD's DNG-style `DefaultCropOrigin` / `DefaultCropSize`, clipped to the `w × h` frame.
fn dng_default_crop(raw: &Ifd, w: usize, h: usize) -> Option<Rect> {
if let (Some([x, y]), Some([cw, ch])) = (raw.u64s(t::DEFAULT_CROP_ORIGIN).as_deref(), raw.u64s(t::DEFAULT_CROP_SIZE).as_deref())
&& *cw > 0
&& *ch > 0
{
return Rect::new(*x as usize, *y as usize, *cw as usize, *ch as usize).clipped(w, h);
return Some(Rect::new(*x as usize, *y as usize, *cw as usize, *ch as usize).clipped(w, h));
}
None
}
/// The image area for files without Sony's crop tags (`0x74c7/0x74c8`, written since about 2017): the DNG-style
/// default crop when the raw IFD has one, else the image size the camera records (see [`recorded_size_crop`]).
///
/// `centred` places that recorded-size window in the middle of the frame (on even offsets, keeping the CFA phase)
/// instead of at the left: the packed 12-bit frame (DSLR-A900) has no padding at either edge, and its camera JPEG
/// is centred on the frame.
fn default_crop(raw: &Ifd, mn: Option<&makernote::MakerNote>, exif_size: Option<(u64, u64)>, w: usize, h: usize, centred: bool) -> Rect {
let full = Rect::new(0, 0, w, h);
if let Some(crop) = dng_default_crop(raw, w, h) {
return crop;
}
match mn.and_then(|m| m.ifd.u64s(MN_FULL_IMAGE_SIZE)).as_deref() {
Some([fh, fw]) => recorded_size_crop((*fw, *fh), exif_size, w, h).unwrap_or(full),
Some([fh, fw]) => recorded_size_crop((*fw, *fh), exif_size, w, h, centred).unwrap_or(full),
_ => full,
}
}
@ -486,8 +521,9 @@ fn default_crop(raw: &Ifd, mn: Option<&makernote::MakerNote>, exif_size: Option<
/// top. An in-camera aspect ratio narrower than the frame (16:9) is centred vertically, as measured by registering
/// each camera JPEG on its raw (the ILCE-7SM2's 4240 × 2384 at y 228–232 of 2848; the DSLR-A580's 4912 × 2760 at
/// y 258–262 of 3280; issue #535): its height comes from `FullImageSize` (ILCE-7SM2) or, when that still says
/// 3:2, from an Exif image size of the same width (DSLR-A580).
fn recorded_size_crop(full: (u64, u64), exif: Option<(u64, u64)>, w: usize, h: usize) -> Option<Rect> {
/// 3:2, from an Exif image size of the same width (DSLR-A580). `centred` centres the window in both directions
/// instead (see [`default_crop`]).
fn recorded_size_crop(full: (u64, u64), exif: Option<(u64, u64)>, w: usize, h: usize, centred: bool) -> Option<Rect> {
let (fw, fh) = (usize::try_from(full.0).ok()?, usize::try_from(full.1).ok()?);
// only a plausible trim of the width: never more than 64 pixels, never an enlargement
if fw == 0 || fh == 0 || fw > w || fw.saturating_add(64) < w || fh > h {
@ -497,11 +533,13 @@ fn recorded_size_crop(full: (u64, u64), exif: Option<(u64, u64)>, w: usize, h: u
Some((ew, eh)) if ew == fw && eh > 0 && eh < fh => eh,
_ => fh,
};
// even offsets keep the CFA phase
let x = if centred { ((w - fw) / 2) & !1 } else { 0 };
if height.saturating_add(64) >= h {
return Some(Rect::new(0, 0, fw, height));
return Some(Rect::new(x, if centred { ((h - height) / 2) & !1 } else { 0 }, fw, height));
}
// an in-camera aspect ratio: at least half the frame's height, centred (an even offset keeps the CFA phase)
(height * 2 >= h).then(|| Rect::new(0, ((h - height) / 2) & !1, fw, height))
// an in-camera aspect ratio: at least half the frame's height, centred
(height * 2 >= h).then(|| Rect::new(x, ((h - height) / 2) & !1, fw, height))
}
fn raw_ifd(tiff: &Tiff) -> Option<&Ifd> {
@ -528,6 +566,7 @@ pub(crate) fn decode(bytes: &[u8], mode: Mode) -> Result<RawImage> {
// archive measure exactly width x height. A200/A230/A350 files carry a strip 4-23 % larger than that (and A290/A390 one
// smaller), which is a different packing, so they are refused rather than read with the wrong layout.
let one_byte_per_sample = chunks.len() == 1 && strip_len == (w * h) as u64;
let packed12 = info.compression == 32767 && is_packed12(&info, chunks.len(), strip_len);
let (data, out_bits) = match info.compression {
7 if linear_rgb => (RawData::U16(read_ycbcr_tiles(bytes, &info, raw, mode)?), 14),
32767 if one_byte_per_sample && mode == Mode::Header => {
@ -547,6 +586,21 @@ pub(crate) fn decode(bytes: &[u8], mode: Mode) -> Result<RawImage> {
});
(RawData::U16(data), 14)
}
32767 if packed12 && mode == Mode::Header => {
chunk_bytes(bytes, &chunks[0]).ok_or_else(|| RawError::Corrupt("raw strip outside file".into()))?;
(RawData::U16(Vec::new()), 12)
}
32767 if packed12 => {
let src = chunk_bytes(bytes, &chunks[0]).ok_or_else(|| RawError::Corrupt("raw strip outside file".into()))?;
let row_bytes = w / 2 * 3;
let mut data = vec![0u16; w * h];
data.par_chunks_mut(w).enumerate().for_each(|(y, out)| {
if let Some(row) = src.get(y * row_bytes..(y + 1) * row_bytes) {
unpack_row12(row, out);
}
});
(RawData::U16(data), 12)
}
32767 => return Err(RawError::Unsupported("Sony ARW version 1 / packed compressed variant (raw strip is not one byte per pixel)".into())),
7 if is_quad_tiled(bytes, &info) => match mode {
Mode::Full => (RawData::U16(read_quad_tiles(bytes, &info)?), bits),
@ -611,7 +665,7 @@ pub(crate) fn decode(bytes: &[u8], mode: Mode) -> Result<RawImage> {
(Some([x, y]), Some([cw, ch])) if *cw > 0 && *ch > 0 => Rect::new(*x as usize, *y as usize, *cw as usize, *ch as usize).clipped(w, h),
_ => {
let exif_size = tiff.exif().and_then(|e| Some((e.u64(t::PIXEL_X_DIMENSION)?, e.u64(t::PIXEL_Y_DIMENSION)?)));
default_crop(raw, mn.as_ref(), exif_size, w, h)
default_crop(raw, mn.as_ref(), exif_size, w, h, packed12)
}
};
let mut metadata = lightcraft_meta::from_tiff(&tiff);
@ -635,7 +689,19 @@ pub(crate) fn decode(bytes: &[u8], mode: Mode) -> Result<RawImage> {
// makes a neutral surface magenta. WB edits remain relative to this as-shot RGB.
wb_multipliers: if linear_rgb { Some([1.0; 3]) } else { wb },
linearized: false,
opcodes: OpcodeLists::default(),
// The distortion table is centred on the DNG-style default crop when the file has one: on the ILCE-7RM4A
// it starts at x = 32 where Sony's crop tags (the image crop, unchanged) start at 0, and the warp was
// validated against Sony's exports with that centre. The opcode's centre is in active-area coordinates, so
// it stays on the optical centre whichever crop frames the image.
opcodes: OpcodeLists {
list3: if linear_rgb {
Vec::new()
} else {
let geometry = dng_default_crop(raw, w, h).unwrap_or(crop);
super::arw_lens::distortion(&model, raw, Rect::new(0, 0, w, h), geometry).into_iter().collect()
},
..Default::default()
},
metadata,
};
img.validate_for(mode)?;
@ -684,21 +750,24 @@ mod tests {
#[test]
fn recorded_size_crops_in_camera_aspect_ratios() {
// 3:2: anchored at the top-left, as before (ILCE-7S, DSLR-A700 whose Exif size is the whole frame)
assert_eq!(recorded_size_crop((4240, 2832), Some((4240, 2832)), 4288, 2848), Some(Rect::new(0, 0, 4240, 2832)));
assert_eq!(recorded_size_crop((4272, 2848), Some((4288, 2856)), 4288, 2856), Some(Rect::new(0, 0, 4272, 2848)));
assert_eq!(recorded_size_crop((4240, 2832), None, 4288, 2848), Some(Rect::new(0, 0, 4240, 2832)));
assert_eq!(recorded_size_crop((4240, 2832), Some((4240, 2832)), 4288, 2848, false), Some(Rect::new(0, 0, 4240, 2832)));
assert_eq!(recorded_size_crop((4272, 2848), Some((4288, 2856)), 4288, 2856, false), Some(Rect::new(0, 0, 4272, 2848)));
assert_eq!(recorded_size_crop((4240, 2832), None, 4288, 2848, false), Some(Rect::new(0, 0, 4240, 2832)));
// 16:9 from FullImageSize (ILCE-7SM2) and from the Exif size when FullImageSize says 3:2 (DSLR-A580): centred
assert_eq!(recorded_size_crop((4240, 2384), Some((4240, 2384)), 4288, 2848), Some(Rect::new(0, 232, 4240, 2384)));
assert_eq!(recorded_size_crop((4912, 3264), Some((4912, 2760)), 4928, 3280), Some(Rect::new(0, 260, 4912, 2760)));
assert_eq!(recorded_size_crop((4240, 2384), Some((4240, 2384)), 4288, 2848, false), Some(Rect::new(0, 232, 4240, 2384)));
assert_eq!(recorded_size_crop((4912, 3264), Some((4912, 2760)), 4928, 3280, false), Some(Rect::new(0, 260, 4912, 2760)));
// offsets stay even (CFA phase)
assert_eq!(recorded_size_crop((4240, 2386), None, 4288, 2848).map(|r| r.y), Some(230));
assert_eq!(recorded_size_crop((4240, 2386), None, 4288, 2848, false).map(|r| r.y), Some(230));
// not a crop of this frame: wider than it, far narrower, under half its height, empty, hostile values
assert_eq!(recorded_size_crop((4300, 2832), None, 4288, 2848), None);
assert_eq!(recorded_size_crop((4000, 2832), None, 4288, 2848), None);
assert_eq!(recorded_size_crop((4240, 1000), None, 4288, 2848), None);
assert_eq!(recorded_size_crop((0, 2832), None, 4288, 2848), None);
assert_eq!(recorded_size_crop((u64::MAX, u64::MAX), Some((u64::MAX, 1)), 4288, 2848), None);
assert_eq!(recorded_size_crop((4240, 2832), Some((4240, 0)), 4288, 2848), Some(Rect::new(0, 0, 4240, 2832)));
assert_eq!(recorded_size_crop((4300, 2832), None, 4288, 2848, false), None);
assert_eq!(recorded_size_crop((4000, 2832), None, 4288, 2848, false), None);
assert_eq!(recorded_size_crop((4240, 1000), None, 4288, 2848, false), None);
assert_eq!(recorded_size_crop((0, 2832), None, 4288, 2848, false), None);
assert_eq!(recorded_size_crop((u64::MAX, u64::MAX), Some((u64::MAX, 1)), 4288, 2848, false), None);
assert_eq!(recorded_size_crop((4240, 2832), Some((4240, 0)), 4288, 2848, false), Some(Rect::new(0, 0, 4240, 2832)));
// centred (packed 12-bit DSLR-A900: 6048 x 4032 in the middle of 6080 x 4048)
assert_eq!(recorded_size_crop((6048, 4032), Some((6048, 4032)), 6080, 4048, true), Some(Rect::new(16, 8, 6048, 4032)));
assert_eq!(recorded_size_crop((6046, 4030), None, 6080, 4048, true), Some(Rect::new(16, 8, 6046, 4030)));
}
/// A 32767-compressed ARW whose single strip is `strip` bytes for a 32 x 4 image.
@ -732,6 +801,64 @@ mod tests {
}
}
/// A 32767-compressed, 12-bit ARW of `w` x `h` whose single strip is `strip`.
fn packed_arw(w: u32, h: u32, strip: Vec<u8>) -> Vec<u8> {
use lightcraft_tiff::{IfdBuilder, ImageData, TiffWriter, Value};
let mut raw = IfdBuilder::new();
raw.set(t::MAKE, Value::Ascii("SONY".into()));
raw.set(t::MODEL, Value::Ascii("DSLR-A900".into()));
raw.set(t::IMAGE_WIDTH, Value::Long(vec![w]));
raw.set(t::IMAGE_LENGTH, Value::Long(vec![h]));
raw.set(t::BITS_PER_SAMPLE, Value::Short(vec![12]));
raw.set(t::SAMPLES_PER_PIXEL, Value::Short(vec![1]));
raw.set(t::PHOTOMETRIC, Value::Short(vec![photometric::CFA]));
raw.set(t::COMPRESSION, Value::Short(vec![32767]));
raw.set(t::CFA_REPEAT_PATTERN_DIM, Value::Short(vec![2, 2]));
raw.set(t::CFA_PATTERN_EP, Value::Byte(vec![0, 1, 1, 2]));
raw.set(TONE_CURVE, Value::Short(vec![8000, 10400, 12900, 14100]));
raw.set_image(ImageData::Strips { rows_per_strip: h, strips: vec![strip] });
TiffWriter::default().write(&[raw]).unwrap()
}
/// Pack 12-bit samples as the file does: `b0 = a & 0xff`, `b1 = a >> 8 | b << 4 & 0xf0`, `b2 = b >> 4`.
fn pack12(samples: &[u16]) -> Vec<u8> {
samples.chunks(2).flat_map(|p| [p[0] as u8, (p[0] >> 8) as u8 | ((p[1] & 0xf) << 4) as u8, (p[1] >> 4) as u8]).collect()
}
#[test]
fn packed_12_bit_strip_of_one_and_a_half_bytes_per_pixel_decodes() {
let (w, h) = (8usize, 4usize);
let samples: Vec<u16> = (0..w * h).map(|i| (i as u16 * 129 + 130) & 0xfff).collect();
let file = packed_arw(w as u32, h as u32, pack12(&samples));
let full = decode(&file, Mode::Full).unwrap();
let RawData::U16(ref got) = full.data else { panic!("integer ARW") };
assert_eq!(got, &samples);
assert_eq!((full.width, full.height, full.bits), (w, h, 12));
assert_eq!(full.cfa.as_ref().map(|c| c.pattern.clone()), Some(vec![0, 1, 1, 2]));
assert_eq!(full.black.values.first().copied().unwrap_or(0.0), 128.0);
let header = decode(&file, Mode::Header).unwrap();
assert_eq!(header.info(), full.info());
// the first three bytes 0x12 0xA3 0x45 are the samples 0x312 and 0x45A (least-significant-bit first)
let mut row = vec![0u16; 2];
unpack_row12(&[0x12, 0xa3, 0x45], &mut row);
assert_eq!(row, [0x312, 0x45a]);
}
#[test]
fn packed_12_bit_needs_the_exact_strip_size() {
let (w, h) = (8usize, 4usize);
let exact = w * h * 3 / 2;
for strip in [exact - 1, exact + 1, exact + 3 * h] {
for mode in [Mode::Header, Mode::Full] {
let err = decode(&packed_arw(w as u32, h as u32, vec![0u8; strip]), mode).unwrap_err();
assert!(matches!(err, RawError::Unsupported(ref m) if m.contains("packed compressed variant")), "{strip} {mode:?}: {err:?}");
}
}
// an odd width cannot be packed in pairs
let err = decode(&packed_arw(7, 4, vec![0u8; 7 * 4 * 3 / 2]), Mode::Full).unwrap_err();
assert!(matches!(err, RawError::Unsupported(_)), "{err:?}");
}
/// Encode one 16-value set as an ARW2 block (the paper's scheme, used here to test the decoder).
pub(crate) fn encode_block(v: &[u16; 16]) -> [u8; 16] {
let (imax, &max) = v.iter().enumerate().max_by_key(|(i, x)| (**x, usize::MAX - *i)).unwrap();

230
crates/raw/src/vendor/arw_lens.rs vendored Normal file
View File

@ -0,0 +1,230 @@
//! Sony's plain raw-IFD distortion table, expressed as a DNG warp for the shared optics pipeline.
//!
//! Tag identity/layout: https://exiftool.org/TagNames/EXIF.html (`0x7037`, signed shorts).
//! Geometry inferred independently from files and Sony Imaging Edge exports; see docs/sony-lens-corrections.md.
//! No lens database or external decoder code. The unrelated MakerNote table has a different scale.
use crate::{Opcode, Rect};
use lightcraft_tiff::{Ifd, Value};
const DISTORTION: u16 = 0x7037;
const SAMPLES: usize = 1024;
const MAX_ERROR: f64 = 0.0005;
/// Only independently validated camera models and the 16-entry layout. Unknown models/layouts
/// and implausible/non-smooth maps are left uncorrected. See docs/sony-lens-corrections.md.
/// A camera setting of Off does not remove the lens data: the user can enable it in the developer.
pub(super) fn distortion(model: &str, ifd: &Ifd, active: Rect, crop: Rect) -> Option<Opcode> {
// A matching table shape does not establish its scale or radial normalization on another body.
// Keep this exact: ILCE-7RM4 and ILCE-7RM4A are distinct models, not interchangeable aliases.
if model != "ILCE-7RM4A" {
return None;
}
let Value::SShort(values) = &ifd.get(DISTORTION)?.value else { return None };
let [16, table @ ..] = values.as_slice() else { return None };
let table: &[i16; 16] = table.try_into().ok()?;
if table.iter().all(|v| *v == 0) || table.iter().any(|v| !(-4096..=4096).contains(v)) {
return None;
}
let crop = crop.clipped(active.width, active.height);
if active.width < 2 || active.height < 2 || crop.width < 2 || crop.height < 2 {
return None;
}
// Only native 3:2 framing has been validated. An in-camera aspect crop may retain a table
// normalized to the full sensor frame; do not silently reinterpret it as the smaller crop.
if (crop.width as f64 / crop.height as f64 - 1.5).abs() > 0.005 {
return None;
}
// Interpolate fractional radial scale at evenly spaced radii from centre to cropped corner.
let scale = |r: f64| {
let at = r.clamp(0.0, 1.0) * 15.0;
let i = (at.floor() as usize).min(14);
let a = f64::from(table[i]);
let b = f64::from(table[i + 1]);
1.0 + (a + (b - a) * (at - i as f64)) / 16384.0
};
let (cw, ch) = (crop.width as f64 - 1.0, crop.height as f64 - 1.0);
let diagonal = cw.hypot(ch);
// Zoom until the entire rectangular output boundary is inside the source. For a piecewise-linear
// scale its maximum on that boundary occurs at its nearest radius or at one of the remaining knots.
let nearest = cw.min(ch) / diagonal;
let max_scale = (0..16).map(|i| i as f64 / 15.0).filter(|&r| r >= nearest).map(scale).fold(scale(nearest), f64::max);
let zoom = 1.0 / max_scale;
let basis = |r: f64| {
let r2 = r * r;
[r, r * r2, r * r2 * r2, r * r2 * r2 * r2]
};
let mut matrix = [[0.0; 4]; 4];
let mut rhs = [0.0; 4];
let mut previous = -1.0;
for j in 0..=SAMPLES {
let r = j as f64 / SAMPLES as f64;
let source = r * scale(r) * zoom;
if source <= previous {
return None;
}
previous = source;
let terms = basis(r);
for (i, &a) in terms.iter().enumerate() {
rhs[i] += a * source;
for (k, &b) in terms.iter().enumerate() {
matrix[i][k] += a * b;
}
}
}
let mut k = solve4(matrix, rhs)?;
// Keep the polynomial within a fraction of a preview pixel of the file's table, and forbid folds.
for j in 0..=SAMPLES {
let r = j as f64 / SAMPLES as f64;
let fit = basis(r).iter().zip(k).map(|(a, b)| a * b).sum::<f64>();
let r2 = r * r;
let derivative = k[0] + r2 * (3.0 * k[1] + r2 * (5.0 * k[2] + r2 * 7.0 * k[3]));
if (fit - r * scale(r) * zoom).abs() > MAX_ERROR || derivative <= 0.0 {
return None;
}
}
// DNG uses the farthest *active-area* corner, not the cropped diagonal. Preserve the same mapping
// after the engine translates the opcode into default-cropped and EXIF-oriented coordinates.
let (ax, ay) = (active.width as f64 - 1.0, active.height as f64 - 1.0);
let (cx, cy) = (crop.x as f64 + cw / 2.0, crop.y as f64 + ch / 2.0);
let farthest = cx.max(ax - cx).hypot(cy.max(ay - cy));
let ratio2 = (2.0 * farthest / diagonal).powi(2);
for (i, coefficient) in k.iter_mut().enumerate() {
*coefficient *= ratio2.powi(i as i32);
}
Some(Opcode::WarpRectilinear { planes: vec![[k[0], k[1], k[2], k[3], 0.0, 0.0]], center: [cx / ax, cy / ay] })
}
/// Small fixed-size least-squares system; partial pivoting, no input-sized allocations.
fn solve4(mut a: [[f64; 4]; 4], mut b: [f64; 4]) -> Option<[f64; 4]> {
for col in 0..4 {
let pivot = (col..4).max_by(|&i, &j| a[i][col].abs().total_cmp(&a[j][col].abs()))?;
if !a[pivot][col].is_finite() || a[pivot][col].abs() < 1e-12 {
return None;
}
a.swap(col, pivot);
b.swap(col, pivot);
for row in col + 1..4 {
let factor = a[row][col] / a[col][col];
for j in col..4 {
a[row][j] -= factor * a[col][j];
}
b[row] -= factor * b[col];
}
}
let mut x = [0.0; 4];
for i in (0..4).rev() {
x[i] = (b[i] - (i + 1..4).map(|j| a[i][j] * x[j]).sum::<f64>()) / a[i][i];
}
x.iter().all(|v| v.is_finite()).then_some(x)
}
#[cfg(test)]
mod tests {
use super::*;
use lightcraft_tiff::Entry;
fn ifd(value: Value) -> Ifd {
Ifd { entries: vec![Entry { tag: DISTORTION, value, offset: 0 }], ..Default::default() }
}
fn table(values: &[i16]) -> Ifd {
ifd(Value::SShort(std::iter::once(16).chain(values.iter().copied()).collect()))
}
const TELE: [i16; 16] = [-2, 1, 7, 17, 31, 48, 68, 93, 120, 152, 187, 225, 265, 309, 356, 403];
const WIDE: [i16; 16] = [14, 0, -27, -59, -106, -158, -223, -291, -367, -440, -519, -593, -667, -731, -792, -839];
const AREA: Rect = Rect { x: 0, y: 0, width: 9504, height: 6336 };
#[test]
fn matches_independent_sony_reference_geometry() {
// On/off feature correspondences from Imaging Edge 4.1 at 600 mm, independently fitted.
let Opcode::WarpRectilinear { planes, center } = distortion("ILCE-7RM4A", &table(&TELE), AREA, AREA).unwrap() else { panic!("warp") };
assert_eq!(center, [0.5, 0.5]);
let reference = [0.9760159044, 0.0254891644, -0.0005984764, -0.0010120638];
for i in 0..=100 {
let r = i as f64 / 100.0;
let evaluate = |k: &[f64]| r * (k[0] + r * r * (k[1] + r * r * (k[2] + r * r * k[3])));
assert!((evaluate(&planes[0]) - evaluate(&reference)).abs() < 0.00015);
}
}
#[test]
fn barrel_framing_and_offset_crop_keep_the_same_geometry() {
let crop = Rect::new(32, 20, 9504, 6336);
let active = Rect::new(0, 0, 9600, 6376);
let Opcode::WarpRectilinear { planes: a, .. } = distortion("ILCE-7RM4A", &table(&WIDE), AREA, AREA).unwrap() else { panic!("warp") };
let Opcode::WarpRectilinear { planes: b, center } = distortion("ILCE-7RM4A", &table(&WIDE), active, crop).unwrap() else { panic!("warp") };
assert!(a[0][0] > 1.02 && a[0][0] < 1.03, "barrel correction expands the middle");
assert!((center[0] * 9599.0 - 4783.5).abs() < 1e-10);
assert!((center[1] * 6375.0 - 3187.5).abs() < 1e-10);
let ratio = (4815.5f64.hypot(3187.5)) / (4751.5f64.hypot(3167.5));
for i in 0..4 {
assert!((a[0][i] - b[0][i] / ratio.powi(2 * i as i32)).abs() < 1e-10);
}
}
#[test]
fn refuses_missing_unknown_malformed_or_unusable_tables() {
for bad in [
Ifd::default(),
table(&[]),
table(&TELE[..15]),
table(&[0; 16]),
table(&[i16::MIN; 16]),
table(&[0, 4096, -4096, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]),
ifd(Value::SShort(vec![15; 17])),
ifd(Value::Short(vec![16; 17])),
ifd(Value::Double(vec![f64::NAN; 17])),
] {
assert!(distortion("ILCE-7RM4A", &bad, AREA, AREA).is_none());
}
assert!(distortion("ILCE-7RM4A", &table(&TELE), Rect::new(0, 0, 1, 1), AREA).is_none());
assert!(distortion("ILCE-7RM4A", &table(&TELE), AREA, Rect::new(0, 0, 9504, 5346)).is_none());
assert!(distortion("ILCE-7RM4A", &table(&TELE), AREA, Rect::new(99999, 0, 10, 10)).is_none());
}
#[test]
fn refuses_unvalidated_models_even_with_a_valid_table() {
let valid = table(&TELE);
assert!(distortion("ILCE-7RM4A", &valid, AREA, AREA).is_some());
for model in ["", "ILCE-7M3", "ILCE-7M4", "ILCE-7RM4", "ILCE-7RM5", "ILCE-9M2", "DSC-RX100M3", "ILCE-7RM4A unknown"] {
assert!(distortion(model, &valid, AREA, AREA).is_none(), "{model}");
}
}
#[test]
fn header_and_full_decode_preserve_correction_through_dng() {
use lightcraft_tiff::{ByteOrder, IfdBuilder, ImageData, TiffWriter, tags as t};
for (model, expected) in [("ILCE-7RM4A", 1), ("ILCE-7RM4", 0), ("ILCE-7M3", 0), ("", 0)] {
for order in [ByteOrder::Little, ByteOrder::Big] {
let mut raw = IfdBuilder::new();
raw.set(t::MAKE, Value::Ascii("SONY".into()));
raw.set(t::MODEL, Value::Ascii(model.into()));
raw.set(t::IMAGE_WIDTH, Value::Long(vec![32]));
raw.set(t::IMAGE_LENGTH, Value::Long(vec![24]));
raw.set(t::BITS_PER_SAMPLE, Value::Short(vec![16]));
raw.set(t::SAMPLES_PER_PIXEL, Value::Short(vec![1]));
raw.set(t::PHOTOMETRIC, Value::Short(vec![t::photometric::CFA]));
raw.set(t::COMPRESSION, Value::Short(vec![1]));
raw.set(t::WHITE_LEVEL, Value::Long(vec![16383]));
raw.set(0x74c7, Value::Long(vec![0, 2]));
raw.set(t::DEFAULT_CROP_ORIGIN, Value::Long(vec![2, 2]));
raw.set(t::DEFAULT_CROP_SIZE, Value::Long(vec![30, 20]));
raw.set(0x74c8, Value::Long(vec![30, 20]));
raw.set(DISTORTION, Value::SShort(std::iter::once(16).chain(TELE).collect()));
// Zero pixels have the same byte representation in both orders.
raw.set_image(ImageData::Strips { rows_per_strip: 24, strips: vec![vec![0; 32 * 24 * 2]] });
let file = TiffWriter { order, ..Default::default() }.write(&[raw]).unwrap();
let header = super::super::arw::decode(&file, crate::Mode::Header).unwrap();
let full = super::super::arw::decode(&file, crate::Mode::Full).unwrap();
assert_eq!(header.info(), full.info());
assert_eq!(full.opcodes.list3.len(), expected, "{model} {order:?}");
// the image keeps Sony's crop tags; the warp is centred on the DNG default crop (x 2..32 of 32)
assert_eq!(full.crop, Rect::new(0, 2, 30, 20));
if let Some(Opcode::WarpRectilinear { center, .. }) = full.opcodes.list3.first() {
assert!((center[0] - 16.5 / 31.0).abs() < 1e-12 && (center[1] - 11.5 / 23.0).abs() < 1e-12, "{center:?}");
}
let dng = crate::write_dng(&full, &Default::default()).unwrap();
assert_eq!(crate::decode(&dng).unwrap().opcodes.list3, full.opcodes.list3);
}
}
}
}

1
crates/raw/src/vendor/mod.rs vendored
View File

@ -1,6 +1,7 @@
//! Vendor raw formats (TIFF-based) and helpers shared by them.
pub mod arw;
mod arw_lens;
pub mod cr2;
pub mod cr3;
pub mod crx;

62
crates/raw/tests/corpus.rs
View File

@ -516,11 +516,12 @@ fn corpus_sony_sr2_white_balance() {
let dir = corpus_root().join("raw");
// (file, R and B levels)
let cases = [
("arw-sony-a500.arw", [2212, 1416]), // 27152-byte SR2SubIFD layout (2.04 / 1.35)
("arw-sony-a33.arw", [2344, 1508]), // 29000-byte layout (2.29 / 1.48)
("arw-sony-a700.arw", [2128, 1564]), // 62112-byte layout (too few neutral pixels)
("arw-sony-a3500-5600k.arw", [2932, 1576]), // 5600 K (2.76 / 1.53; Tag2010 1.89 / 3.08)
("arw-sony-a7s-shade.arw", [2932, 1372]), // Shade (2.90 / 1.32; Tag2010 2.63 / 1.52)
("arw-sony-a500.arw", [2212, 1416]), // 27152-byte SR2SubIFD layout (2.04 / 1.35)
("arw-sony-a33.arw", [2344, 1508]), // 29000-byte layout (2.29 / 1.48)
("arw-sony-a700.arw", [2128, 1564]), // 62112-byte layout (too few neutral pixels)
("arw-sony-a3500-5600k.arw", [2932, 1576]), // 5600 K (2.76 / 1.53; Tag2010 1.89 / 3.08)
("arw-sony-a7s-shade.arw", [2932, 1372]), // Shade (2.90 / 1.32; Tag2010 2.63 / 1.52)
("arw-sony-a900-packed12.arw", [2688, 1516]), // packed 12-bit (the maker note's plain 0x0020 block agrees)
];
let mut seen = 0;
for (name, [r, b]) in cases {
@ -537,6 +538,22 @@ fn corpus_sony_sr2_white_balance() {
eprintln!("Sony SR2SubIFD white balance checked on {seen} files");
}
/// The DSLR-A900's packed 12-bit strip decodes (issue #535, item 6): 12 bits on a black of 128, and the camera
/// JPEG's `FullImageSize` window centred in the frame, which has no padding at either edge.
#[test]
fn corpus_sony_a900_packed12() {
let name = "arw-sony-a900-packed12.arw";
let Ok(bytes) = std::fs::read(corpus_root().join("raw").join(name)) else {
eprintln!("skip: {name} absent");
return;
};
let img = decode(&bytes).unwrap_or_else(|e| panic!("{name}: {e}"));
assert_eq!((img.width, img.height, img.bits), (6080, 4048, 12));
assert_eq!(img.black.values, vec![128.0]);
let c = img.crop;
assert_eq!((c.x, c.y, c.width, c.height), (16, 8, 6048, 4032));
}
/// Issue #535: pre-2017 Sony bodies without crop tags shot in the camera's 16:9 mode. The ILCE-7SM2 records the
/// 16:9 size in `FullImageSize`, the DSLR-A580 only in the Exif image size; both crops are centred vertically (each
/// camera JPEG registered on its raw) and drop the right-edge padding, so the image matches its JPEG's aspect ratio
@ -879,3 +896,38 @@ fn corpus_raws_keep_their_container_and_are_not_thumbnail_shells() {
}
eprintln!("{checked} corpus raws keep their container");
}
/// Matching Sony table dimensions are not evidence of a shared correction model. Exercise the
/// actual corpus headers and decoded files as well as the synthetic model-boundary unit tests.
#[test]
fn sony_embedded_distortion_is_limited_to_validated_models() {
let root = corpus_root().join("raw");
let mut checked = 0;
for (name, model, expected) in [
("arw-sony-a7rm4a-compressed.arw", "ILCE-7RM4A", 1),
("arw-sony-a9m2-compressed.arw", "ILCE-9M2", 0),
("arw-sony-a7m3-compressed.arw", "ILCE-7M3", 0),
("arw-sony-a7m3-uncompressed.arw", "ILCE-7M3", 0),
("arw-sony-a7m4-14bit.arw", "ILCE-7M4", 0),
("arw-sony-a7m4-lossless-l.arw", "ILCE-7M4", 0),
("arw-sony-a7m4-lossless-m.arw", "ILCE-7M4", 0),
("arw-sony-a7m4-lossless-s.arw", "ILCE-7M4", 0),
("arw-sony-a7rm2-12bit-uncompressed.arw", "ILCE-7RM2", 0),
("arw-sony-rx100.arw", "DSC-RX100", 0),
("arw-sony-rx100m3.arw", "DSC-RX100M3", 0),
] {
let path = root.join(name);
if !path.exists() {
eprintln!("skip: {} absent", path.display());
continue;
}
let bytes = std::fs::read(&path).unwrap();
let full = decode(&bytes).unwrap_or_else(|e| panic!("{name}: {e}"));
assert_eq!(probe_info(&bytes).unwrap(), full.info(), "{name}");
assert_eq!(full.metadata.model.as_deref(), Some(model), "{name}");
assert_eq!(full.opcodes.list3.len(), expected, "{name}: only the validated Sony model gets a distortion warp");
eprintln!("{name}: {expected} warp(s); header and full decode agree");
checked += 1;
}
eprintln!("Sony distortion model boundary: {checked} corpus files checked");
}

1
crates/ui-egui/Cargo.toml
View File

@ -28,6 +28,7 @@ log = { workspace = true }
lightcraft-denoise = { workspace = true, features = ["runtime"] }
lightcraft-codecs = { workspace = true }
lightcraft-faces = { workspace = true }
lightcraft-merge = { workspace = true }
[build-dependencies]
serde_json = { workspace = true }

9
crates/ui-egui/locales/de-formats.json
View File

@ -221,5 +221,12 @@
"; {n} gained synonyms": "; mit neuen Synonymen: {n}",
"Exported {n} keyword{}": "Stichwörter exportiert: {n}{:.0}",
" — Capture One won't import a list with ; , < or > in a keyword: {names}": " — Capture One importiert keine Liste mit ; , < oder > in einem Stichwort: {names}",
" — left out, as a keyword list can't hold them: {names}": " — ausgelassen, weil eine Stichwortliste sie nicht aufnehmen kann: {names}"
" — left out, as a keyword list can't hold them: {names}": " — ausgelassen, weil eine Stichwortliste sie nicht aufnehmen kann: {names}",
"{done} of {total}": "{done} von {total}",
"{done} of {total} MB": "{done} von {total} MB",
"{pct} %": "{pct} %",
"+{n} more": "+{n} weitere",
"1 task is still running: {names}. Quitting stops it.": "1 Aufgabe läuft noch: {names}. Beenden bricht sie ab.",
"{n} tasks are still running: {names}. Quitting stops them.": "{n} Aufgaben laufen noch: {names}. Beenden bricht sie ab.",
"{what} merge cancelled": "{what}-Zusammenfügung abgebrochen"
}

17
crates/ui-egui/locales/de.json
View File

@ -2016,5 +2016,20 @@
"Stopped by an error (see the log)": "Durch einen Fehler angehalten (siehe Protokoll)",
"Not saved": "Nicht gespeichert",
"A folder is being synchronized": "Ein Ordner wird gerade synchronisiert",
"Wait for the import to finish": "Warten Sie, bis der Import abgeschlossen ist"
"Wait for the import to finish": "Warten Sie, bis der Import abgeschlossen ist",
"Show less": "Weniger anzeigen",
"Quit while tasks are running?": "Beenden, während Aufgaben laufen?",
"Exporting contact sheet": "Kontaktabzug wird exportiert",
"Reading folder": "Ordner wird gelesen",
"Scanning folder": "Ordner wird durchsucht",
"Building previews": "Vorschauen werden erstellt",
"Building smart previews": "Smart-Vorschauen werden erstellt",
"Discarding smart previews": "Smart-Vorschauen werden verworfen",
"Importing Lightroom catalog": "Lightroom-Katalog wird importiert",
"Reading Lightroom catalog": "Lightroom-Katalog wird gelesen",
"Merging": "Zusammenfügen",
"Downloading model": "Modell wird heruntergeladen",
"Finding faces": "Gesichter werden gesucht",
"Synchronizing folder": "Ordner wird synchronisiert",
"Looking for changes": "Änderungen werden gesucht"
}

9
crates/ui-egui/locales/es-formats.json
View File

@ -221,5 +221,12 @@
"; {n} gained synonyms": "; con sinónimos nuevos: {n}",
"Exported {n} keyword{}": "Palabras clave exportadas: {n}{:.0}",
" — Capture One won't import a list with ; , < or > in a keyword: {names}": " — Capture One no importa una lista con ; , < o > en una palabra clave: {names}",
" — left out, as a keyword list can't hold them: {names}": " — omitidas, porque una lista de palabras clave no puede contenerlas: {names}"
" — left out, as a keyword list can't hold them: {names}": " — omitidas, porque una lista de palabras clave no puede contenerlas: {names}",
"{done} of {total}": "{done} de {total}",
"{done} of {total} MB": "{done} de {total} MB",
"{pct} %": "{pct} %",
"+{n} more": "+{n} más",
"1 task is still running: {names}. Quitting stops it.": "1 tarea sigue en curso: {names}. Salir la detiene.",
"{n} tasks are still running: {names}. Quitting stops them.": "{n} tareas siguen en curso: {names}. Salir las detiene.",
"{what} merge cancelled": "Combinación {what} cancelada"
}

17
crates/ui-egui/locales/es.json
View File

@ -1861,5 +1861,20 @@
"Stopped by an error (see the log)": "Detenido por un error (ver el registro)",
"Not saved": "Sin guardar",
"A folder is being synchronized": "Se está sincronizando una carpeta",
"Wait for the import to finish": "Espera a que termine la importación"
"Wait for the import to finish": "Espera a que termine la importación",
"Show less": "Mostrar menos",
"Quit while tasks are running?": "¿Salir con tareas en curso?",
"Exporting contact sheet": "Exportando hoja de contactos",
"Reading folder": "Leyendo carpeta",
"Scanning folder": "Explorando carpeta",
"Building previews": "Generando vistas previas",
"Building smart previews": "Generando vistas previas inteligentes",
"Discarding smart previews": "Descartando vistas previas inteligentes",
"Importing Lightroom catalog": "Importando catálogo de Lightroom",
"Reading Lightroom catalog": "Leyendo catálogo de Lightroom",
"Merging": "Combinando",
"Downloading model": "Descargando modelo",
"Finding faces": "Buscando caras",
"Synchronizing folder": "Sincronizando carpeta",
"Looking for changes": "Buscando cambios"
}

11
crates/ui-egui/locales/fr-formats.json
View File

@ -221,5 +221,12 @@
"; {n} gained synonyms": " ; avec de nouveaux synonymes : {n}",
"Exported {n} keyword{}": "Mots-clés exportés : {n}{:.0}",
" — Capture One won't import a list with ; , < or > in a keyword: {names}": " — Capture One n’importe pas une liste dont un mot-clé contient ; , < ou > : {names}",
" — left out, as a keyword list can't hold them: {names}": " — omis, car une liste de mots-clés ne peut pas les contenir : {names}"
}
" — left out, as a keyword list can't hold them: {names}": " — omis, car une liste de mots-clés ne peut pas les contenir : {names}",
"{done} of {total}": "{done} sur {total}",
"{done} of {total} MB": "{done} sur {total} Mo",
"{pct} %": "{pct} %",
"+{n} more": "+{n} autres",
"1 task is still running: {names}. Quitting stops it.": "1 tâche est encore en cours : {names}. Quitter l'arrête.",
"{n} tasks are still running: {names}. Quitting stops them.": "{n} tâches sont encore en cours : {names}. Quitter les arrête.",
"{what} merge cancelled": "Fusion {what} annulée"
}

17
crates/ui-egui/locales/fr.json
View File

@ -1861,5 +1861,20 @@
"Stopped by an error (see the log)": "Arrêté par une erreur (voir le journal)",
"Not saved": "Non enregistré",
"A folder is being synchronized": "Un dossier est en cours de synchronisation",
"Wait for the import to finish": "Attendez la fin de l'importation"
"Wait for the import to finish": "Attendez la fin de l'importation",
"Show less": "Afficher moins",
"Quit while tasks are running?": "Quitter pendant que des tâches sont en cours ?",
"Exporting contact sheet": "Exportation de la planche-contact",
"Reading folder": "Lecture du dossier",
"Scanning folder": "Analyse du dossier",
"Building previews": "Génération des aperçus",
"Building smart previews": "Génération des aperçus dynamiques",
"Discarding smart previews": "Suppression des aperçus dynamiques",
"Importing Lightroom catalog": "Importation du catalogue Lightroom",
"Reading Lightroom catalog": "Lecture du catalogue Lightroom",
"Merging": "Fusion en cours",
"Downloading model": "Téléchargement du modèle",
"Finding faces": "Recherche des visages",
"Synchronizing folder": "Synchronisation du dossier",
"Looking for changes": "Recherche des modifications"
}

9
crates/ui-egui/locales/ja-formats.json
View File

@ -221,5 +221,12 @@
"; {n} gained synonyms": "。{n}件に同義語を追加",
"Exported {n} keyword{}": "{n}件のキーワードを書き出しました{:.0}",
" — Capture One won't import a list with ; , < or > in a keyword: {names}": " — Capture One は ; , < > を含むキーワードがあるリストを読み込みません: {names}",
" — left out, as a keyword list can't hold them: {names}": " — キーワードリストに書けないため省略: {names}"
" — left out, as a keyword list can't hold them: {names}": " — キーワードリストに書けないため省略: {names}",
"{done} of {total}": "{total}件中{done}件",
"{done} of {total} MB": "{done} / {total} MB",
"{pct} %": "{pct}%",
"+{n} more": "ほか{n}件",
"1 task is still running: {names}. Quitting stops it.": "1件のタスクが実行中です：{names}。終了すると停止します。",
"{n} tasks are still running: {names}. Quitting stops them.": "{n}件のタスクが実行中です：{names}。終了すると停止します。",
"{what} merge cancelled": "{what}合成をキャンセルしました"
}

17
crates/ui-egui/locales/ja.json
View File

@ -1863,5 +1863,20 @@
"Stopped by an error (see the log)": "エラーで停止しました（ログを参照）",
"Not saved": "保存されていません",
"A folder is being synchronized": "フォルダを同期中です",
"Wait for the import to finish": "読み込みが終わるまでお待ちください"
"Wait for the import to finish": "読み込みが終わるまでお待ちください",
"Show less": "表示を減らす",
"Quit while tasks are running?": "タスクの実行中に終了しますか？",
"Exporting contact sheet": "コンタクトシートを書き出し中",
"Reading folder": "フォルダーを読み込み中",
"Scanning folder": "フォルダーをスキャン中",
"Building previews": "プレビューを作成中",
"Building smart previews": "スマートプレビューを作成中",
"Discarding smart previews": "スマートプレビューを破棄中",
"Importing Lightroom catalog": "Lightroom カタログを読み込み中",
"Reading Lightroom catalog": "Lightroom カタログを読み取り中",
"Merging": "合成中",
"Downloading model": "モデルをダウンロード中",
"Finding faces": "顔を検索中",
"Synchronizing folder": "フォルダを同期中",
"Looking for changes": "変更を確認中"
}

9
crates/ui-egui/locales/pt-br-formats.json
View File

@ -221,5 +221,12 @@
"; {n} gained synonyms": "; {n} receberam sinônimos",
"Exported {n} keyword{}": "Palavras-chave exportadas: {n}{:.0}",
" — Capture One won't import a list with ; , < or > in a keyword: {names}": " — O Capture One não importa uma lista com ; , < ou > em uma palavra-chave: {names}",
" — left out, as a keyword list can't hold them: {names}": " — omitidas, pois uma lista de palavras-chave não pode contê-las: {names}"
" — left out, as a keyword list can't hold them: {names}": " — omitidas, pois uma lista de palavras-chave não pode contê-las: {names}",
"{done} of {total}": "{done} de {total}",
"{done} of {total} MB": "{done} de {total} MB",
"{pct} %": "{pct}%",
"+{n} more": "+{n} mais",
"1 task is still running: {names}. Quitting stops it.": "1 tarefa ainda está em andamento: {names}. Sair a interrompe.",
"{n} tasks are still running: {names}. Quitting stops them.": "{n} tarefas ainda estão em andamento: {names}. Sair as interrompe.",
"{what} merge cancelled": "{what} mesclagem cancelada"
}

17
crates/ui-egui/locales/pt-br.json
View File

@ -1861,5 +1861,20 @@
"Stopped by an error (see the log)": "Interrompido por um erro (veja o log)",
"Not saved": "Não salvo",
"A folder is being synchronized": "Uma pasta está sendo sincronizada",
"Wait for the import to finish": "Aguarde a importação terminar"
"Wait for the import to finish": "Aguarde a importação terminar",
"Show less": "Mostrar menos",
"Quit while tasks are running?": "Sair com tarefas em andamento?",
"Exporting contact sheet": "Exportando folha de contato",
"Reading folder": "Lendo pasta",
"Scanning folder": "Verificando pasta",
"Building previews": "Gerando prévias",
"Building smart previews": "Gerando prévias inteligentes",
"Discarding smart previews": "Descartando prévias inteligentes",
"Importing Lightroom catalog": "Importando catálogo do Lightroom",
"Reading Lightroom catalog": "Lendo catálogo do Lightroom",
"Merging": "Mesclando",
"Downloading model": "Baixando modelo",
"Finding faces": "Procurando rostos",
"Synchronizing folder": "Sincronizando pasta",
"Looking for changes": "Procurando alterações"
}

9
crates/ui-egui/locales/ru-formats.json
View File

@ -221,5 +221,12 @@
"; {n} gained synonyms": "; синонимы добавлены к {n}",
"Exported {n} keyword{}": "Экспортировано ключевых слов: {n}{:.0}",
" — Capture One won't import a list with ; , < or > in a keyword: {names}": " — Capture One не импортирует список, где в ключевом слове есть ; , < или >: {names}",
" — left out, as a keyword list can't hold them: {names}": " — пропущены: список ключевых слов не может их содержать: {names}"
" — left out, as a keyword list can't hold them: {names}": " — пропущены: список ключевых слов не может их содержать: {names}",
"{done} of {total}": "{done} из {total}",
"{done} of {total} MB": "{done} из {total} МБ",
"{pct} %": "{pct} %",
"+{n} more": "ещё {n}",
"1 task is still running: {names}. Quitting stops it.": "Ещё выполняется задача: {names}. Выход прервёт её.",
"{n} tasks are still running: {names}. Quitting stops them.": "Ещё выполняется задач: {n} — {names}. Выход прервёт их.",
"{what} merge cancelled": "{what}: слияние отменено"
}

18
crates/ui-egui/locales/ru.json
View File

@ -1574,5 +1574,21 @@
"Stopped by an error (see the log)": "Остановлено из-за ошибки (см. журнал)",
"Not saved": "Не сохранено",
"A folder is being synchronized": "Идёт синхронизация папки",
"Wait for the import to finish": "Дождитесь окончания импорта"
"Wait for the import to finish": "Дождитесь окончания импорта",
"Show less": "Свернуть",
"Quit while tasks are running?": "Выйти, пока выполняются задачи?",
"Exporting contact sheet": "Экспорт контрольного листа",
"Reading folder": "Чтение папки",
"Scanning folder": "Сканирование папки",
"Building previews": "Создание просмотров",
"Building smart previews": "Создание смарт-просмотров",
"Discarding smart previews": "Удаление смарт-просмотров",
"Importing Lightroom catalog": "Импорт каталога Lightroom",
"Reading Lightroom catalog": "Чтение каталога Lightroom",
"Merging": "Слияние",
"Downloading model": "Загрузка модели",
"Finding faces": "Поиск лиц",
"Quit Anyway": "Всё равно выйти",
"Synchronizing folder": "Синхронизация папки",
"Looking for changes": "Поиск изменений"
}

9
crates/ui-egui/locales/uk-formats.json
View File

@ -221,5 +221,12 @@
"; {n} gained synonyms": "; отримали синоніми: {n}",
"Exported {n} keyword{}": "Експортовано ключових слів: {n}{:.0}",
" — Capture One won't import a list with ; , < or > in a keyword: {names}": " — Capture One не імпортує список, у якому ключове слово містить ; , < або >: {names}",
" — left out, as a keyword list can't hold them: {names}": " — пропущено, бо список ключових слів не може їх містити: {names}"
" — left out, as a keyword list can't hold them: {names}": " — пропущено, бо список ключових слів не може їх містити: {names}",
"{done} of {total}": "{done} з {total}",
"{done} of {total} MB": "{done} з {total} МБ",
"{pct} %": "{pct} %",
"+{n} more": "+{n} ще",
"1 task is still running: {names}. Quitting stops it.": "1 завдання ще виконується: {names}. Вихід зупинить його.",
"{n} tasks are still running: {names}. Quitting stops them.": "Ще виконуються завдання ({n}): {names}. Вихід зупинить їх.",
"{what} merge cancelled": "Об’єднання {what} скасовано"
}

17
crates/ui-egui/locales/uk.json
View File

@ -2055,5 +2055,20 @@
"Stopped by an error (see the log)": "Зупинено через помилку (див. журнал)",
"Not saved": "Не збережено",
"A folder is being synchronized": "Тека синхронізується",
"Wait for the import to finish": "Дочекайтеся завершення імпорту"
"Wait for the import to finish": "Дочекайтеся завершення імпорту",
"Show less": "Показати менше",
"Quit while tasks are running?": "Вийти, поки виконуються завдання?",
"Exporting contact sheet": "Експорт аркуша мініатюр",
"Reading folder": "Читання папки",
"Scanning folder": "Сканування папки",
"Building previews": "Створення попередніх переглядів",
"Building smart previews": "Створення смарт-переглядів",
"Discarding smart previews": "Видалення смарт-переглядів",
"Importing Lightroom catalog": "Імпорт каталогу Lightroom",
"Reading Lightroom catalog": "Читання каталогу Lightroom",
"Merging": "Об’єднання",
"Downloading model": "Завантаження моделі",
"Finding faces": "Пошук облич",
"Synchronizing folder": "Синхронізація папки",
"Looking for changes": "Пошук змін"
}

9
crates/ui-egui/locales/zh-hans-formats.json
View File

@ -221,5 +221,12 @@
"; {n} gained synonyms": "；{n} 个获得了同义词",
"Exported {n} keyword{}": "已导出 {n} 个关键字{:.0}",
" — Capture One won't import a list with ; , < or > in a keyword: {names}": " — Capture One 不会导入关键字中含有 ; , < 或 > 的列表：{names}",
" — left out, as a keyword list can't hold them: {names}": " — 已省略，关键字列表无法容纳: {names}"
" — left out, as a keyword list can't hold them: {names}": " — 已省略，关键字列表无法容纳: {names}",
"{done} of {total}": "{done}／{total}",
"{done} of {total} MB": "{done}／{total} MB",
"{pct} %": "{pct}%",
"+{n} more": "还有 {n} 项",
"1 task is still running: {names}. Quitting stops it.": "仍有 1 个任务在运行：{names}。退出将停止它。",
"{n} tasks are still running: {names}. Quitting stops them.": "仍有 {n} 个任务在运行：{names}。退出将停止它们。",
"{what} merge cancelled": "已取消 {what} 合并"
}

17
crates/ui-egui/locales/zh-hans.json
View File

@ -1861,5 +1861,20 @@
"Stopped by an error (see the log)": "因错误而停止（见日志）",
"Not saved": "未保存",
"A folder is being synchronized": "正在同步文件夹",
"Wait for the import to finish": "请等待导入完成"
"Wait for the import to finish": "请等待导入完成",
"Show less": "收起",
"Quit while tasks are running?": "要在任务运行时退出吗？",
"Exporting contact sheet": "正在导出联系表",
"Reading folder": "正在读取文件夹",
"Scanning folder": "正在扫描文件夹",
"Building previews": "正在构建预览",
"Building smart previews": "正在构建智能预览",
"Discarding smart previews": "正在丢弃智能预览",
"Importing Lightroom catalog": "正在导入 Lightroom 目录",
"Reading Lightroom catalog": "正在读取 Lightroom 目录",
"Merging": "正在合并",
"Downloading model": "正在下载模型",
"Finding faces": "正在查找人脸",
"Synchronizing folder": "正在同步文件夹",
"Looking for changes": "正在查找更改"
}

9
crates/ui-egui/locales/zh-hant-formats.json
View File

@ -221,5 +221,12 @@
"; {n} gained synonyms": "；{n} 個獲得了同義詞",
"Exported {n} keyword{}": "已匯出 {n} 個關鍵字{:.0}",
" — Capture One won't import a list with ; , < or > in a keyword: {names}": " — Capture One 不會匯入關鍵字中含有 ; , < 或 > 的清單：{names}",
" — left out, as a keyword list can't hold them: {names}": " — 已省略，關鍵字清單無法容納: {names}"
" — left out, as a keyword list can't hold them: {names}": " — 已省略，關鍵字清單無法容納: {names}",
"{done} of {total}": "{done}／{total}",
"{done} of {total} MB": "{done}／{total} MB",
"{pct} %": "{pct}%",
"+{n} more": "還有 {n} 項",
"1 task is still running: {names}. Quitting stops it.": "仍有 1 個工作在執行：{names}。結束將停止它。",
"{n} tasks are still running: {names}. Quitting stops them.": "仍有 {n} 個工作在執行：{names}。結束將停止它們。",
"{what} merge cancelled": "已取消{what}合併"
}

17
crates/ui-egui/locales/zh-hant.json
View File

@ -1861,5 +1861,20 @@
"Stopped by an error (see the log)": "因錯誤而停止（請見記錄）",
"Not saved": "未儲存",
"A folder is being synchronized": "正在同步資料夾",
"Wait for the import to finish": "請等待匯入完成"
"Wait for the import to finish": "請等待匯入完成",
"Show less": "收合",
"Quit while tasks are running?": "要在工作執行時結束嗎？",
"Exporting contact sheet": "正在匯出聯絡表",
"Reading folder": "正在讀取檔案夾",
"Scanning folder": "正在掃描檔案夾",
"Building previews": "正在建立預覽",
"Building smart previews": "正在建立智慧型預覽",
"Discarding smart previews": "正在捨棄智慧型預覽",
"Importing Lightroom catalog": "正在匯入 Lightroom 目錄",
"Reading Lightroom catalog": "正在讀取 Lightroom 目錄",
"Merging": "正在合併",
"Downloading model": "正在下載模型",
"Finding faces": "正在尋找臉孔",
"Synchronizing folder": "正在同步資料夾",
"Looking for changes": "正在尋找變更"
}

4
crates/ui-egui/src/control.rs
View File

@ -105,7 +105,7 @@ pub fn inspect(app: &LightcraftApp, ctx: &egui::Context) -> Value {
"status": app.ui.status,
"copied": app.copied,
"notices": app.notices,
"quitPrompt": app.quit_prompt,
"quitPrompt": app.quit_prompt.as_ref().map(crate::QuitPrompt::text),
// how hard the background face scan may work right now, and why it is judged so (see `panels::faces::scan_pace`)
"faceScan": {"pace": app.caches.faces_pace, "focused": app.caches.faces_in_front, "pending": app.caches.faces_pending, "indexed": app.caches.faces_indexed},
"unsaved": app.session.unsaved().map(|(n, e)| json!({"ops": n, "error": e})),
@ -114,6 +114,8 @@ pub fn inspect(app: &LightcraftApp, ctx: &egui::Context) -> Value {
"export": {"running": app.export.as_ref().map(crate::export_task::ExportTask::status), "last": app.last_export_result},
"import": app.import.as_ref().map(crate::import::ImportTask::status),
"tasks": app.tasks.labels(),
// every long-running task in flight, as the activity stack shows them (`activity.list`)
"activity": app.session.activity.list(),
// commands waiting on a native file dialog shown off the UI thread (`pick`)
"fileDialogs": app.pending_picks.iter().map(|p| p.command.clone()).collect::<Vec<_>>(),
"memory": memory(app),

69
crates/ui-egui/src/export_task.rs
View File

@ -1,8 +1,8 @@
//! Background export: the Export dialog, File → Export with Preset and Export with Previous hand
//! their batch to a worker thread so the window stays responsive. Photos are prepared on the UI
//! thread ([`lightcraft_engine::export::prepare_export`]: cheap, needs the session) and rendered,
//! encoded and written on the worker (several side by side: [`run_batch`]); a progress panel shows
//! the count and a Cancel button.
//! encoded and written on the worker (several side by side: [`run_batch`]); its row in the
//! activity stack shows the count, the file in progress and ✕ (issue #345).
//!
//! Needs [`crate::Services::write_shared`] (a thread-safe writer); without it (web) the batch runs
//! synchronously as before.
@ -11,12 +11,11 @@ use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::mpsc::{Receiver, channel};
use std::sync::{Arc, Mutex};
use egui::Align2;
use lightcraft_engine::activity::{Cancel, TaskGuard};
use lightcraft_engine::export::{Destination, ExportOptions, PreparedExport, run_batch};
use serde_json::{Value, json};
use crate::LightcraftApp;
use crate::theme::Tokens;
pub struct ExportTask {
pub total: usize,
@ -24,6 +23,8 @@ pub struct ExportTask {
pub progress: Arc<Mutex<(usize, String)>>,
pub cancel: Arc<AtomicBool>,
rx: Receiver<Result<Vec<Value>, String>>,
/// The export's row in the activity stack (`activity.cancel` sets `cancel`).
guard: TaskGuard,
}
impl ExportTask {
@ -59,11 +60,13 @@ pub fn start(app: &mut LightcraftApp, items: Vec<PreparedExport>, opts: ExportOp
std::thread::Builder::new().name("export".into()).spawn(work).map_err(|e| e.to_string())?;
#[cfg(target_arch = "wasm32")]
work();
app.export = Some(ExportTask { total, progress, cancel, rx });
let guard = app.session.activity.start("export", "Exporting", Cancel::Flag(cancel.clone()));
guard.progress(0, total as u64);
app.export = Some(ExportTask { total, progress, cancel, rx, guard });
Ok(json!({"background": true, "total": total}))
}
/// Export a single paginated PDF using the same progress and cancellation UI as image exports.
/// Export a single paginated PDF with the same activity row and cancellation as image exports.
pub fn start_contact_sheet(app: &mut LightcraftApp, params: &Value) -> Result<Value, String> {
if app.export.is_some() {
return Err("an export is already running".into());
@ -103,11 +106,13 @@ pub fn start_contact_sheet(app: &mut LightcraftApp, params: &Value) -> Result<Va
std::thread::Builder::new().name("contact-sheet".into()).spawn(work).map_err(|e| e.to_string())?;
#[cfg(target_arch = "wasm32")]
work();
app.export = Some(ExportTask { total, progress, cancel, rx });
let row = app.session.activity.start("export", "Exporting contact sheet", Cancel::Flag(cancel.clone()));
row.progress(0, total as u64);
app.export = Some(ExportTask { total, progress, cancel, rx, guard: row });
Ok(json!({"background": true, "total": total}))
}
/// Per frame: draw the progress panel; when the batch finishes, report it.
/// Per frame: keep the activity row up to date; when the batch finishes, report it.
pub fn poll(app: &mut LightcraftApp, ctx: &egui::Context) {
let Some(task) = &app.export else { return };
match task.rx.try_recv() {
@ -144,30 +149,8 @@ pub fn poll(app: &mut LightcraftApp, ctx: &egui::Context) {
}
Err(std::sync::mpsc::TryRecvError::Empty) => {
let (done, current) = task.progress.lock().map(|g| g.clone()).unwrap_or_default();
let (total, cancel) = (task.total, task.cancel.clone());
let t = Tokens::get(ctx);
egui::Window::new(crate::i18n::tr("Exporting"))
.title_bar(false)
.resizable(false)
.anchor(Align2::LEFT_BOTTOM, [16.0, -56.0])
.fixed_size([300.0, 64.0])
.show(ctx, |ui| {
ui.horizontal(|ui| {
ui.label(
egui::RichText::new(crate::i18n::tr_format!("Exporting {} of {total}", (done + 1).min(total), total = total))
.color(t.text),
);
ui.with_layout(egui::Layout::right_to_left(egui::Align::Center), |ui| {
let stopping = cancel.load(Ordering::Relaxed);
if crate::widgets::text_button(ui, "exportCancel", if stopping { "Stopping…" } else { "Cancel" }, false).clicked() {
cancel.store(true, Ordering::Relaxed);
}
});
});
ui.add(egui::ProgressBar::new(done as f32 / total.max(1) as f32).desired_width(280.0));
ui.label(egui::RichText::new(current).size(11.0).color(t.text_dim));
});
ctx.request_repaint_after(std::time::Duration::from_millis(100));
task.guard.progress(done as u64, task.total as u64);
task.guard.detail(&current);
}
Err(std::sync::mpsc::TryRecvError::Disconnected) => {
app.export = None;
@ -214,7 +197,7 @@ mod contact_sheet_tests {
assert_eq!(h.app.pending_picks.len(), 1);
let deadline = Instant::now() + Duration::from_secs(20);
loop {
// The progress window lays out text, which egui only allows inside a frame.
// The activity stack lays out text, which egui only allows inside a frame.
h.step();
if h.app.last_export_result.is_some() {
break;
@ -232,6 +215,26 @@ mod contact_sheet_tests {
assert_eq!(h.app.last_export_result.as_ref().unwrap()["files"][0]["photos"], 1);
}
#[test]
fn contact_sheet_export_shows_a_row_and_its_cross_stops_it() {
let mut app = LightcraftApp::new(
lightcraft_engine::Session::with_demo(),
crate::Services { write_shared: Some(Arc::new(|_, _| Ok(()))), ..Default::default() },
);
let ids: Vec<_> = app.session.catalog.photos().take(24).map(|p| p.id.0).collect();
start_contact_sheet(&mut app, &json!({"path": "Sheet.pdf", "ids": ids})).unwrap();
let rows = app.session.activity.list();
assert_eq!(rows.len(), 1, "{rows:?}");
assert_eq!((rows[0].kind, rows[0].label.as_str(), rows[0].total), ("export", "Exporting contact sheet", 24));
assert!(rows[0].cancellable);
app.session.activity.cancel(rows[0].id).unwrap();
assert!(app.export.as_ref().unwrap().cancel.load(Ordering::Relaxed), "the stack's ✕ stops the sheet");
let task = app.export.take().unwrap();
assert!(task.rx.recv_timeout(Duration::from_secs(20)).unwrap().is_err(), "a stopped sheet is not written");
drop(task);
assert!(app.session.activity.list().is_empty(), "the row goes with the export");
}
#[test]
fn cancelled_contact_sheet_never_calls_writer() {
let mut app = LightcraftApp::new(

15
crates/ui-egui/src/headless.rs
View File

@ -1787,6 +1787,8 @@ mod tests {
assert!(h.app.export.is_some(), "running in the background");
let running = h.request("ui.inspect", json!({}), t);
assert_eq!(running["result"]["export"]["running"]["total"], 3, "{}", running["result"]["export"]);
// …and as a row in the activity stack (issue #345)
assert_eq!(running["result"]["activity"][0]["kind"], "export", "{}", running["result"]["activity"]);
let t0 = Instant::now();
while h.app.export.is_some() && t0.elapsed() < Duration::from_secs(60) {
h.step();
@ -1795,7 +1797,9 @@ mod tests {
let w = written.lock().unwrap().clone();
assert_eq!(w.len(), 3, "{w:?}");
assert!(w.iter().all(|p| p.starts_with("/lc-test-out/")));
let last = h.request("ui.inspect", json!({}), t)["result"]["export"]["last"].clone();
let after = h.request("ui.inspect", json!({}), t)["result"].clone();
assert_eq!(after["activity"], json!([]), "the row is gone");
let last = after["export"]["last"].clone();
assert_eq!(last["files"].as_array().map(Vec::len), Some(3), "{last}");
assert!(last["files"][0]["width"].as_u64().is_some_and(|w| w <= 64));
}
@ -2434,9 +2438,14 @@ mod tests {
assert!(!h.app.scan.as_ref().unwrap().copy, "the running scan keeps its options");
let r = h.request("ui.inspect", json!({}), t);
assert!(r["result"]["scan"].is_object(), "{r}");
let r = h.request("ui.clickWidget", json!({"id": "button:scanCancel"}), t);
// Cancel is the ✕ of the scan's row in the activity stack (shown once it is half a second old)
let id = h.app.session.activity.list().first().map(|t| t.id).expect("a scan row");
let cross = format!("activity:cancel:{id}");
assert!(h.step_until(t, |h| h.app.widgets.iter().any(|(w, _)| *w == cross)), "the row shows");
let r = h.request("ui.clickWidget", json!({"id": cross}), t);
assert_eq!(r["ok"], true, "{r}");
assert!(h.app.scan.is_none(), "Cancel closes the scan at once");
h.step();
assert!(h.app.scan.is_none(), "✕ closes the scan at once");
gate.store(true, std::sync::atomic::Ordering::Relaxed);
h.settle(SETTLE);
assert!(h.app.ui.dialog.is_none(), "a cancelled scan opens no review");

110
crates/ui-egui/src/import.rs
View File

@ -5,7 +5,8 @@
//! chosen folder, filed by day, by month, into one folder or by a custom folder template,
//! optionally renamed), an album (existing or new), a preset and keywords to apply. Importing
//! runs on a worker thread (files are probed, copied or moved there) and its batches join the
//! catalog between frames, with a progress window and Cancel; the whole import is one undo step.
//! catalog between frames, with a row and ✕ in the activity stack (issue #345); the whole import is
//! one undo step.
use egui::{Align2, Color32, Rect, Sense, Stroke, StrokeKind, pos2, vec2};
use lightcraft_engine::import::{ImportCandidate, ScanInput, ScanOutput, ScanProgress, scan_with};
@ -235,6 +236,8 @@ pub struct ImportTask {
/// Auto Import: the selection to keep.
keep_selection: Option<lightcraft_engine::Selection>,
run: Option<ImportRun>,
/// The import's row in the activity stack (its ✕ and `activity.cancel` set the run's cancel flag).
guard: Option<lightcraft_engine::activity::TaskGuard>,
}
impl ImportTask {
@ -391,6 +394,14 @@ pub struct ScanTask {
browse: bool,
/// What is being scanned (the review's source).
sources: Vec<String>,
/// The scan's row in the activity stack (its ✕ and `activity.cancel` set `progress.cancel`).
guard: lightcraft_engine::activity::TaskGuard,
}
/// The activity row of a scan: the review's ("Scanning folder") or the Local view's ("Reading folder").
fn scan_guard(app: &LightcraftApp, progress: &ScanProgress, browse: bool) -> lightcraft_engine::activity::TaskGuard {
let label = if browse { "Reading folder" } else { "Scanning folder" };
app.session.activity.start("scan", label, lightcraft_engine::activity::Cancel::Flag(progress.cancel.clone()))
}
/// An import and a Synchronize Folder never run at once: each readies its files against the
@ -416,7 +427,8 @@ pub fn open(app: &mut LightcraftApp, paths: Vec<String>) -> Result<Value, String
std::thread::spawn(job);
#[cfg(target_arch = "wasm32")]
job();
app.scan = Some(ScanTask { progress, rx, copy: false, browse: false, sources });
let guard = scan_guard(app, &progress, false);
app.scan = Some(ScanTask { progress, rx, copy: false, browse: false, sources, guard });
Ok(json!({"scanning": true}))
}
@ -477,7 +489,8 @@ pub fn browse(app: &mut LightcraftApp, path: &str, subfolders: Option<bool>) ->
std::thread::spawn(job);
#[cfg(target_arch = "wasm32")]
job();
app.scan = Some(ScanTask { progress, rx, copy: false, browse: true, sources: Vec::new() });
let guard = scan_guard(app, &progress, true);
app.scan = Some(ScanTask { progress, rx, copy: false, browse: true, sources: Vec::new(), guard });
app.renderer.forget_imports();
Ok(json!({"path": dir_s, "subfolders": subfolders, "scanning": true}))
}
@ -488,7 +501,18 @@ pub fn poll_scan(app: &mut LightcraftApp, ctx: &egui::Context) {
ctx.request_repaint_after(std::time::Duration::from_millis(100));
let out = match task.rx.try_recv() {
Ok(o) => o,
Err(std::sync::mpsc::TryRecvError::Empty) => return,
Err(std::sync::mpsc::TryRecvError::Empty) => {
if task.guard.is_cancelled() {
// ✕: the worker stops at its next file; don't wait for it (a NAS read can take a while)
app.scan = None;
return;
}
let (total, done) = (task.progress.total.load(Ordering::Relaxed), task.progress.done.load(Ordering::Relaxed));
task.guard.progress(done as u64, total as u64);
// while the folders are listed the total isn't known: say what is happening instead
task.guard.detail(if total == 0 { crate::i18n::tr("Looking for photos…") } else { "" });
return;
}
Err(_) => {
app.scan = None;
app.toast(ctx, crate::i18n::tr("Scan failed"));
@ -531,37 +555,6 @@ pub fn poll_scan(app: &mut LightcraftApp, ctx: &egui::Context) {
app.ui.dialog = Some(crate::state::Dialog::Import { opts: Box::new(d) });
}
/// The progress window while a folder is being scanned.
pub fn scan_progress(app: &mut LightcraftApp, ctx: &egui::Context) {
let Some(task) = &app.scan else { return };
let t = Tokens::get(ctx);
let total = task.progress.total.load(Ordering::Relaxed);
let done = task.progress.done.load(Ordering::Relaxed);
let text = if total == 0 {
if task.browse { "Reading folder…" } else { "Looking for photos…" }.to_string()
} else {
crate::i18n::tr_format!("Reading photos… {done} of {total}", done = done, total = total)
};
let mut cancel = false;
egui::Window::new(crate::i18n::tr("Scanning"))
.title_bar(false)
.resizable(false)
.anchor(Align2::CENTER_BOTTOM, [0.0, -80.0])
.fixed_size([340.0, 80.0])
.show(ctx, |ui| {
ui.label(egui::RichText::new(text).color(t.text));
ui.add(egui::ProgressBar::new(done as f32 / total.max(1) as f32).desired_width(320.0));
let r = ui.button(crate::i18n::tr("Cancel"));
register(ui.ctx(), "button:scanCancel", r.rect);
cancel = r.clicked();
});
if cancel {
// the worker stops at its next file; don't wait for it (a NAS read can take a while)
task.progress.cancel.store(true, Ordering::Relaxed);
app.scan = None;
}
}
impl ScanTask {
/// `{done, total}` for `ui.inspect` (total is 0 while the folders are still being listed).
pub fn status(&self) -> Value {
@ -648,7 +641,11 @@ pub fn tick(app: &mut LightcraftApp, ctx: &egui::Context) {
task.keep_selection = Some(app.session.selection.clone());
}
match started {
Ok(run) => task.run = Some(run),
Ok(run) => {
let label = if task.browse { "Reading folder" } else { "Importing" };
task.guard = Some(app.session.activity.start("import", label, lightcraft_engine::activity::Cancel::Flag(run.cancel.clone())));
task.run = Some(run);
}
Err(e) => {
log::warn!("import: {e}");
app.toast(ctx, crate::i18n::tr_format!("Import failed: {e}", e = e));
@ -671,6 +668,13 @@ pub fn tick(app: &mut LightcraftApp, ctx: &egui::Context) {
task.total = run.total.load(Ordering::Relaxed).max(task.total);
task.done = run.done.load(Ordering::Relaxed);
}
if let Some(guard) = &task.guard {
guard.progress(task.done as u64, task.total as u64);
// ✕ in the stack (or `activity.cancel`): no further files are started
if guard.is_cancelled() {
task.cancelled = true;
}
}
if !finished {
app.import = Some(task);
return;
@ -812,40 +816,6 @@ fn show_existing(app: &mut LightcraftApp, ctx: &egui::Context, existing: &[u64])
true
}
/// The progress window while an import runs, with Cancel (files already copied or added stay;
/// nothing new is started).
pub fn progress(app: &mut LightcraftApp, ctx: &egui::Context) {
let Some(task) = &app.import else { return };
let t = Tokens::get(ctx);
let frac = task.done as f32 / task.total.max(1) as f32;
let text = if task.cancelled {
crate::i18n::tr("Stopping…").to_string()
} else if task.browse {
crate::i18n::tr_format!("Reading photos… {} of {}", task.done, task.total)
} else {
crate::i18n::tr_format!("Adding photos… {} of {}", task.done, task.total)
};
let mut cancel = false;
egui::Window::new(crate::i18n::tr("Importing"))
.title_bar(false)
.resizable(false)
.anchor(Align2::CENTER_BOTTOM, [0.0, -80.0])
.fixed_size([340.0, 80.0])
.show(ctx, |ui| {
ui.label(egui::RichText::new(text).color(t.text));
ui.add(egui::ProgressBar::new(frac).desired_width(320.0));
let r = ui.add_enabled(!task.cancelled, egui::Button::new(crate::i18n::tr("Cancel")));
register(ui.ctx(), "button:importCancel", r.rect);
cancel = r.clicked();
});
if cancel && let Some(task) = app.import.as_mut() {
task.cancelled = true;
if let Some(run) = &task.run {
run.cancel.store(true, Ordering::Relaxed);
}
}
}
/// The dialog body: options, then the candidate grid.
pub fn body(app: &mut LightcraftApp, ui: &mut egui::Ui, d: &mut ImportDialog) {
let t = Tokens::get(ui.ctx());

46
crates/ui-egui/src/lib.rs
View File

@ -36,6 +36,8 @@ pub mod theme;
pub mod titlebar;
pub mod widgets;
#[cfg(test)]
mod tests_activity;
#[cfg(test)]
mod tests_album_picker;
#[cfg(test)]
@ -189,6 +191,24 @@ pub struct Perf {
pub fps: f64,
}
/// Why quitting stopped to ask first (`panels::notices`).
#[derive(Clone, Debug, PartialEq)]
pub enum QuitPrompt {
/// Changes couldn't be written to disk (issue #103).
Unsaved(String),
/// Tasks that quitting would cut short are running: an import, an export… (issue #345).
Tasks(String),
}
impl QuitPrompt {
/// The prompt's message.
pub fn text(&self) -> &str {
match self {
QuitPrompt::Unsaved(text) | QuitPrompt::Tasks(text) => text,
}
}
}
pub struct LightcraftApp {
pub(crate) model_setup: model_setup::Pending,
/// Per-catalog-revision caches of library-wide results the panels show every frame
@ -213,8 +233,8 @@ pub struct LightcraftApp {
pub headless_host: bool,
/// Warnings to show one at a time (damaged settings files…, issue #103).
pub notices: Vec<String>,
/// Quitting was stopped because changes couldn't be saved: the prompt's text.
pub quit_prompt: Option<String>,
/// Quitting was stopped to ask first: changes couldn't be saved, or tasks are running.
pub quit_prompt: Option<QuitPrompt>,
/// Quit Anyway was chosen: the window may close with unsaved changes.
pub quit_confirmed: bool,
control_rx: Option<Receiver<ControlRequest>>,
@ -302,6 +322,8 @@ pub struct LightcraftApp {
/// The library failed to open at launch: the blocking window, then the temporary-session
/// banner (issue #100). Cleared once a library opens.
pub library_problem: Option<panels::library_problem::LibraryProblem>,
/// The activity stack shows every task, not just the first few ("+N more" was clicked).
pub activity_expanded: bool,
}
impl LightcraftApp {
@ -367,6 +389,7 @@ impl LightcraftApp {
display_applied: None,
display_error: None,
library_problem: None,
activity_expanded: false,
model_setup: Default::default(),
}
}
@ -454,7 +477,7 @@ impl LightcraftApp {
ctx.request_repaint_after(std::time::Duration::from_secs_f64((due - now).clamp(0.05, interval)));
}
/// Announce the start and end of a Build Previews run.
/// Announce the end of a Build Previews run.
fn preview_build_status(&mut self, ctx: &egui::Context) {
use std::sync::atomic::Ordering;
let Some(b) = self.session.preview_build.clone() else { return };
@ -477,14 +500,7 @@ impl LightcraftApp {
self.toast(ctx, msg);
}
} else {
if self.ui.preview_build_seen != Some((key, false)) {
self.ui.preview_build_seen = Some((key, false));
let msg = match b.what {
"" => crate::i18n::tr_format!("Building previews for {} photos…", b.total),
what => crate::i18n::tr_format!("Working on {what} for {} photos…", b.total, what = crate::i18n::tr(what)),
};
self.toast(ctx, msg);
}
// The activity stack shows the run while it works; poll for the end.
ctx.request_repaint_after(std::time::Duration::from_millis(250));
}
}
@ -852,7 +868,8 @@ impl LightcraftApp {
let undo0 = app.session.undo.len();
app.import = Some(import::ImportTask::new(paths, params, undo0, false).auto());
};
if let Err(e) = tasks::spawn(self, LABEL, work, done) {
// listed every few seconds: quiet, no row in the activity stack
if let Err(e) = tasks::spawn(self, LABEL, None, work, done) {
log::warn!("{e}");
}
}
@ -1049,11 +1066,8 @@ impl LightcraftApp {
panels::notices::show(self, &ctx);
panels::dialogs::show(self, &ctx);
panels::library_problem::show(self, &ctx);
import::progress(self, &ctx);
sync::progress_window(self, &ctx);
import::scan_progress(self, &ctx);
lightroom_import::progress(self, &ctx);
export_task::poll(self, &ctx);
panels::activity::show(self, &ctx);
pick::poll(self, &ctx);
panels::grid::drag_feedback(self, &ctx);
panels::left::album_drag_feedback(self, &ctx);

156
crates/ui-egui/src/lightroom_import.rs
View File

@ -8,12 +8,10 @@ use std::path::PathBuf;
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
use std::sync::{Arc, mpsc};
use egui::Align2;
use lightcraft_engine::activity::{Cancel, TaskGuard};
use serde_json::{Value, json};
use crate::LightcraftApp;
use crate::theme::Tokens;
use crate::widgets::register;
#[derive(Clone, Copy)]
enum Kind {
@ -34,7 +32,7 @@ enum Phase {
Finalizing,
}
/// A Lightroom task in flight. The receiver is polled from the egui owner thread.
/// A Lightroom task in flight. The receiver is polled from the egui owner thread; the activity stack shows it.
pub struct LightroomTask {
kind: Kind,
path: PathBuf,
@ -44,9 +42,53 @@ pub struct LightroomTask {
rx: mpsc::Receiver<Message>,
phase: Phase,
cancelled: bool,
guard: TaskGuard,
}
impl LightroomTask {
#[cfg(not(target_arch = "wasm32"))]
fn new(
s: &lightcraft_engine::Session,
kind: Kind,
path: PathBuf,
cancel: Arc<AtomicBool>,
total: Arc<AtomicUsize>,
done: Arc<AtomicUsize>,
rx: mpsc::Receiver<Message>,
) -> Self {
let label = match kind {
Kind::Inspect => "Reading Lightroom catalog",
Kind::Import => "Importing Lightroom catalog",
};
let guard = s.activity.start("lightroom", label, Cancel::Flag(cancel.clone()));
let task = Self { kind, path, cancel, total, done, rx, phase: Phase::Reading, cancelled: false, guard };
task.refresh();
task
}
/// Move to `phase`; only reading the catalog can stop, adding the photos runs to the end.
fn enter(&mut self, phase: Phase) {
self.phase = phase;
self.guard.set_cancellable(phase == Phase::Reading);
self.refresh();
}
/// Bring the activity row up to date: the count while reading, then what is being done.
fn refresh(&self) {
let detail = match (self.phase, self.kind) {
(Phase::Reading, Kind::Inspect) => "",
(Phase::Reading, Kind::Import) => crate::i18n::tr("Reading Lightroom catalog…"),
(Phase::Committing, _) => crate::i18n::tr("Adding Lightroom photos…"),
(Phase::Finalizing, _) => crate::i18n::tr("Saving Lightroom import index…"),
};
if self.phase == Phase::Reading {
self.guard.progress(self.done.load(Ordering::Relaxed) as u64, self.total.load(Ordering::Relaxed) as u64);
} else {
self.guard.progress(0, 0);
}
self.guard.detail(detail);
}
#[cfg(not(target_arch = "wasm32"))]
fn status(&self) -> Value {
let kind = match self.kind {
@ -236,7 +278,7 @@ fn start_inspect(app: &mut LightcraftApp, path: PathBuf, ctx: &egui::Context) ->
let (tx, rx) = mpsc::channel();
spawn_inspect(path.clone(), cancel.clone(), total.clone(), done.clone(), tx, ctx.clone())?;
app.lightroom_last = None;
app.lightroom = Some(LightroomTask { kind: Kind::Inspect, path: path.clone(), cancel, total, done, rx, phase: Phase::Reading, cancelled: false });
app.lightroom = Some(LightroomTask::new(&app.session, Kind::Inspect, path.clone(), cancel, total, done, rx));
Ok(json!({"running": true, "kind": "inspect", "path": path}))
}
@ -250,7 +292,7 @@ fn start_import(app: &mut LightcraftApp, path: PathBuf, update_existing: bool, c
let (tx, rx) = mpsc::channel();
spawn_prepare(job, cancel.clone(), tx, ctx.clone())?;
app.lightroom_last = None;
app.lightroom = Some(LightroomTask { kind: Kind::Import, path: path.clone(), cancel, total, done, rx, phase: Phase::Reading, cancelled: false });
app.lightroom = Some(LightroomTask::new(&app.session, Kind::Import, path.clone(), cancel, total, done, rx));
Ok(json!({"running": true, "kind": "import", "path": path}))
}
@ -296,6 +338,7 @@ pub fn tick(app: &mut LightcraftApp, ctx: &egui::Context) {
let message = match task.rx.try_recv() {
Ok(m) => m,
Err(mpsc::TryRecvError::Empty) => {
task.refresh();
app.lightroom = Some(task);
return;
}
@ -337,7 +380,7 @@ pub fn tick(app: &mut LightcraftApp, ctx: &egui::Context) {
);
return;
}
task.phase = Phase::Committing;
task.enter(Phase::Committing);
let mut finalization = None;
let mut report = None;
let committed = app.session.execute_fn("library.importLightroom", |s| {
@ -347,7 +390,7 @@ pub fn tick(app: &mut LightcraftApp, ctx: &egui::Context) {
Ok(completion.report)
});
let report = report.unwrap_or(Value::Null);
task.phase = Phase::Finalizing;
task.enter(Phase::Finalizing);
let (tx, rx) = mpsc::channel();
let commit_error = committed.as_ref().err().map(ToString::to_string);
let Some(finalization) = finalization else {
@ -409,50 +452,67 @@ pub fn wait_for(_app: &mut LightcraftApp, _ctx: &egui::Context, _timeout: std::t
unsupported_wasm()
}
/// The progress window for inspect/import, with cancellation.
pub fn progress(app: &mut LightcraftApp, ctx: &egui::Context) {
let Some(task) = app.lightroom.as_ref() else { return };
let total = task.total.load(Ordering::Relaxed);
let done = task.done.load(Ordering::Relaxed);
let title = match task.kind {
Kind::Inspect => crate::i18n::tr("Inspecting Lightroom Catalog"),
Kind::Import => crate::i18n::tr("Importing Lightroom Catalog"),
};
let text = match task.phase {
Phase::Reading if total == 0 => crate::i18n::tr("Reading Lightroom catalog…").into(),
Phase::Reading => crate::i18n::tr_format!("Reading Lightroom catalog… {done} of {total}", done = done, total = total),
Phase::Committing => crate::i18n::tr("Adding Lightroom photos…").into(),
Phase::Finalizing => crate::i18n::tr("Saving Lightroom import index…").into(),
};
let mut cancel = false;
let t = Tokens::get(ctx);
egui::Window::new(title).title_bar(false).resizable(false).anchor(Align2::CENTER_BOTTOM, [0.0, -80.0]).fixed_size([360.0, 92.0]).show(
ctx,
|ui| {
ui.label(egui::RichText::new(text).color(t.text));
ui.add(egui::ProgressBar::new(done as f32 / total.max(1) as f32).desired_width(340.0));
let r =
ui.add_enabled(task.phase == Phase::Reading && !task.cancel.load(Ordering::Relaxed), egui::Button::new(crate::i18n::tr("Cancel")));
register(ui.ctx(), "button:lightroomCancel", r.rect);
cancel = r.clicked();
},
);
if cancel {
if let Some(task) = app.lightroom.as_mut() {
task.cancel();
}
ctx.request_repaint();
}
if app.lightroom.is_some() {
ctx.request_repaint_after(std::time::Duration::from_millis(100));
}
}
#[cfg(test)]
mod tests {
use super::inspect_report;
use super::*;
use lightcraft_engine::lightroom_catalog::CatalogImport;
/// A task as `start_import` makes it, fed by a channel the test holds instead of a worker.
fn fake_import(h: &crate::headless::Headless) -> (LightroomTask, mpsc::Sender<Message>, Arc<AtomicBool>, Arc<AtomicUsize>, Arc<AtomicUsize>) {
let (tx, rx) = mpsc::channel();
let (cancel, total, done) = (Arc::new(AtomicBool::new(false)), Arc::new(AtomicUsize::new(0)), Arc::new(AtomicUsize::new(0)));
let task = LightroomTask::new(&h.app.session, Kind::Import, "fixture.lrcat".into(), cancel.clone(), total.clone(), done.clone(), rx);
(task, tx, cancel, total, done)
}
fn demo() -> crate::headless::Headless {
let app = crate::LightcraftApp::new(lightcraft_engine::Session::with_demo(), crate::Services { png: None, ..Default::default() });
let mut h = crate::headless::Headless::new(app, [1200.0, 800.0], 1.0);
// no render left running at exit (see tests_activity::demo)
h.settle(std::time::Duration::from_secs(20));
h
}
#[test]
fn lightroom_row_is_not_cancellable_while_committing() {
let mut h = demo();
let (task, tx, cancel, total, done) = fake_import(&h);
h.app.lightroom = Some(task);
total.store(10, Ordering::Relaxed);
done.store(3, Ordering::Relaxed);
h.step();
let rows = h.app.session.activity.list();
assert_eq!(rows.len(), 1);
assert_eq!((rows[0].kind, rows[0].label.as_str()), ("lightroom", "Importing Lightroom catalog"));
assert_eq!((rows[0].done, rows[0].total), (3, 10));
assert!(rows[0].cancellable, "reading the catalog can stop");
h.app.lightroom.as_mut().unwrap().enter(Phase::Committing);
let row = h.app.session.activity.list().remove(0);
assert!(!row.cancellable, "once photos are being added it runs to the end");
assert!(h.app.session.activity.cancel(row.id).is_err());
assert!(!cancel.load(Ordering::Relaxed));
tx.send(Message::Finalized(Ok(()))).unwrap();
h.step();
assert!(h.app.lightroom.is_none());
assert!(h.app.session.activity.list().is_empty(), "the row goes with the task");
}
#[test]
fn lightroom_cancel_from_the_stack_stops_the_read() {
let mut h = demo();
let (task, tx, cancel, _, _) = fake_import(&h);
h.app.lightroom = Some(task);
h.step();
let id = h.app.session.activity.list()[0].id;
h.app.session.activity.cancel(id).unwrap();
assert!(cancel.load(Ordering::Relaxed), "the worker sees the stack's ✕");
tx.send(Message::Prepared(Err("cancelled".into()))).unwrap();
h.step();
assert!(h.app.lightroom.is_none());
assert_eq!(h.app.lightroom_last, Some(serde_json::json!({"cancelled": true})));
assert!(h.app.session.activity.list().is_empty());
}
#[test]
fn inspect_report_keeps_empty_catalog_shape() {
let report = inspect_report(CatalogImport {

2
crates/ui-egui/src/menus.rs
View File

@ -1212,7 +1212,7 @@ pub fn run_ui_command(app: &mut LightcraftApp, id: &str, p: &Value) -> Option<Re
}
};
app.ui.last_find_missing = None;
if let Err(e) = crate::tasks::spawn(app, LABEL, work, done) {
if let Err(e) = crate::tasks::spawn(app, LABEL, Some("findMissing"), work, done) {
return Some(Err(e));
}
if p.get("wait").and_then(Value::as_bool).unwrap_or(false) {

21
crates/ui-egui/src/merge.rs
View File

@ -4,7 +4,7 @@
//! Previews and the final merge run [`lightcraft_engine::merge::MergeJob`]s on worker threads with
//! progress and cancellation; the window stays responsive. Changing an option cancels the running
//! preview and starts a new one. The final merge keeps running after the dialog closes (progress
//! in a toast); when it finishes, the result is written, imported and selected
//! in the activity stack); when it finishes, the result is written, imported and selected
//! ([`lightcraft_engine::Session::finish_merge`]).
use std::sync::atomic::{AtomicBool, Ordering};
@ -12,6 +12,7 @@ use std::sync::mpsc::{Receiver, channel};
use std::sync::{Arc, Mutex};
use lightcraft_catalog::PhotoId;
use lightcraft_engine::activity::{Cancel, TaskGuard, Unit};
use lightcraft_engine::merge::{MergeJob, MergeOutput};
use serde::{Deserialize, Serialize};
use serde_json::{Value, json};
@ -88,6 +89,8 @@ pub struct MergeTask {
pub progress: Arc<Mutex<(f32, String)>>,
pub cancel: Arc<AtomicBool>,
rx: Receiver<Result<MergeOutput, String>>,
/// The final merge's row in the activity stack (previews have none).
guard: Option<TaskGuard>,
}
/// Merge state of the app.
@ -140,7 +143,7 @@ fn spawn(app: &LightcraftApp, options: &MergeDialog, ids: &[PhotoId], preview: b
std::thread::Builder::new().name("photo-merge".into()).spawn(work).map_err(|e| e.to_string())?;
#[cfg(target_arch = "wasm32")]
work();
Ok(MergeTask { preview, job, options: options.clone(), progress, cancel, rx })
Ok(MergeTask { preview, job, options: options.clone(), progress, cancel, rx, guard: None })
}
/// Open the merge dialog for the selection.
@ -169,7 +172,11 @@ pub fn start_final(app: &mut LightcraftApp, opts: &MergeDialog) -> Result<Value,
t.cancel.store(true, Ordering::Relaxed);
}
let ids = app.merge.ids.clone();
let task = spawn(app, opts, &ids, false)?;
let mut task = spawn(app, opts, &ids, false)?;
let guard = app.session.activity.start("merge", "Merging", Cancel::Flag(task.cancel.clone()));
guard.set_unit(Unit::Percent);
guard.progress(0, 1000);
task.guard = Some(guard);
app.merge.final_task = Some(task);
app.merge.last.insert(opts.command.clone(), opts.clone());
Ok(json!({"started": true, "photos": ids.len()}))
@ -265,6 +272,7 @@ pub fn poll(app: &mut LightcraftApp, ctx: &egui::Context) {
app.merge.last_result = Some(v);
app.toast(ctx, crate::i18n::tr_format!("{what} merge added", what = what));
}
Err(_) if t.cancel.load(Ordering::Relaxed) => app.toast(ctx, crate::i18n::tr_format!("{what} merge cancelled", what = what)),
Err(e) => {
app.ui.status = e.to_string();
app.toast(ctx, crate::i18n::tr_format!("{what} merge failed: {e}", e = e, what = what));
@ -273,8 +281,11 @@ pub fn poll(app: &mut LightcraftApp, ctx: &egui::Context) {
}
Err(_) => {
let (f, stage) = t.progress.lock().map(|g| g.clone()).unwrap_or_default();
let now = ctx.input(|i| i.time);
app.ui.toast = Some((crate::i18n::tr_format!("Merging… {stage} {:.0}%", f * 100.0, stage = stage), now + 0.5, None));
if let Some(guard) = &t.guard {
let f = if f.is_finite() { f.clamp(0.0, 1.0) } else { 0.0 };
guard.progress((f * 1000.0) as u64, 1000);
guard.detail(crate::i18n::tr(&stage));
}
ctx.request_repaint_after(std::time::Duration::from_millis(100));
}
}

165
crates/ui-egui/src/panels/activity.rs Normal file
View File

@ -0,0 +1,165 @@
//! The activity stack (issue #345): every long-running task in one place, floating in the top-left corner under the
//! top bar while something runs — a row each with its name, a progress bar, what it is working on, and ✕ for the ones
//! that can be cancelled. The tasks come from the engine's registry ([`lightcraft_engine::activity`]); ✕ runs
//! `activity.cancel` like any other frontend would.
use egui::{Align, Layout, Margin, Rect, RichText, Sense, Stroke, pos2, vec2};
use lightcraft_engine::activity::{TaskInfo, Unit};
use serde_json::json;
use crate::LightcraftApp;
use crate::i18n::{tr, tr_format};
use crate::icons::{Icon, paint};
use crate::theme::Tokens;
use crate::widgets::register;
/// A task shows once it has run this long, so quick ones don't flash.
pub const SHOW_AFTER_MS: u64 = 500;
/// Rows shown before "+N more".
pub const MAX_ROWS: usize = 3;
/// The stack's width (points).
pub const WIDTH: f32 = 280.0;
const MARGIN: i8 = 8;
const CROSS: f32 = 16.0;
/// "3 of 25", "12 of 340 MB", "40 %"; empty while the amount of work isn't known.
pub fn count_text(t: &TaskInfo) -> String {
if t.total == 0 {
return String::new();
}
match t.unit {
Unit::Count => tr_format!("{done} of {total}", done = t.done, total = t.total),
Unit::Bytes => tr_format!("{done} of {total} MB", done = megabytes(t.done), total = megabytes(t.total)),
Unit::Percent => tr_format!("{pct} %", pct = t.done.saturating_mul(100) / t.total),
}
}
fn megabytes(bytes: u64) -> String {
let mb = bytes as f64 / 1_048_576.0;
if mb < 10.0 { format!("{mb:.1}") } else { format!("{:.0}", mb.floor()) }
}
/// The row's second line: "Stopping…" once cancelled, else the count and what is being worked on.
fn detail_line(t: &TaskInfo) -> String {
if t.cancelling {
return tr("Stopping…").to_string();
}
let count = count_text(t);
match (count.is_empty(), t.detail.is_empty()) {
(false, false) => format!("{count} · {}", t.detail),
(false, true) => count,
(true, _) => t.detail.clone(),
}
}
/// Draw the stack (called every frame, after the panels and dialogs).
pub fn show(app: &mut LightcraftApp, ctx: &egui::Context) {
let all = app.session.activity.list();
if all.is_empty() {
app.activity_expanded = false;
return;
}
// look again soon: a young task becomes due, bars move
ctx.request_repaint_after(std::time::Duration::from_millis(100));
let tasks: Vec<&TaskInfo> = all.iter().filter(|t| t.age_ms >= SHOW_AFTER_MS).collect();
if tasks.is_empty() {
return;
}
let t = Tokens::get(ctx);
let shown = if app.activity_expanded { tasks.len() } else { tasks.len().min(MAX_ROWS) };
let hidden = tasks.len().saturating_sub(shown);
let max_height = (ctx.content_rect().height() - t.top_bar_h - 60.0).max(80.0);
let mut cancel = None;
let mut toggle = false;
let frame = egui::Frame::NONE.fill(t.chrome).stroke(Stroke::new(1.0, t.field_border)).corner_radius(6).inner_margin(Margin::same(MARGIN));
egui::Area::new(egui::Id::new("activity-stack")).order(egui::Order::Middle).fixed_pos(pos2(8.0, t.top_bar_h + 8.0)).show(ctx, |ui| {
// An area lays out in the size it had last frame, and the scroll area would take that as all the room there
// is: the stack would never grow when rows are added. Give it the whole height below the top bar instead
// (the scroll area keeps to `max_height`).
ui.set_max_height((ctx.content_rect().height() - t.top_bar_h - 16.0).max(80.0));
frame.show(ui, |ui| {
let inner = WIDTH - 2.0 * (f32::from(MARGIN) + 1.0);
ui.set_width(inner);
ui.spacing_mut().item_spacing.y = 3.0;
egui::ScrollArea::vertical().max_height(max_height).auto_shrink([false, true]).show(ui, |ui| {
for (i, task) in tasks.iter().take(shown).enumerate() {
if i > 0 {
ui.add_space(6.0);
}
if row(ui, &t, task, inner) {
cancel = Some(task.id);
}
}
});
if hidden > 0 || (app.activity_expanded && tasks.len() > MAX_ROWS) {
ui.add_space(4.0);
let text = if hidden > 0 { tr_format!("+{n} more", n = hidden) } else { tr("Show less").to_string() };
let r = ui.add(egui::Label::new(RichText::new(text).font(t.font(11.0)).color(t.accent)).sense(Sense::click()));
register(ui.ctx(), "activity:more", r.rect);
toggle = r.clicked();
}
});
});
if let Some(id) = cancel {
self::cancel(app, id);
}
if toggle {
app.activity_expanded = !app.activity_expanded;
}
}
/// ✕ on task `id`'s row. The task may have just finished or stopped being cancellable: nothing to tell the user then,
/// so the command goes to the engine directly (`LightcraftApp::run` would put its error in the status bar).
pub(crate) fn cancel(app: &mut LightcraftApp, id: u64) {
let _ = app.session.execute("activity.cancel", &json!({"id": id}));
}
/// One task: name and ✕, the bar, the detail line. True when ✕ was clicked.
fn row(ui: &mut egui::Ui, t: &Tokens, task: &TaskInfo, width: f32) -> bool {
let mut clicked = false;
let r = ui.vertical(|ui| {
ui.set_width(width);
ui.horizontal(|ui| {
let cross = task.cancellable && !task.cancelling;
let label_w = if cross { width - CROSS - 6.0 } else { width };
ui.allocate_ui_with_layout(vec2(label_w, 18.0), Layout::left_to_right(Align::Center), |ui| {
// the whole width, so the cross lands at the right edge however short the name
ui.set_width(label_w);
ui.add(egui::Label::new(RichText::new(tr(&task.label)).font(t.font(12.5)).color(t.text)).truncate());
});
if cross {
let (rect, resp) = ui.allocate_exact_size(vec2(CROSS, CROSS), Sense::click());
paint(ui.painter(), rect.shrink(2.0), Icon::Close, if resp.hovered() { t.text } else { t.icon });
register(ui.ctx(), format!("activity:cancel:{}", task.id), rect);
clicked = resp.on_hover_text(tr("Cancel")).clicked();
}
});
bar(ui, t, task, width);
let line = detail_line(task);
if !line.is_empty() {
ui.add(egui::Label::new(RichText::new(line).font(t.font(11.0)).color(t.text_dim)).truncate());
}
});
register(ui.ctx(), format!("activity:row:{}", task.id), r.response.rect);
clicked
}
/// The progress bar; while the amount of work isn't known, a segment sweeps across.
fn bar(ui: &mut egui::Ui, t: &Tokens, task: &TaskInfo, width: f32) {
let (rect, _) = ui.allocate_exact_size(vec2(width, 5.0), Sense::hover());
let p = ui.painter();
p.rect_filled(rect, 2.5, t.field);
let fill = if task.total > 0 {
let frac = (task.done as f64 / task.total as f64).clamp(0.0, 1.0) as f32;
Rect::from_min_size(rect.min, vec2(rect.width() * frac, rect.height()))
} else {
let phase = (ui.input(|i| i.time) * 0.7).fract() as f32;
let w = rect.width() * 0.3;
let x = rect.left() + (rect.width() + w) * phase - w;
Rect::from_min_max(pos2(x.max(rect.left()), rect.top()), pos2((x + w).min(rect.right()), rect.bottom()))
};
if fill.width() > 0.0 {
p.rect_filled(fill, 2.5, if task.cancelling { t.text_dim } else { t.accent });
}
}

5
crates/ui-egui/src/panels/faces.rs
View File

@ -669,8 +669,9 @@ pub fn pump(app: &mut LightcraftApp, ctx: &egui::Context) {
app.caches.faces_active = v["active"] == true;
app.caches.faces_indexed = v["indexedFaces"].as_u64().unwrap_or(0);
app.caches.faces_pending = v["pendingPhotos"].as_u64().unwrap_or(0);
// the progress bar's whole: the most photos that were left at once since the scan last finished
app.caches.faces_peak = if app.caches.faces_pending == 0 { 0 } else { app.caches.faces_peak.max(app.caches.faces_pending) };
// the progress bar's whole: the most photos that were left at once since the scan last finished (the engine keeps
// it, so Settings and the activity stack agree)
app.caches.faces_peak = v["peak"].as_u64().unwrap_or(0);
// Frames are drawn only when something asks for one, so nothing here wakes the window needlessly: while there is work it
// asks to be called again in 50 ms (the progress in Settings, the next photos for the workers); with recognition on and
// nothing to do it looks again every few seconds (a few wake-ups a minute); with recognition off it asks for nothing

1
crates/ui-egui/src/panels/mod.rs
View File

@ -1,5 +1,6 @@
//! Window regions and panels.
pub mod activity;
pub mod bottombar;
pub mod chips;
pub mod compare;

44
crates/ui-egui/src/panels/notices.rs
View File

@ -1,12 +1,12 @@
//! Things the user must not miss (issue #103): settings files that were damaged or unreadable when
//! the library opened (and host warnings such as a damaged `ui.json`), and quitting while changes
//! are only in memory.
//! are only in memory or while tasks that would be cut short are running (issue #345).
use egui::{Align2, RichText, vec2};
use crate::LightcraftApp;
use crate::theme::Tokens;
use crate::widgets::register;
use crate::{LightcraftApp, QuitPrompt};
/// Collect the library's settings-file warnings (each is shown once).
pub fn logic(app: &mut LightcraftApp) {
@ -14,10 +14,25 @@ pub fn logic(app: &mut LightcraftApp) {
app.notices.extend(new);
}
/// The window may close now (`true`), or the quit prompt is shown (`false`): changes that
/// couldn't be written are retried first — the log, then a snapshot of everything in memory.
/// The window may close now (`true`), or the quit prompt is shown (`false`). Running tasks that
/// can be cancelled (an import, an export…) are asked about first; then changes that couldn't be
/// written are retried — the log, then a snapshot of everything in memory.
pub fn may_close(app: &mut LightcraftApp) -> bool {
if app.quit_confirmed || app.session.unsaved().is_none() {
if app.quit_confirmed {
return true;
}
// before the library is touched: a Cancel here keeps working with it
let running = app.session.activity.running_cancellable();
if !running.is_empty() {
let names = running.iter().map(|t| crate::i18n::tr(&t.label)).collect::<Vec<_>>().join(", ");
app.quit_prompt = Some(QuitPrompt::Tasks(if running.len() == 1 {
crate::i18n::tr_format!("1 task is still running: {names}. Quitting stops it.", names = names)
} else {
crate::i18n::tr_format!("{n} tasks are still running: {names}. Quitting stops them.", n = running.len(), names = names)
}));
return false;
}
if app.session.unsaved().is_none() {
return true;
}
let _ = app.session.persist();
@ -29,13 +44,13 @@ pub fn may_close(app: &mut LightcraftApp) -> bool {
match app.session.unsaved() {
None => true,
Some((n, e)) => {
app.quit_prompt = Some(crate::i18n::tr_format!(
app.quit_prompt = Some(QuitPrompt::Unsaved(crate::i18n::tr_format!(
"{n} change{} couldn't be written to disk: {e}\nIf you quit now, {} lost.",
if n == 1 { "" } else { "s" },
if n == 1 { "it is" } else { "they are" },
n = n,
e = e
));
)));
false
}
}
@ -85,7 +100,20 @@ fn window(ctx: &egui::Context, id: &str, title: &str, text: &str, buttons: &[(&s
}
pub fn show(app: &mut LightcraftApp, ctx: &egui::Context) {
if let Some(text) = app.quit_prompt.clone() {
if let Some(QuitPrompt::Tasks(text)) = app.quit_prompt.clone() {
let buttons = [("quitAnyway", "Quit Anyway", Choice::QuitAnyway), ("quitCancel", "Cancel", Choice::Cancel)];
let Some(i) = window(ctx, "quit-tasks", "Quit while tasks are running?", &text, &buttons) else { return };
app.quit_prompt = None;
if matches!(buttons[i].2, Choice::QuitAnyway) {
app.session.activity.cancel_all();
// unsaved changes still get their own question
if may_close(app) {
app.ui.quit = true;
}
}
return;
}
if let Some(QuitPrompt::Unsaved(text)) = app.quit_prompt.clone() {
let buttons = [
("quitRetry", "Try Saving Again", Choice::Retry),
("quitAnyway", "Quit Anyway", Choice::QuitAnyway),

58
crates/ui-egui/src/sync.rs
View File

@ -1,11 +1,13 @@
//! Synchronize Folder in the app: the folder is scanned on a worker thread (a network share can
//! take minutes; no frame waits for it) while its dialog is open, and the dialog then says what
//! changed and what to do about it. Synchronize runs `folder.synchronize`, which acts on exactly
//! the scan the dialog showed (see `lightcraft_engine::sync`).
//! the scan the dialog showed (see `lightcraft_engine::sync`). Both the scan and the run show a row in the activity
//! stack (issue #345); its ✕ stops them like the dialog's Cancel.
use std::sync::Arc;
use std::sync::atomic::Ordering;
use lightcraft_engine::activity::{Cancel, TaskGuard};
use lightcraft_engine::sync::{FolderChanges, SyncChoice, SyncCommit, SyncInput, SyncJob, SyncProgress, SyncStep, scan_with};
use serde_json::{Value, json};
@ -18,6 +20,8 @@ pub struct SyncTask {
path: String,
progress: Arc<SyncProgress>,
rx: std::sync::mpsc::Receiver<FolderChanges>,
/// Its row in the activity stack; ✕ cancels the scan and closes the dialog.
guard: TaskGuard,
}
/// Open the dialog for folder `path` (called `name` in it) and start scanning.
@ -34,7 +38,9 @@ pub fn open(app: &mut LightcraftApp, path: &str, name: &str, disk: bool) -> Resu
std::thread::spawn(job);
#[cfg(target_arch = "wasm32")]
job();
app.sync = Some(SyncTask { path: path.to_string(), progress, rx });
let guard = app.session.activity.start("sync", "Looking for changes", Cancel::Flag(progress.files.cancel.clone()));
guard.detail(name);
app.sync = Some(SyncTask { path: path.to_string(), progress, rx, guard });
let d = SyncChoice::default();
app.ui.dialog = Some(Dialog::SynchronizeFolder {
path: path.to_string(),
@ -78,6 +84,14 @@ pub fn poll(app: &mut LightcraftApp, ctx: &egui::Context) {
cancel(app);
return;
}
if task.guard.is_cancelled() {
// ✕ on its row: a scan stopped part-way has nothing to show
cancel(app);
app.ui.dialog = None;
return;
}
let (done, total) = task.progress.counts();
task.guard.progress(done as u64, total as u64);
ctx.request_repaint_after(std::time::Duration::from_millis(100));
let changes = match task.rx.try_recv() {
Ok(c) => c,
@ -210,7 +224,9 @@ pub fn confirm(app: &mut LightcraftApp, dlg: &Dialog) -> Result<Value, String> {
std::thread::spawn(job);
#[cfg(target_arch = "wasm32")]
job();
app.sync_run = Some(SyncRun { name: name.clone(), progress, crashed, rx, commit: Some(commit), stopping: false, not_saved: Vec::new() });
let guard = app.session.activity.start("sync", "Synchronizing folder", Cancel::Flag(progress.files.cancel.clone()));
guard.detail(name);
app.sync_run = Some(SyncRun { name: name.clone(), progress, crashed, rx, commit: Some(commit), stopping: false, not_saved: Vec::new(), guard });
Ok(json!({"started": true}))
}
@ -227,6 +243,8 @@ pub struct SyncRun {
commit: Option<SyncCommit>,
/// Cancel was pressed: nothing new is started; what was readied is still committed.
stopping: bool,
/// Its row in the activity stack (its ✕ is Cancel).
guard: TaskGuard,
}
impl SyncRun {
@ -253,6 +271,11 @@ impl Drop for SyncRun {
pub fn poll_run(app: &mut LightcraftApp, ctx: &egui::Context) {
let Some(run) = app.sync_run.as_mut() else { return };
ctx.request_repaint_after(std::time::Duration::from_millis(100));
if run.guard.is_cancelled() {
run.stopping = true;
}
let (done, total) = run.counts();
run.guard.progress(done as u64, total as u64);
let mut steps = Vec::new();
let finished = loop {
match run.rx.try_recv() {
@ -317,32 +340,3 @@ pub fn poll_run(app: &mut LightcraftApp, ctx: &egui::Context) {
app.toast_for(ctx, format!("{name}: {text}"), 12.0);
}
}
/// The progress window while a synchronize runs, with Cancel (what was done stays, as one undo
/// step; nothing new is started).
pub fn progress_window(app: &mut LightcraftApp, ctx: &egui::Context) {
let Some(run) = &app.sync_run else { return };
let t = Tokens::get(ctx);
let (done, total) = run.counts();
let text =
if run.stopping { crate::i18n::tr("Stopping…").to_string() } else { format!("{} {done} / {total}", crate::i18n::tr("Synchronizing…")) };
let frac = if total == 0 { 0.0 } else { done as f32 / total as f32 };
let stopping = run.stopping;
let mut cancel = false;
egui::Window::new(crate::i18n::tr("Synchronize Folder"))
.id(egui::Id::new("sync-progress"))
.title_bar(false)
.resizable(false)
.anchor(egui::Align2::CENTER_BOTTOM, [0.0, -80.0])
.fixed_size([340.0, 80.0])
.show(ctx, |ui| {
ui.label(egui::RichText::new(text).color(t.text));
ui.add(egui::ProgressBar::new(frac).desired_width(320.0));
let r = ui.add_enabled(!stopping, egui::Button::new(crate::i18n::tr("Cancel")));
crate::widgets::register(ui.ctx(), "button:syncCancel", r.rect);
cancel = r.clicked();
});
if cancel && let Some(run) = app.sync_run.as_mut() {
run.cancel();
}
}

12
crates/ui-egui/src/tasks.rs
View File

@ -1,7 +1,8 @@
//! Background tasks for commands whose file-system work can take long on a slow or offline drive
//! (Find Missing Photos walking a folder, listing the auto-import folder…): the work runs on a
//! worker thread and its result is applied on the UI thread between frames, so the window keeps
//! answering. Imports, scans, exports and preview builds have their own tasks with progress.
//! answering. Imports, scans, exports and preview builds have their own tasks with progress. A task the user started
//! (Find Missing Photos) shows in the activity stack while it runs; quiet ones (the auto-import listing) don't.
use std::sync::mpsc::{Receiver, TryRecvError, channel};
@ -13,6 +14,8 @@ type Finish = Box<dyn FnOnce(&mut LightcraftApp, &egui::Context) + Send>;
struct Task {
label: String,
rx: Receiver<Finish>,
/// Its row in the activity stack (none for quiet tasks); goes with the task.
_guard: Option<lightcraft_engine::activity::TaskGuard>,
}
/// The background tasks in flight.
@ -39,10 +42,12 @@ impl Tasks {
}
/// Run `work` on a worker thread, then `done(app, ctx, result)` on the UI thread (in the browser
/// build, which has no threads, `work` runs at once and `done` on the next frame).
/// build, which has no threads, `work` runs at once and `done` on the next frame). With `kind`, the
/// activity stack shows `label` meanwhile, without a count and without ✕ (the work can't be stopped).
pub fn spawn<T: Send + 'static>(
app: &mut LightcraftApp,
label: &str,
kind: Option<&'static str>,
work: impl FnOnce() -> T + Send + 'static,
done: impl FnOnce(&mut LightcraftApp, &egui::Context, T) + Send + 'static,
) -> Result<(), String> {
@ -59,7 +64,8 @@ pub fn spawn<T: Send + 'static>(
std::thread::Builder::new().name(format!("lc-task-{label}")).spawn(job).map_err(|e| format!("{label}: could not start: {e}"))?;
#[cfg(target_arch = "wasm32")]
job();
app.tasks.running.push(Task { label: label.to_string(), rx });
let guard = kind.map(|k| app.session.activity.start(k, label, lightcraft_engine::activity::Cancel::No));
app.tasks.running.push(Task { label: label.to_string(), rx, _guard: guard });
Ok(())
}

395
crates/ui-egui/src/tests_activity.rs Normal file
View File

@ -0,0 +1,395 @@
//! Issue #345: the activity stack — one place, top-left, for every long-running task, with ✕ to cancel.
use std::time::Duration;
use lightcraft_engine::activity::{Cancel, TaskInfo, Unit};
use serde_json::json;
use crate::headless::Headless;
use crate::panels::activity::{WIDTH, count_text};
use crate::{LightcraftApp, Services};
const T: Duration = Duration::from_secs(20);
fn demo() -> Headless {
let app = LightcraftApp::new(lightcraft_engine::Session::with_demo(), Services { png: None, ..Default::default() });
let mut h = Headless::new(app, [1200.0, 800.0], 1.0);
// let the first renders finish: a render still running when the test binary exits can crash in the
// GPU driver's teardown (the preview pool doesn't join its workers)
h.settle(T);
h
}
fn has(h: &Headless, id: &str) -> bool {
h.app.widgets.iter().any(|(w, _)| w == id)
}
fn rows(h: &Headless) -> usize {
h.app.widgets.iter().filter(|(w, _)| w.starts_with("activity:row:")).count()
}
fn rect(h: &Headless, id: &str) -> egui::Rect {
h.app.widgets.iter().find(|(w, _)| w == id).map(|(_, r)| *r).unwrap_or_else(|| panic!("no widget {id}"))
}
/// Rows show once their task is half a second old.
fn wait_visible(h: &mut Headless) {
std::thread::sleep(Duration::from_millis(600));
h.step();
h.step();
}
/// Click a widget and let the synthetic events (move, press, release) play out.
fn click(h: &mut Headless, id: &str) {
h.step();
h.step();
assert_eq!(h.request("ui.clickWidget", json!({"id": id}), T)["ok"], true, "{id}");
for _ in 0..4 {
h.step();
}
}
#[test]
fn rows_appear_after_half_a_second_and_cancel_by_click() {
let mut h = demo();
let g = h.app.session.activity.start("export", "Exporting", Cancel::Yes);
g.progress(2, 5);
h.step();
// a frame on a loaded machine can outlast the half second: only judge a task still young after it
if h.app.session.activity.list()[0].age_ms < crate::panels::activity::SHOW_AFTER_MS {
assert!(!has(&h, &format!("activity:row:{}", g.id())), "not before 0.5 s");
}
wait_visible(&mut h);
assert!(has(&h, &format!("activity:row:{}", g.id())));
click(&mut h, &format!("activity:cancel:{}", g.id()));
assert!(g.is_cancelled());
let inspect = h.request("ui.inspect", json!({}), T);
assert_eq!(inspect["result"]["activity"][0]["cancelling"], true, "{}", inspect["result"]["activity"]);
}
#[test]
fn more_than_three_rows_overflow_and_expand() {
let mut h = demo();
let guards: Vec<_> = (0..5).map(|i| h.app.session.activity.start("export", &format!("Task {i}"), Cancel::Yes)).collect();
wait_visible(&mut h);
assert_eq!(rows(&h), 3);
assert!(has(&h, "activity:more"));
click(&mut h, "activity:more");
assert_eq!(rows(&h), 5);
// expanded, the stack grows to show them all
let stack = h.view.ctx.memory(|m| m.area_rect(egui::Id::new("activity-stack"))).expect("the stack is drawn");
for g in &guards {
let r = rect(&h, &format!("activity:row:{}", g.id()));
assert!(stack.contains_rect(r), "row {} at {r:?} is cut off by the stack at {stack:?}", g.id());
}
drop(guards);
h.step();
assert_eq!(rows(&h), 0);
}
#[test]
fn not_cancellable_rows_have_no_cross() {
let mut h = demo();
let g = h.app.session.activity.start("faces", "Finding faces", Cancel::No);
wait_visible(&mut h);
assert!(has(&h, &format!("activity:row:{}", g.id())));
assert!(!has(&h, &format!("activity:cancel:{}", g.id())));
}
#[test]
fn count_text_by_unit() {
let mut t = TaskInfo {
id: 1,
kind: "export",
label: "Exporting".into(),
done: 3,
total: 25,
unit: Unit::Count,
detail: String::new(),
cancellable: true,
cancelling: false,
age_ms: 600,
};
assert_eq!(count_text(&t), "3 of 25");
(t.done, t.total, t.unit) = (12 * 1_048_576, 340 * 1_048_576, Unit::Bytes);
assert_eq!(count_text(&t), "12 of 340 MB");
(t.done, t.total, t.unit) = (400, 1000, Unit::Percent);
assert_eq!(count_text(&t), "40 %");
t.total = 0;
assert_eq!(count_text(&t), "");
}
#[test]
fn stale_cancel_is_harmless() {
let mut h = demo();
let g = h.app.session.activity.start("export", "Exporting", Cancel::Yes);
wait_visible(&mut h);
let id = g.id();
drop(g);
let r = h.request("engine.execute", json!({"command": "activity.cancel", "params": {"id": id}}), T);
assert_eq!(r["ok"], false, "an error result, not a panic: {r}");
h.step();
assert!(h.app.ui.toast.is_none(), "{:?}", h.app.ui.toast);
// the same through the row's ✕ (clicked in the frame the task ended): no message in the status bar either
h.app.ui.status = "Ready".into();
crate::panels::activity::cancel(&mut h.app, id);
h.step();
assert_eq!(h.app.ui.status, "Ready");
assert!(h.app.ui.toast.is_none(), "{:?}", h.app.ui.toast);
}
#[test]
fn long_detail_stays_inside_the_stack() {
let mut h = demo();
let g = h.app.session.activity.start("export", "Exportieren", Cancel::Yes);
g.detail(&"IMG_".repeat(75));
wait_visible(&mut h);
let row = rect(&h, &format!("activity:row:{}", g.id()));
assert!(row.width() <= WIDTH + 1.0, "{row:?}");
}
#[test]
fn cross_sits_at_the_right_edge_of_its_row() {
let mut h = demo();
let g = h.app.session.activity.start("export", "Exporting", Cancel::Yes);
wait_visible(&mut h);
let row = rect(&h, &format!("activity:row:{}", g.id()));
let cross = rect(&h, &format!("activity:cancel:{}", g.id()));
assert!(cross.right() >= row.right() - 2.0, "the cross is at the right, not after the label: {cross:?} in {row:?}");
}
#[test]
fn quit_with_a_running_task_asks_and_quit_anyway_cancels() {
let mut h = demo();
let g = h.app.session.activity.start("export", "Exporting", Cancel::Yes);
wait_visible(&mut h);
assert!(!crate::panels::notices::may_close(&mut h.app));
assert!(matches!(h.app.quit_prompt, Some(crate::QuitPrompt::Tasks(_))), "{:?}", h.app.quit_prompt);
h.step();
assert!(!has(&h, "button:quitRetry"));
click(&mut h, "button:quitAnyway");
assert!(g.is_cancelled() && h.quit_requested());
}
#[test]
fn quit_cancel_keeps_the_task_running() {
let mut h = demo();
let g = h.app.session.activity.start("import", "Importing", Cancel::Yes);
assert!(!crate::panels::notices::may_close(&mut h.app));
click(&mut h, "button:quitCancel");
assert!(!g.is_cancelled() && h.app.quit_prompt.is_none() && !h.app.quit_confirmed);
}
#[test]
fn cancelling_task_does_not_block_quit() {
let mut h = demo();
let g = h.app.session.activity.start("export", "Exporting", Cancel::Yes);
h.app.session.activity.cancel(g.id()).unwrap();
let _f = h.app.session.activity.start("faces", "Finding faces", Cancel::No);
assert!(crate::panels::notices::may_close(&mut h.app));
}
/// A slow disk: every file takes `ms` to write (the UI must not wait for it).
fn slow_writer(ms: u64) -> crate::SharedWrite {
std::sync::Arc::new(move |_: &str, _: &[u8]| {
std::thread::sleep(Duration::from_millis(ms));
Ok(())
})
}
/// Export the first `n` photos in view through the Export dialog (small JPEGs, a made-up folder).
fn start_export(h: &mut Headless, n: usize) {
let ids: Vec<u64> = h.app.session.visible_cloned().iter().take(n).map(|p| p.0).collect();
assert_eq!(ids.len(), n, "the demo library has {n} photos");
h.request("engine.execute", json!({"command": "library.select", "params": {"ids": ids}}), T);
h.request("engine.execute", json!({"command": "dialog.export", "params": {}}), T);
h.step();
if let Some(crate::state::Dialog::Export { full_size, resize, dir, .. }) = &mut h.app.ui.dialog {
*full_size = false;
*resize = lightcraft_engine::export::Resize::long_edge(64);
*dir = "/lc-test-out".into();
}
let r = h.request("ui.dialog.confirm", json!({}), T);
assert_eq!(r["ok"], true, "{r}");
}
#[test]
fn export_shows_a_row_and_stops_on_activity_cancel() {
let mut h = demo();
h.app.services.write_shared = Some(slow_writer(200));
start_export(&mut h, 5);
assert!(h.app.export.is_some(), "running in the background");
let tasks = h.request("ui.inspect", json!({}), T)["result"]["activity"].clone();
assert_eq!(tasks[0]["kind"], "export", "{tasks}");
assert_eq!(tasks[0]["total"], 5, "{tasks}");
let id = tasks[0]["id"].as_u64().unwrap();
assert_eq!(h.request("engine.execute", json!({"command": "activity.cancel", "params": {"id": id}}), T)["ok"], true);
assert!(h.step_until(Duration::from_secs(60), |h| h.app.export.is_none()));
assert!(h.app.session.activity.list().is_empty());
assert!(h.app.last_export_result.as_ref().is_some_and(|r| r["cancelled"] == true), "{:?}", h.app.last_export_result);
assert!(!has(&h, "button:exportCancel"), "the old window is gone");
}
#[test]
fn dead_export_worker_leaves_no_row() {
let mut h = demo();
h.app.services.write_shared = Some(std::sync::Arc::new(|_: &str, _: &[u8]| -> Result<(), String> { panic!("synthetic writer panic") }));
start_export(&mut h, 2);
assert!(h.step_until(Duration::from_secs(60), |h| h.app.export.is_none()));
assert!(h.app.session.activity.list().is_empty());
}
/// A folder of `n` stand-in JPEGs and an app whose file probe takes 100 ms each (a slow drive).
fn slow_import(tag: &str, n: usize) -> (Headless, std::path::PathBuf) {
let dir = std::env::temp_dir().join(format!("lc-activity-{tag}-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&dir);
std::fs::create_dir_all(&dir).unwrap();
for i in 0..n {
std::fs::write(dir.join(format!("IMG_{i:03}.jpg")), format!("not really a jpeg {i}")).unwrap();
}
let mut s = lightcraft_engine::Session::new();
s.media.file_probe = Some(std::sync::Arc::new(|p: &str| {
std::thread::sleep(Duration::from_millis(100));
Ok(lightcraft_engine::media::ProbeInfo {
width: 60,
height: 40,
format: "JPEG".into(),
content_hash: Some(p.to_string()),
..Default::default()
})
}));
let mut h = Headless::new(LightcraftApp::new(s, Services { png: None, ..Default::default() }), [1200.0, 800.0], 1.0);
h.settle(T);
(h, dir)
}
#[test]
fn import_cancelled_by_command_reports_cancelled() {
let n = 40;
let (mut h, dir) = slow_import("cancel", n);
let undo0 = h.app.session.undo.len();
crate::import::start_paths(&mut h.app, vec![dir.to_string_lossy().to_string()]).unwrap();
assert!(h.step_until(Duration::from_secs(60), |h| h.app.import.as_ref().is_some_and(|t| t.imported >= 2)));
let tasks = h.app.session.activity.list();
assert_eq!(tasks.first().map(|t| t.kind), Some("import"), "{tasks:?}");
h.app.session.activity.cancel(tasks[0].id).unwrap();
assert!(h.step_until(Duration::from_secs(60), |h| h.app.import.is_none()));
assert!(h.app.session.activity.list().is_empty());
assert!(h.app.ui.toast.as_ref().is_some_and(|t| t.0.starts_with("Import cancelled")), "{:?}", h.app.ui.toast);
assert!(h.app.session.catalog.len() < n, "stopped part-way");
assert_eq!(h.app.session.undo.len(), undo0 + 1, "one undo step");
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
fn scan_shows_a_row_and_cancel_closes_it() {
let (mut h, dir) = slow_import("scan", 120);
crate::import::open(&mut h.app, vec![dir.to_string_lossy().to_string()]).unwrap();
h.step();
let tasks = h.app.session.activity.list();
assert_eq!(tasks.first().map(|t| t.kind), Some("scan"), "{tasks:?}");
assert!(tasks[0].cancellable);
h.app.session.activity.cancel(tasks[0].id).unwrap();
// like the old Cancel: the scan is dropped at once, without waiting for the file being read
assert!(h.step_until(Duration::from_secs(10), |h| h.app.scan.is_none()));
assert!(h.app.session.activity.list().is_empty());
for _ in 0..3 {
h.step();
}
assert!(h.app.ui.dialog.is_none(), "no review opened");
let _ = std::fs::remove_dir_all(&dir);
}
/// A library of three bracketed DNGs (−2, 0, +2 EV) on disk, all selected.
fn bracket(tag: &str) -> (Headless, std::path::PathBuf) {
let dir = std::env::temp_dir().join(format!("lc-activity-{tag}-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&dir);
std::fs::create_dir_all(&dir).unwrap();
let mut paths = Vec::new();
for (i, bytes) in lightcraft_merge::synth::bracket_dngs(1600, 1200, &[-2.0, 0.0, 2.0]).unwrap().into_iter().enumerate() {
let p = dir.join(format!("IMG_{i}.dng"));
std::fs::write(&p, bytes).unwrap();
paths.push(p.to_string_lossy().to_string());
}
let mut session = lightcraft_engine::Session::new().with_fs();
let r = session.execute("library.import", &json!({"paths": paths})).unwrap();
let ids: Vec<u64> = r["imported"].as_array().unwrap().iter().map(|v| v.as_u64().unwrap()).collect();
session.execute("library.select", &json!({"ids": ids})).unwrap();
let app = LightcraftApp::new(session, Services { png: None, ..Default::default() });
let mut h = Headless::new(app, [1200.0, 800.0], 1.0);
h.settle(T);
(h, dir)
}
#[test]
fn final_merge_shows_a_row_and_cancels() {
let (mut h, dir) = bracket("merge");
let before = h.app.session.catalog.len();
crate::merge::start_last(&mut h.app, "merge.hdr").unwrap();
h.step();
let rows = h.app.session.activity.list();
assert_eq!(rows.len(), 1);
assert_eq!((rows[0].kind, rows[0].label.as_str(), rows[0].unit), ("merge", "Merging", Unit::Percent));
assert!(rows[0].cancellable);
assert_eq!(rows[0].total, 1000);
h.app.session.activity.cancel(rows[0].id).unwrap();
assert!(h.step_until(T, |h| h.app.merge.final_task.is_none()));
assert!(h.app.merge.last_result.is_none(), "nothing merged");
assert_eq!(h.app.session.catalog.len(), before, "no photo added");
assert!(h.app.session.activity.list().is_empty());
let toast = h.app.ui.toast.as_ref().map(|t| t.0.clone()).unwrap_or_default();
assert_eq!(toast, "HDR merge cancelled");
let _ = std::fs::remove_dir_all(dir);
}
#[test]
fn find_missing_shows_an_indeterminate_row() {
let mut h = demo();
let dir = std::env::temp_dir().join(format!("lc-activity-findmissing-{}", std::process::id()));
std::fs::create_dir_all(&dir).unwrap();
// run directly, not through a request (which runs frames): the search of an empty folder is over at once, but its
// result is applied between frames, and the row is there until then
let r = crate::menus::run_ui_command(&mut h.app, "file.findMissing", &json!({"folder": dir.to_string_lossy()})).unwrap().unwrap();
assert_eq!(r["background"], true, "{r}");
let rows = h.app.session.activity.list();
assert_eq!(rows.len(), 1, "{rows:?}");
assert_eq!((rows[0].kind, rows[0].label.as_str()), ("findMissing", "Find Missing Photos"));
assert_eq!((rows[0].total, rows[0].cancellable), (0, false), "indeterminate, no ✕");
assert!(h.step_until(T, |h| h.app.tasks.is_empty()));
assert!(h.app.session.activity.list().is_empty(), "the row goes with the task");
let _ = std::fs::remove_dir_all(dir);
}
#[test]
fn auto_import_listing_has_no_row() {
let mut h = demo();
let dir = std::env::temp_dir().join(format!("lc-activity-autoimport-{}", std::process::id()));
std::fs::create_dir_all(&dir).unwrap();
h.app.session.import_defaults.auto_folder = Some(dir.to_string_lossy().to_string());
// the watched folder is listed every few seconds, quietly
assert!(h.step_until(Duration::from_secs(10), |h| h.app.tasks.is_running("Auto Import")));
assert!(h.app.session.activity.list().is_empty(), "{:?}", h.app.session.activity.list());
h.app.session.import_defaults.auto_folder = None;
let _ = std::fs::remove_dir_all(dir);
}
#[test]
fn rows_added_later_grow_the_stack() {
let mut h = demo();
let a = h.app.session.activity.start("export", "Exporting", Cancel::Yes);
a.progress(1, 24);
a.detail("LC01347.jpg");
wait_visible(&mut h);
// two more tasks while the stack already shows one row: it grows to show them, nothing is cut off
let b = h.app.session.activity.start("previews", "Building previews", Cancel::Yes);
let c = h.app.session.activity.start("lightroom", "Reading Lightroom catalog", Cancel::Yes);
wait_visible(&mut h);
for _ in 0..5 {
h.step();
}
let stack = h.view.ctx.memory(|m| m.area_rect(egui::Id::new("activity-stack"))).expect("the stack is drawn");
for g in [&a, &b, &c] {
let r = rect(&h, &format!("activity:row:{}", g.id()));
assert!(stack.contains_rect(r), "row {} at {r:?} is cut off by the stack at {stack:?}", g.id());
}
}

10
crates/ui-egui/src/tests_offline.rs
View File

@ -123,7 +123,8 @@ fn import_reads_files_off_the_ui_thread_and_can_be_cancelled() {
let dir = std::env::temp_dir().join(format!("lc-ui-slow-import-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&dir);
std::fs::create_dir_all(&dir).unwrap();
let n = 40;
// enough files that the import is still running when its activity row shows (after 0.5 s)
let n = 120;
for i in 0..n {
std::fs::write(dir.join(format!("IMG_{i:03}.jpg")), format!("not really a jpeg {i}")).unwrap();
}
@ -163,8 +164,11 @@ fn import_reads_files_off_the_ui_thread_and_can_be_cancelled() {
assert!(h.app.import.is_some(), "still importing");
let status = h.app.import.as_ref().map(crate::import::ImportTask::status).unwrap_or_default();
assert!(status["total"].as_u64().unwrap_or(0) >= 8, "{status}");
// Cancel: the files being read finish, nothing new starts
let r = h.request("ui.clickWidget", json!({"id": "button:importCancel"}), Duration::from_secs(10));
// Cancel (the ✕ of its row in the activity stack): the files being read finish, nothing new starts
let id = h.app.session.activity.list().first().map(|t| t.id).expect("an import row");
let cross = format!("activity:cancel:{id}");
assert!(h.step_until(Duration::from_secs(10), |h| h.app.widgets.iter().any(|(w, _)| *w == cross)), "the row shows");
let r = h.request("ui.clickWidget", json!({"id": cross}), Duration::from_secs(10));
assert_eq!(r["ok"], true, "{r}");
let t0 = Instant::now();
while h.app.import.is_some() && t0.elapsed() < Duration::from_secs(60) {

2
crates/ui-egui/src/tests_quit_unsaved.rs
View File

@ -79,7 +79,7 @@ fn quitting_with_unsaved_changes_asks_first() {
assert_eq!(r["ok"], false, "{r}");
assert!(!crate::panels::notices::may_close(&mut h.app), "retried, still failing: ask");
h.step();
let prompt = h.app.quit_prompt.clone().unwrap();
let prompt = h.app.quit_prompt.clone().unwrap().text().to_string();
assert!(prompt.contains("1 change couldn't be written to disk") && prompt.contains("drive unplugged"), "{prompt}");
for b in ["button:quitRetry", "button:quitAnyway", "button:quitCancel"] {
assert!(has(&h, b), "{b}");

52
crates/ui-egui/src/tests_sync.rs
View File

@ -9,7 +9,7 @@
//! * Ticking "Remove missing photos" also moves those to Recently Deleted.
//! * Cancel leaves the library as it was.
//! * Synchronize hands the work to the background at once: the dialog closes, the app keeps
//! answering, and the photos arrive over the next frames under a progress window, as one undo
//! answering, and the photos arrive over the next frames under a row in the activity stack, as one undo
//! step.
use std::time::Duration;
@ -218,7 +218,7 @@ fn a_stopped_synchronize_says_it_was_stopped_and_stays_at_most_one_step() {
let steps = h.app.session.undo.len();
let dlg = h.app.ui.dialog.take().expect("the dialog is open");
assert!(crate::panels::dialogs::confirm_dialog(&mut h.app, &dlg).is_ok());
// the progress window's Cancel, before any frame committed anything
// Cancel (what the row's ✕ does), before any frame committed anything
h.app.sync_run.as_mut().expect("running").cancel();
assert!(h.step_until(T, |h| h.app.sync_run.is_none()), "it ends");
let toast = h.app.ui.toast.as_ref().map(|t| t.0.clone()).unwrap_or_default();
@ -254,3 +254,51 @@ fn a_synchronize_and_an_import_never_run_at_once() {
let r = crate::panels::dialogs::confirm_dialog(&mut h.app, &d);
assert!(r.is_err() && h.app.sync_run.is_none(), "no synchronize while importing: {r:?}");
}
#[test]
fn a_synchronize_shows_a_row_and_its_cross_stops_it() {
let dir = Scratch::new("row");
let mut h = changed_folder(&dir);
open_and_scan(&mut h, &dir.path("trip"));
let dlg = h.app.ui.dialog.take().expect("the dialog is open");
assert!(crate::panels::dialogs::confirm_dialog(&mut h.app, &dlg).is_ok());
let rows = h.app.session.activity.list();
assert_eq!(rows.len(), 1, "{rows:?}");
assert_eq!((rows[0].kind, rows[0].label.as_str()), ("sync", "Synchronizing folder"));
assert!(rows[0].detail.ends_with("trip"), "the folder as the dialog named it: {:?}", rows[0].detail);
assert!(rows[0].cancellable);
// the row's ✕ (as `activity.cancel`): what was done stays, and the end says it was stopped
h.app.session.activity.cancel(rows[0].id).unwrap();
assert!(h.step_until(T, |h| h.app.sync_run.is_none()), "it ends");
let toast = h.app.ui.toast.as_ref().map(|t| t.0.clone()).unwrap_or_default();
assert!(toast.contains("Stopped"), "{toast:?}");
assert!(h.app.session.activity.list().is_empty(), "the row goes with the run");
}
#[test]
fn the_folders_scan_shows_a_row_and_its_cross_closes_the_dialog() {
let dir = Scratch::new("scanrow");
let mut h = changed_folder(&dir);
// a scan that can't finish until the test lets it: reading the new file waits on a flag
let gate = std::sync::Arc::new(std::sync::atomic::AtomicBool::new(false));
let g = gate.clone();
h.app.session.media.file_probe = Some(std::sync::Arc::new(move |_: &str| {
while !g.load(std::sync::atomic::Ordering::Relaxed) {
std::thread::sleep(Duration::from_millis(5));
}
Ok(lightcraft_engine::media::ProbeInfo { format: "PNG".into(), ..Default::default() })
}));
crate::sync::open(&mut h.app, &dir.path("trip"), "trip", false).unwrap();
let rows = h.app.session.activity.list();
assert_eq!(rows.len(), 1, "{rows:?}");
assert_eq!((rows[0].kind, rows[0].label.as_str(), rows[0].detail.as_str()), ("sync", "Looking for changes", "trip"));
assert!(rows[0].cancellable);
h.app.session.activity.cancel(rows[0].id).unwrap();
h.step();
assert!(h.app.sync.is_none(), "✕ stops the scan");
assert!(h.app.ui.dialog.is_none(), "and closes its dialog: there is nothing to show");
gate.store(true, std::sync::atomic::Ordering::Relaxed);
assert!(h.app.session.activity.list().is_empty());
h.settle(SETTLE);
assert!(h.app.ui.dialog.is_none(), "a stopped scan opens nothing later");
}

65
docs/background-tasks.md Normal file
View File

@ -0,0 +1,65 @@
# Long-running tasks and the activity stack
Everything that takes a while runs off the UI thread and shows up in **one** place: the activity stack in the
top-left corner under the top bar (issue #345). Each task has a row with its name, a progress bar (a sweep while
the amount of work isn't known), the count and what it's working on, and ✕ if it can stop. Quitting asks first while
a task that can stop is running. Agents see the same list through `activity.list` / `activity.cancel`, and
`ui.inspect` reports it as `activity`.
The registry is `lightcraft_engine::activity` (`crates/engine/src/activity.rs`, `Session::activity`). The stack is
`crates/ui-egui/src/panels/activity.rs`. Every job that can last more than a moment uses it; **don't add a progress
window, panel or progress toast of your own**. A dialog may still show progress inline (the Synchronize Folder
dialog counts files while it scans), and a toast at the end saying what happened is fine.
## Adding a task
```rust
use lightcraft_engine::activity::{Cancel, Unit};
// when the job starts, with the cancel flag the worker already checks between files
let guard = app.session.activity.start("export", "Exporting", Cancel::Flag(cancel.clone()));
guard.progress(0, total as u64);
// every frame (or from the worker, through `guard.handle()`)
guard.progress(done as u64, total as u64);
guard.detail(&file_name);
```
- **Keep the guard for exactly as long as the job runs.** Put it in the struct the UI polls (`ExportTask`, `ScanTask`,
`SyncRun`…), or move it into the worker thread. Dropping it removes the row, also when the worker panics, so a
dead job never leaves a stuck bar. A worker that only reports progress takes a `TaskHandle` (`guard.handle()`,
`Clone + Send`).
- **`kind`** is a stable camelCase id that agents filter by (`export`, `import`, `scan`, `sync`, `previews`,
`download`, `faces`…). **`label`** is a short English name in sentence case ("Building previews"). The stack shows it
translated, so add it to every catalog in `crates/ui-egui/locales/`.
- **Progress:** `progress(done, total)`. A `total` of 0 means the amount of work isn't known yet, and the bar sweeps.
`set_unit(Unit::Bytes)` makes the count read "12 of 340 MB"; `Unit::Percent` reads "40 %". The default `Unit::Count`
reads "3 of 25".
- **Detail:** what the job is on (a file name, a model's name). Detail set by engine code is a name, never a sentence.
UI code may set translated text.
- **Cancel:**
- `Cancel::Flag(Arc<AtomicBool>)` adopts the flag the job already checks. The row's ✕, `activity.cancel` and the
job's own Cancel then all set the same atomic.
- `Cancel::Yes`: the job polls `guard.is_cancelled()` itself.
- `Cancel::No`: the job can't stop; it gets no ✕ and doesn't make Quit ask.
- `set_cancellable(false)` for a phase that has to finish once started (a Lightroom import adding its photos).
- **Quiet periodic work** shows no row: the auto-import folder listing runs every few seconds through
`tasks::spawn(.., None, ..)`.
## Examples in the code
| Job | Where | Shows how to |
|---|---|---|
| Export, contact sheet | `crates/ui-egui/src/export_task.rs` | adopt the job's flag; update the row each frame |
| Build Previews | `crates/engine/src/cmd/previews.rs` | move the guard into the worker thread |
| Model downloads | `crates/engine/src/face_download.rs` | report progress through a `TaskHandle` in shared state |
| Lightroom catalog import | `crates/ui-egui/src/lightroom_import.rs` | stop being cancellable for a phase that must finish |
| Face scan, AI Denoise | `crates/engine/src/cmd/face_recognize.rs`, `crates/engine/src/denoise.rs` | a continuous job with no ✕, kept in session state |
| Find Missing Photos | `crates/ui-egui/src/tasks.rs` | a generic background task with a row (`kind: Some(..)`) |
## Testing
Assert the row (`session.activity.list()`: kind, label, total, `cancellable`), that `activity.cancel(id)` stops the
job, and that the row is gone once the job ends. `crates/ui-egui/src/tests_activity.rs` has the helpers, and the
job-specific tests show the pattern (`tests_sync.rs`, `export_task.rs`, `tests_denoise.rs`…). A row only appears
on screen once its task is 0.5 s old, so tests that click its ✕ wait for `activity:cancel:<id>` first.

43
docs/camera-preview-colour.md
View File

@ -99,6 +99,49 @@ Measured before → after (mean L* and CIE76 ΔE at 200 px): on the 24 ILCE-7CR
Picture styles other than Standard are lowered to the Standard curve too where it is darker; Lightroom ignores them as well. Nikon Active D-Lighting, Canon Auto Lighting Optimizer and Fujifilm D-Range settings are likely to have the same effect and are not handled yet. Render cache version 23 redraws thumbnails; smart previews built before keep the brightened tone curve in their header until rebuilt.
## Spectral camera matrices and the order of precedence
For 52 camera models LightCraft has colour matrices fitted to the camera's measured spectral sensitivities
(`crates/raw/src/spectral.rs`, generated into `spectral_table.rs` by `tools/spectral_camera_table.py` from
[rawtoaces-data](https://github.com/AcademySoftwareFoundation/rawtoaces-data) v1.1.0, Apache-2.0): 22 Canon, 11 Nikon,
11 Sony, 3 Fujifilm and 5 others, plus the other names rawtoaces-data lists for them (EOS Rebel T3i and Kiss X5 for
the EOS 600D; bodies it lists as sharing a sensor, such as the GFX100S with the GFX 100). Per model the generator fits,
under CIE Standard Illuminant A and under D65, a white-preserving 3×3 from white-balanced camera RGB to XYZ by the
least mean CIE76 error over rawtoaces-data's 190 training reflectances, and writes DNG `ColorMatrix1/2` and
`ForwardMatrix1/2`. The existing DNG colour model then interpolates them for the file's as-shot white, which the
vendor white-balance multipliers give through the same matrices. No number comes from Adobe or from a raw decoder.
A raw starts from the first of these that is available (`camera_preview::starting_colour`):
1. the file's own colour matrices (DNG);
2. a camera profile for its model (bundled or local), with tone and chroma fitted to the file's JPEG;
3. the file-local fit to its own JPEG described above;
4. the model's spectral matrices, with LightCraft's default tone curve and no JPEG fit;
5. the neutral fallback.
The spectral matrices only replace the neutral fallback: a photo whose JPEG fit is accepted (with or without a
profile) renders exactly as before, and a covered model takes its spectral matrices only when that fit fails the
gates or the file has no usable JPEG. White balance stays relative to the as-shot look for every raw without matrices
of its own, as before, so the catalog's `Photo::relative_wb` and the controls are unchanged. `lightcraft-cli
calibrate` still pools the uncorrected camera colour of covered models, so a profile can be fitted for them.
No colour-difference measurement of the spectral matrices against the camera JPEG, or against the file-local fit,
is recorded here yet. Whether they should come before the file-local fit for these models is open until one is.
## Smart previews and the look version
A smart preview keeps the result of the fit twice: the colour matrix is baked into its pixels and the tone and chroma curves are stored in its header. When the fit changes, a preview built earlier keeps the old look until it is rebuilt, and the render cache version does not reach it (renders are keyed on the library's files, proxies are files of their own).
The fit therefore has a version, `LOOK_VERSION` in `crates/engine/src/camera_preview.rs`, and every smart preview records the version that wrote it in its header line (`look_version`). **Bump the constant in any change that makes the fit give a different result for the same file**: the matrix, hue/saturation, tone or chroma fit, their gates and fallbacks, or how a camera profile feeds them. Changes that leave the fitted look alone don't need it. A header without the field (everything built before it existed) counts as version 0, and so does a value that is not a non-negative integer; a value from a newer build is left alone.
A proxy older than the constant is brought up to date by the existing Build Smart Previews path (`smart_run`), which also repairs damaged proxies:
- only proxies of raws that take the per-file look (`file_local_look`: ARW, NEF, RW2, RAF, CR3, CR2, PEF, SRW, ORF…) go stale; the fit never reaches JPEG, PNG, TIFF or DNG proxies, which are never rebuilt for it;
- when the library is opened and its smart previews folder has not been checked at this `LOOK_VERSION` (the `look-version` marker file in the folder), a background thread looks at those proxies, rebuilds the stale ones from their originals and then writes the marker, so the scan runs once per version bump, not at every opening (no proxy is created, nothing is shown, the UI never waits on a drive);
- Build Smart Previews does the same for the photos it is run on (`refreshed` in its result), whatever the marker says;
- when the original can't be read (offline drive), the proxy and its curve are kept exactly as they are (`staleKept`). The old version in its header is the mark: the next Build Smart Previews with the original online rebuilds it. A rebuilt proxy carries the current version, so nothing is rebuilt twice. A proxy is only ever rebuilt from the original, never from another proxy;
- the two never write the same proxy at once: each proxy is checked and written under one lock.
## Nikon crop and preview colour metadata
Nikon maker-note `CropArea` (0x0045) supplies the default `[left, top, width, height]` crop. The decoder validates the rectangle against the active sensor area and falls back to that area for missing or invalid values. The CFA origin is unchanged; cropping follows demosaicing.

4
docs/control-protocol.md
View File

@ -30,7 +30,7 @@ mode ([mcp.md](mcp.md)) is a thin layer over this channel. Implementation:
| `engine.commands` | — | Engine + UI commands: id, label, menu, shortcut, params doc, enabled |
| `ui.menu.list` | — | Menu entries (flat: id, label, menu path, shortcut, enabled) |
| `ui.menu.tree` | — | The menu bar as shown (File … Help): items `{id, params?, label, shortcut?, enabled, checked?}`, separators, submenus — the model behind the native macOS menu bar and the in-window menus |
| `ui.inspect` | — | UI state, window, `loupe` (`source`, and `region: {full, window, pending}` when a zoomed view also renders the window on screen at no more than 100 % — `full` is the frame it is cut from, `window` its `[x, y, w, h]` — and `regionBefore` for the Before side of a Before/After view), canvas/image rects, `scroll: {grid, filmstrip}` (scroll offsets in points, `null` until drawn), active photo, selection, perf (`frameMs` = layout, `logicMs` = per-frame logic before it, `updateMs` = both, `maxUpdateMs`, `fps`, render queue …, `gpu` = adapter in use, `gpuReason` = why renders don't use the GPU, `gpuFallback` = latest render redone on the CPU and why — see `docs/gpu-pipeline.md`), status, memory (bytes per cache, see `library.memory`; plus stage caches — `budgetBytes`, `trimmed`, `sharedSourceBytes` (the one device copy of a big original) — and textures), `export: {running: {total, done, current} \| null, last}`, `notices` (warnings waiting to be shown, e.g. a damaged settings file; OK = `button:noticeOk`), `quitPrompt` (why quitting was stopped: unsaved changes; `button:quitRetry` / `button:quitAnyway` / `button:quitCancel`), `import: {done, total, imported, cancelled} \| null` (an import runs on a worker thread), `tasks` (other background work: `Find Missing Photos`, `Auto Import`), `fileDialogs` (commands waiting on a native file dialog, which runs off the UI thread; they run again with the answer when it closes), `copied` (the text the UI last put on the clipboard, `null` until something is copied) |
| `ui.inspect` | — | UI state, window, `loupe` (`source`, and `region: {full, window, pending}` when a zoomed view also renders the window on screen at no more than 100 % — `full` is the frame it is cut from, `window` its `[x, y, w, h]` — and `regionBefore` for the Before side of a Before/After view), canvas/image rects, `scroll: {grid, filmstrip}` (scroll offsets in points, `null` until drawn), active photo, selection, perf (`frameMs` = layout, `logicMs` = per-frame logic before it, `updateMs` = both, `maxUpdateMs`, `fps`, render queue …, `gpu` = adapter in use, `gpuReason` = why renders don't use the GPU, `gpuFallback` = latest render redone on the CPU and why — see `docs/gpu-pipeline.md`), status, memory (bytes per cache, see `library.memory`; plus stage caches — `budgetBytes`, `trimmed`, `sharedSourceBytes` (the one device copy of a big original) — and textures), `export: {running: {total, done, current} \| null, last}`, `notices` (warnings waiting to be shown, e.g. a damaged settings file; OK = `button:noticeOk`), `quitPrompt` (why quitting was stopped: tasks that would be cut short are running — `button:quitAnyway` cancels them and quits, `button:quitCancel` keeps working — or unsaved changes: `button:quitRetry` / `button:quitAnyway` / `button:quitCancel`), `import: {done, total, imported, cancelled} \| null` (an import runs on a worker thread), `tasks` (other background work: `Find Missing Photos`, `Auto Import`), `activity` (every long-running task in flight as the activity stack in the top-left corner shows it, same shape as `activity.list`; a row appears once its task is 0.5 s old, ids `activity:row:<id>`, ✕ = `activity:cancel:<id>`, `activity:more` expands past 3 rows), `fileDialogs` (commands waiting on a native file dialog, which runs off the UI thread; they run again with the answer when it closes), `copied` (the text the UI last put on the clipboard, `null` until something is copied) |
| `ui.widgets` | `{filter?}` | On-screen widgets `{id, rect: [x, y, w, h]}` (screen points) |
| `ui.clickWidget` / `ui.dragWidget` / `ui.hoverWidget` | `{id, count?, button?, fx?, fy?}` (`button: "right"` right-clicks) / `{id, toX?, toY?, dx?, dy?, steps?}` / `{id, fx?, fy?}` | Real egui input on a widget (hover: the pointer rests on it, e.g. for preset/profile previews) |
| `ui.move` / `ui.click` / `ui.drag` | `{x, y, count?, button?}` / `{x, y, toX, toY, steps?}` | Raw pointer input, screen points |
@ -46,6 +46,8 @@ mode ([mcp.md](mcp.md)) is a thin layer over this channel. Implementation:
| `ui.zoomFactor` | `{factor}` | Set the interface scale (0.5–3; egui's zoom keys are off, Cmd+= / Cmd+- zoom the photo) |
| `ui.screenshot` | `{path?, headless?}` | `{path, width, height}` once the frame (with finished renders) is captured. `headless: true` draws the UI on the CPU (no compositor needed); a windowed capture that gets no frame within 2 s falls back to headless automatically |
| `engine.execute {command: "app.export", params}` | export params (see `docs/mcp.md`), plus `preset`, `dir` / `path`, `ids`, `background` | Writes the files and returns `{files}`; with `background: true` (what the Export dialog and menus use) it returns `{background: true, total}` at once and the batch runs on a worker thread — poll `ui.inspect` → `export` |
| `engine.execute {command: "activity.list"}` | — | The long-running tasks in flight (imports, exports, preview builds, downloads, the face scan…), oldest first: `{tasks: [{id, kind, label, done, total, unit, detail, cancellable, cancelling, ageMs}]}`; `total` 0 = not known yet; `unit` is `count`, `bytes` or `percent`; `label` is English (the app shows it translated) |
| `engine.execute {command: "activity.cancel"}` | `{id}` or `{all: true}` | Asks the task (or every cancellable one) to stop at its next file or photo → `{cancelled: n}`; an unknown id (it may have just finished) or a task that can't be cancelled is an error |
| `ui.render` | `{id?, size?, path?}` | Render a photo (PNG to `path`), `{width, height}` |
| `app.quit` | — | Close the app |

2
docs/faces.md
View File

@ -248,7 +248,7 @@ Turning recognition on starts a scan of the whole library, once, in the backgrou
the running ones, so a worker that finishes has its next photo at once.
- **Earlier scan speed (a different runner).** A mixed raw and JPEG library of 184 photos (the raws read through their embedded previews, 73 photos searched
for faces) took about 29 seconds on that machine at full pace: roughly 6 photos a second, so the first scan of 10,000 photos
is a matter of half an hour; after that only new photos are looked at. Settings ▸ Faces shows how many are left, with a progress bar. After you accept a model's terms
is a matter of half an hour; after that only new photos are looked at. Settings ▸ Faces shows how many are left, with a progress bar, and the activity stack shows "Finding faces" meanwhile. After you accept a model's terms
you are returned to that tab, so the download and then the scan can be watched there; the main window has no status bar.
- Opening another library starts a fresh scan state; nothing learned about one library is used in another.

2
docs/merge.md
View File

@ -9,7 +9,7 @@ All algorithms are our own implementations from published papers (cited in the m
- **App:** select the photos, then *Photo Merge ▸ HDR…* (`Ctrl+H`), *Panorama…* (`Ctrl+M`) or
*HDR Panorama…* from the photo context menu (or the `dialog.merge*` commands). The dialog previews
the merge at ≤ 1024 px and re-runs the preview when an option changes; *Merge* runs the full merge
in the background (progress in a toast). The result is written next to the first photo as
in the background (a row in the activity stack shows its progress, with ✕ to stop it). The result is written next to the first photo as
`<name>-HDR.dng`, `-Pano.dng` or `-HDR-Pano.dng`, imported and selected.
- **Commands** (CLI, MCP, control channel):
- `merge.hdr {ids?, align=true, deghost=none|low|medium|high, autoSettings=true, stack=false, preview=false, showOverlay=false, previewPath?}`

22
docs/parity.md
View File

File diff suppressed because one or more lines are too long

117
docs/sony-lens-corrections.md Normal file
View File

@ -0,0 +1,117 @@
# Sony embedded distortion corrections
LightCraft reads the signed 16-sample distortion table in the raw image IFD (`0x7037`) of
Sony **ILCE-7RM4A (A7R IVA)** Bayer ARWs. Other camera models are deliberately left uncorrected,
even when they contain a table with the same layout.
It converts that table into `OpcodeList3` / `WarpRectilinear`, so the existing profile correction control,
CPU/GPU optics path, EXIF orientation handling, and DNG export use the same correction.
This adds **distortion only**. Sony vignetting and lateral chromatic-aberration tables are not decoded.
The enabled and independently validated model is ILCE-7RM4A with the FE 24–105mm F4 G OSS and
FE 200–600mm F5.6–6.3 G OSS. Linear YCbCr ARWs and non-3:2 camera crops are excluded pending validation. An aspect crop may
retain the full-frame radial normalization; the decoder does not guess that relationship. Older files that carry
only encrypted correction metadata, different table lengths, and rejected tables remain uncorrected.
The camera's distortion Off setting does not erase its table. Newly imported photos use LightCraft's existing
embedded-profile default (enabled); the profile correction control can disable or adjust it.
## Clean-room evidence
The [ExifTool EXIF tag-name documentation](https://exiftool.org/TagNames/EXIF.html) identifies
`0x7037` as 17 signed 16-bit integers. The first value is the count, 16, followed by radial samples.
No external photo decoder implementation or Adobe lens profile was consulted.
Sony Imaging Edge Edit 4.1 was used only as a black-box reference on local scratch copies.
The observed inverse mapping, with radius `r` normalized to the default crop's half diagonal, is:
```
f(r) = 1 + interpolate(table, r) / 16384
source = center + (output - center) * zoom * f(r)
zoom = 1 / max(f(r)) over radii on the rectangular output boundary
```
The border radii range from the shorter half edge / half diagonal to 1. Table knots span 0 through 1.
Linear interpolation approximates the observed map closely; we do not claim to reproduce Sony's
internal interpolator. A fixed-size weighted least-squares fit expresses source radius as
`r * (k0 + k1*r² + k2*r⁴ + k3*r⁶)`. The fit is rejected if radial error exceeds 0.0005 of the
half diagonal (about 0.43 px at a 1440 px 3:2 preview), or the mapping folds. Table values are bounded
and the input must have precisely the supported signed-short layout. Crop offset and normalization
are converted into DNG's active-area coordinate system before creating the opcode. The warp's centre
and normalization come from the standard `DefaultCropOrigin` / `DefaultCropSize` when the file has them,
not from Sony's crop tags: on the tested A7R IVA files the former start at raw x=32 and the latter at
x=0, and the standard origin is the one that aligns with Sony's exports. The image itself keeps the
decoder's crop precedence for every Sony body (Sony's crop tags, then the default crop, then the
recorded image size), so the framing is unchanged; the opcode's centre is in active-area coordinates,
so the warp stays on the optical centre either way. On the A7R IVA that framing still sits 32 raw
pixels left of Sony's exports (eight pixels in a 2376-pixel-wide export).
The main raw-IFD table is important: the similarly named MakerNote table has different values and
must not be substituted into this formula. Main raw-IFD and SR2SubIFD values agreed in the test files.
## Validation
At 2376×1584, independent SIFT correspondences between Sony distortion On and Off TIFF exports gave:
| Lens / focal length | Median residual | 95th percentile | Maximum matched radius |
|---|---:|---:|---:|
| FE 200–600 / 200 mm | 0.043 px | 0.191 px | 0.97 half diagonals |
| FE 200–600 / 600 mm | 0.055 px | 0.231 px | 0.95 half diagonals |
These use the fixed 16384 scale and automatic border framing, not fitted per-image scale or zoom.
A median translation below 0.004 px was removed from the residuals. Correspondences do not cover
all pixels; these are geometric feature residuals, not photometric equality or a worst-case guarantee.
Across 20 sampled RAWs (16 at 24–105 mm, four at 200–600 mm), the polynomial's maximum deviation
from the interpolated file table was below 0.12 px at 1440 px. That measures approximation error,
not agreement with an independent renderer. Camera JPEG and Apple Core Image comparisons support
the wide-lens interpretation but are weaker evidence than same-renderer On/Off pairs.
Actual LightCraft exports after crop normalization were also compared directly with Sony's corrected
TIFFs. At 2376 px wide, median / 95th-percentile feature residuals were 0.35 / 1.04 px at 24 mm,
0.15 / 0.40 px at 200 mm, and 0.13 / 0.56 px at 600 mm (median translation below 0.024 px).
The uncorrected medians at 200 and 600 mm were 8.63 and 10.82 px respectively. Different colour,
sharpening, and demosaicing affect feature localization; these are geometry checks, not colour parity.
An isolated desktop run on Apple M1 Metal detected the profile, rendered without GPU fallback or
warnings, and exposed the existing Optics controls. Existing library edits are not reset: once the
source has loaded, enable lens corrections if they were previously disabled.
Tests cover signedness/length rejection, zero and malformed tables, independent 600 mm reference
geometry, barrel framing, offset crop normalization, both TIFF byte orders, header/full consistency,
and preservation through DNG export. No private photographs or reference TIFFs are committed.
## Camera coverage and regression boundary
The model check is exact: `ILCE-7RM4` (A7R IV without the A suffix), `ILCE-7M3`, unknown/missing
model names, and every other model remain unsupported by this distortion decoder. Matching tag
numbers and table lengths do not establish the scale, interpolation or crop normalization on another
body. Support can expand after independent geometry validation; do not add model-name prefix matching.
The original nine Sony samples listed by `cargo xtask corpus --download` were downloaded from its
CC0 URLs. The manifest now also includes CC0 samples 4822 (A7R IVA / 24–105 mm) and 3989
(A9 II / 200–600 mm), with their SHA-256 hashes recorded alongside the URLs. All eleven files
are checked with `sony_embedded_distortion_is_limited_to_validated_models`:
| Camera | Corpus samples | Raw-IFD table observed | Expected correction |
|---|---|---|---|
| ILCE-7RM4A | Compressed (4822) | 16 samples | One distortion warp |
| ILCE-9M2 | Compressed (3989) | 16 samples | None |
| ILCE-7M3 | Compressed, uncompressed | 16 samples | None |
| ILCE-7M4 | 14-bit, lossless L/M/S | 11 samples in the 14-bit file; 16 in L/M/S | None |
| ILCE-7RM2 | 12-bit uncompressed | 16 samples | None |
| DSC-RX100 | One | No raw-IFD table | None |
| DSC-RX100M3 | One | 11 samples | None |
These are **model-boundary regression checks, not new geometric validation of those models**. In particular,
refusing the A7 III tables resolves the unverified corner warp identified in review. The lossless M/S
files are linear YCbCr and remain excluded independently of the model check. Header and full decode
must agree for every sample. Synthetic tests also verify both TIFF byte orders, exact model matching,
that the warp is centred on the standard default crop while the image keeps Sony's crop, and that DNG export preserves an accepted warp without adding one to rejected
models. The existing independent Sony geometry tests continue to cover the enabled A7R IVA path.
Re-exporting the 24, 200 and 600 mm reference files after the model restriction produced byte-identical
corrected PNGs to the earlier validated exports. Isolated headless app checks showed the A7R IVA profile
controls present and the A7 III profile controls absent, with no notices and completed RAW renders.
Before enabling another body, compare corrected and uncorrected renders against independent references
across multiple lenses/focal lengths and photographs with features near the edges and corners. Record
residuals and spatial coverage, and check crop/aspect and compression variants. Embedded JPEGs are
useful when the camera actually applied distortion correction; an Off JPEG cannot certify an On warp.
Prefer same-renderer Sony On/Off exports when the camera setting or geometry is ambiguous. Table-fit
error alone only checks our polynomial approximation, not whether the table's interpretation is correct.

16
docs/whats-new.md
View File

@ -2,6 +2,15 @@
## October 2026
### Activity stack
- Long-running tasks show in one place, top left under the top bar, as in Lightroom Classic (issue #345): imports and
folder scans, exports (and the contact sheet PDF), Synchronize Folder, Build / Smart Previews, Lightroom catalog import, Photo Merge, model downloads, the face scan,
AI Denoise and Find Missing Photos. Each row has the task's name, a progress bar, the count and what it is working
on; ✕ stops the tasks that can stop. Three rows show, then "+N more". The separate export panel and the progress
windows and toasts are gone.
- Quitting while a task that can be stopped is running asks first ("Quit Anyway" stops it).
- Agents list and stop tasks with `activity.list` and `activity.cancel`; `ui.inspect` reports them as `activity`.
### Smart album rules
- The rule editor's field menu is grouped: Rating, Pick Flag, Color Label and Any Searchable Text at the top, then
submenus for Source, File, Date, Keywords & People, Description, Camera Info, Location, Size, Develop and Assisted
@ -92,7 +101,7 @@
file is gone, photos whose file was renamed or moved within the folder, and XMP sidecars changed by other apps; then
choose what to do: import the new photos and relink the moved ones (the defaults), move the missing ones to Recently
Deleted, read the metadata updates. A folder on a disk that isn't connected is reported as such. Synchronizing
runs in the background with a progress window and Cancel; everything it does is one undo step, and no file on disk
runs in the background with a row in the activity stack (✕ stops it); everything it does is one undo step, and no file on disk
is touched. Agents use `folder.scanChanges` and `folder.synchronize`.
### Folder colour labels
@ -157,6 +166,11 @@
- Samsung SRW files without compression now open as raws: NX5, NX10, NX11, NX20, NX200, NX210, NX1000, NX1100, EX1
and WB2000. The compressed ones (NX1, NX30, NX300, NX500, NX2000, NX3000, NX3300, NX mini) still open from their
camera JPEG. Photos already imported pick the change up on Reload.
- Raws from 52 camera models (22 Canon, 11 Nikon, 11 Sony, 3 Fujifilm and 5 others, plus their other names) whose
camera JPEG can't be fitted no longer open with the neutral fallback: they start from colour matrices fitted to the
camera's measured spectral sensitivities, from the Academy Software Foundation's rawtoaces-data. A camera profile
and the fit to the photo's own JPEG still come first, so photos that had a fitted look keep it. See
[camera preview colour](camera-preview-colour.md#spectral-camera-matrices-and-the-order-of-precedence).
- Canon CR2 raws no longer open too dark with blocked-up shadows: the black level was measured over border columns
that light already reaches (left of the image area Canon records), 4 % of the range too high on the EOS 6D. It is
now measured on the masked columns alone (8 corpus bodies, EOS 40D to 5DS R). On a 6D photo the starting render

5
tools/requirements-spectral-camera-table.txt Normal file
View File

@ -0,0 +1,5 @@
# Python packages for tools/spectral_camera_table.py (crates/raw/src/spectral_table.rs).
# python3 -m venv .venv-spectral && .venv-spectral/bin/pip install -r tools/requirements-spectral-camera-table.txt
numpy==2.5.3
scipy==1.18.1
colour-science==0.4.7

122
tools/spectral_camera_table.py Normal file
View File

@ -0,0 +1,122 @@
# Writes crates/raw/src/spectral_table.rs: DNG-shaped colour matrices per camera model, fitted to the
# measured spectral sensitivities in rawtoaces-data (Academy Software Foundation, Apache-2.0).
#
# git clone https://github.com/AcademySoftwareFoundation/rawtoaces-data ../rawtoaces-data
# git -C ../rawtoaces-data checkout v1.1.0
# python3 -m venv .venv-spectral && .venv-spectral/bin/pip install -r tools/requirements-spectral-camera-table.txt
# .venv-spectral/bin/python tools/spectral_camera_table.py ../rawtoaces-data crates/raw/src/spectral_table.rs
#
# Method (Academy P-2013-001, written from its description; no rawtoaces code is used): the camera and the CIE 1931
# observer both look at the 190 training reflectances of rawtoaces-data under CIE Standard Illuminant A and under D65.
# The camera values are white balanced to that illuminant, and a white-preserving 3x3 to XYZ is fitted by least
# mean CIE76 error in Lab. Per illuminant this gives ColorMatrix (XYZ -> camera, DNG CalibrationIlluminant 17 / 21)
# and ForwardMatrix (white-balanced camera -> XYZ D50). The aliases are rawtoaces-data's own `data/aliases.json`.
import json
import os
import subprocess
import sys
import colour
import numpy as np
from scipy.optimize import minimize
WL = np.arange(380, 781, 5)
D50 = colour.xy_to_XYZ(colour.CCS_ILLUMINANTS["CIE 1931 2 Degree Standard Observer"]["D50"])
def spectral(path, columns):
data = json.load(open(path))["spectral_data"]["data"]["main"]
return np.array([[data[str(w)][i] for i in range(columns)] for w in WL], dtype=float)
def illuminant(name):
sd = colour.SDS_ILLUMINANTS[name]
return np.array([sd[w] for w in WL], dtype=float)
def fit(cmf, refl, ssf, light, target_white):
"""White-balanced camera RGB -> XYZ adapted to `target_white`; maps (1, 1, 1) to `target_white`."""
w = ssf.T @ light
cam = (ssf.T @ (light[:, None] * refl)) / w[:, None]
k = 1.0 / (cmf[:, 1] @ light)
xyz = k * (cmf.T @ (light[:, None] * refl))
white = k * (cmf.T @ light)
cat = colour.adaptation.matrix_chromatic_adaptation_VonKries(white, target_white, transform="Bradford")
xyz = cat @ xyz
target_xy = colour.XYZ_to_xy(target_white)
lab = colour.XYZ_to_Lab(xyz.T, target_xy)
def build(p):
return np.stack([p[:3], p[3:], target_white - p[:3] - p[3:]], axis=1)
def loss(p):
return float(np.mean(np.linalg.norm(colour.XYZ_to_Lab((build(p) @ cam).T, target_xy) - lab, axis=1)))
m0 = xyz @ np.linalg.pinv(cam)
r = minimize(loss, np.concatenate([m0[:, 0], m0[:, 1]]), method="Nelder-Mead", options={"maxiter": 20000, "xatol": 1e-9, "fatol": 1e-9})
return build(r.x), loss(r.x)
def dng_tags(cmf, refl, ssf):
"""[(ColorMatrix, ForwardMatrix, mean training dE76)] for Standard Illuminant A, then D65."""
out = []
for name in ("A", "D65"):
light = illuminant(name)
forward, error = fit(cmf, refl, ssf, light, D50)
k = 1.0 / (cmf[:, 1] @ light)
white = k * (cmf.T @ light)
to_xyz, _ = fit(cmf, refl, ssf, light, white)
cam_from_xyz = np.diag(ssf.T @ light) @ np.linalg.inv(to_xyz)
cam_from_xyz = cam_from_xyz / (cam_from_xyz @ white).max()
out.append((cam_from_xyz.round(6), forward.round(6), error))
return out
def mat(m):
return "Mat3([" + ", ".join("[" + ", ".join(f"{v:.6}" for v in row) + "]" for row in m) + "])"
def main(root, out):
commit = subprocess.run(["git", "-C", root, "rev-parse", "HEAD"], capture_output=True, text=True, check=True).stdout.strip()
cmf = spectral(os.path.join(root, "data/cmf/cmf_1931.json"), 3)
training = json.load(open(os.path.join(root, "data/training/training_spectral.json")))["spectral_data"]["data"]["main"]
refl = np.array([training[str(w)] for w in WL], dtype=float)
rows = []
for f in sorted(os.listdir(os.path.join(root, "data/camera"))):
if not f.endswith("_380_780_5.json"):
continue
path = os.path.join(root, "data/camera", f)
header = json.load(open(path))["header"]
if header.get("license") != "Apache-2.0":
sys.exit(f"{f}: licence {header.get('license')!r}, expected Apache-2.0")
(cm1, fm1, e1), (cm2, fm2, e2) = dng_tags(cmf, refl, spectral(path, 3))
print(f"{f}: training dE76 A {e1:.3f}, D65 {e2:.3f}", file=sys.stderr)
rows.append(
f' Camera {{\n make: "{header["manufacturer"]}",\n model: "{header["model"]}",\n'
f" color_matrix: [{mat(cm1)}, {mat(cm2)}],\n forward_matrix: [{mat(fm1)}, {mat(fm2)}],\n }},"
)
aliases = json.load(open(os.path.join(root, "data/aliases.json")))["data"]["camera"]["model"]
alias_rows = [
f' ("{make}", "{alias}", "{target["model"]}"), // {target.get("comments", "")}'
for make, models in aliases.items()
for alias, target in models.items()
]
rust = (
"// Generated by tools/spectral_camera_table.py from rawtoaces-data "
f"(https://github.com/AcademySoftwareFoundation/rawtoaces-data, commit {commit}).\n"
"// Derived from the camera spectral sensitivities of rawtoaces-data (Academy Software Foundation), licensed under\n"
"// the Apache License 2.0 (LICENSE-APACHE); see NOTICE and assets/ATTRIBUTION.md. Do not edit by hand.\n"
"use super::Camera;\n"
"use lightcraft_color::Mat3;\n\n"
"#[rustfmt::skip]\npub(super) const CAMERAS: &[Camera] = &[\n" + "\n".join(rows) + "\n];\n\n"
"/// `(make, model as sold, table model)`: other names of a camera in [`CAMERAS`], or of a camera with the same sensor.\n"
"#[rustfmt::skip]\npub(super) const ALIASES: &[(&str, &str, &str)] = &[\n" + "\n".join(alias_rows) + "\n];\n"
)
open(out, "w").write(rust)
print(f"{len(rows)} cameras, {len(alias_rows)} aliases -> {out}", file=sys.stderr)
if __name__ == "__main__":
if len(sys.argv) != 3:
sys.exit("usage: spectral_camera_table.py <rawtoaces-data checkout> <output .rs>")
main(sys.argv[1], sys.argv[2])

16
xtask/src/main.rs
View File

@ -313,6 +313,16 @@ fn cmd_ci() -> Result<(), String> {
/// The third field is the SHA-256 of the file as downloaded; `cargo xtask corpus --download` checks it.
/// One per format / compression variant we decode or deliberately report as unsupported (preview only).
const RAW_SAMPLES: &[(&str, &str, &str)] = &[
(
"arw-sony-a7rm4a-compressed.arw",
"https://raw.pixls.us/getfile.php/4822/nice/Sony%20-%20ILCE-7RM4A%20-%2014bit%2014bit%20compressed%20%283:2%29.ARW",
"690c774f1d7bc1db3fa8c2489762743d8e7586e50b6f65d0b6a61cb467972c67",
),
(
"arw-sony-a9m2-compressed.arw",
"https://raw.pixls.us/getfile.php/3989/nice/Sony%20-%20ILCE-9M2%20-%2014bit%2014bit%20compressed%20%283:2%29.ARW",
"161c2a9da2b5f1e50be6117a0b4da0ce249660b5d7de13568d301c4034716b97",
),
(
"arw-sony-a7m3-compressed.arw",
"https://raw.pixls.us/getfile.php/2414/nice/Sony%20-%20ILCE-7M3%20-%2014bit%2014bit%20compressed%20%283:2%29.ARW",
@ -420,6 +430,12 @@ const RAW_SAMPLES: &[(&str, &str, &str)] = &[
"https://raw.pixls.us/data/Sony/DSLR-A580/RAW_SONY_A580.ARW",
"5b0924d39151239dce19e92e08318f4f62a5a8ac276bd7180ec8463d2cfee709",
),
// packed 12-bit ARW (two pixels per three bytes): the DSLR-A900
(
"arw-sony-a900-packed12.arw",
"https://raw.pixls.us/data/Sony/DSLR-A900/_DSC7969.ARW",
"ac7c1532df77c321e8010aa1be60c1b9f245b1ac0db6f38fe617a73aae81af50",
),
// CR2 colour-filter layouts differ by model (issue #85): CR2CFAPattern 3 (GBRG) and 1 (RGGB) samples
(
"cr2-canon-40d.cr2",