Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Brandon Thomas
commit
0801219b17
|
|
@ -8,6 +8,9 @@ plan/
|
|||
.DS_Store
|
||||
.claude/
|
||||
/craft-fonts
|
||||
# nix build / nix develop
|
||||
/result
|
||||
/result-*
|
||||
# model weights are never committed (SAM 3: docs/ai-masks.md)
|
||||
*.safetensors
|
||||
*.safetensors.part
|
||||
|
|
|
|||
|
|
@ -119,4 +119,7 @@ of it. Full standard: `../craftrules/standards/never-crash.md`
|
|||
## Map of the code
|
||||
`geom`, `color`, `raster`, `tiff` (L0) → `raw`, `codecs`, `meta`, `develop` (L1) → `pipeline` → `catalog` → `engine`
|
||||
→ `ui-egui`, `mcp` (L5) → apps `lightcraft` (desktop), `lightcraft-cli` (render/commands/MCP). `scenes` generates demo
|
||||
photos. `xtask` = tooling (`ci`, `layers`, `assets`, `parity`, `wasm`, `corpus`, `stats`).
|
||||
photos. `xtask` = tooling (`ci`, `layers`, `assets`, `parity`, `wasm`, `corpus`, `stats`). `flake.nix` +
|
||||
`nix/package.nix` = the Nix package (`nix build` builds both binaries with the craft-fonts input, installs the
|
||||
desktop file/icons/AppStream metadata and runs `cargo test --workspace`; `nix develop` = dev shell). Community-maintained and not
|
||||
in CI: it may lag behind the workspace; see README → Quick start.
|
||||
|
|
|
|||
|
|
@ -119,4 +119,7 @@ of it. Full standard: `../craftrules/standards/never-crash.md`
|
|||
## Map of the code
|
||||
`geom`, `color`, `raster`, `tiff` (L0) → `raw`, `codecs`, `meta`, `develop` (L1) → `pipeline` → `catalog` → `engine`
|
||||
→ `ui-egui`, `mcp` (L5) → apps `lightcraft` (desktop), `lightcraft-cli` (render/commands/MCP). `scenes` generates demo
|
||||
photos. `xtask` = tooling (`ci`, `layers`, `assets`, `parity`, `wasm`, `corpus`, `stats`).
|
||||
photos. `xtask` = tooling (`ci`, `layers`, `assets`, `parity`, `wasm`, `corpus`, `stats`). `flake.nix` +
|
||||
`nix/package.nix` = the Nix package (`nix build` builds both binaries with the craft-fonts input, installs the
|
||||
desktop file/icons/AppStream metadata and runs `cargo test --workspace`; `nix develop` = dev shell). Community-maintained and not
|
||||
in CI: it may lag behind the workspace; see README → Quick start.
|
||||
|
|
|
|||
|
|
@ -2900,6 +2900,17 @@ dependencies = [
|
|||
"web-time",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "lightcraft-fetch"
|
||||
version = "0.2.1"
|
||||
dependencies = [
|
||||
"log",
|
||||
"rustls",
|
||||
"rustls-rustcrypto",
|
||||
"sha2",
|
||||
"webpki-roots",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "lightcraft-geom"
|
||||
version = "0.2.1"
|
||||
|
|
@ -3039,14 +3050,11 @@ version = "0.2.1"
|
|||
dependencies = [
|
||||
"candle-core",
|
||||
"candle-nn",
|
||||
"lightcraft-fetch",
|
||||
"log",
|
||||
"rustls",
|
||||
"rustls-rustcrypto",
|
||||
"serde",
|
||||
"serde_json",
|
||||
"sha2",
|
||||
"thiserror 2.0.21",
|
||||
"webpki-roots",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
|
|
|
|||
|
|
@ -29,6 +29,7 @@ lightcraft-merge = { path = "crates/merge" }
|
|||
lightcraft-engine = { path = "crates/engine" }
|
||||
lightcraft-ui-egui = { path = "crates/ui-egui" }
|
||||
lightcraft-mcp = { path = "crates/mcp" }
|
||||
lightcraft-fetch = { path = "crates/fetch" }
|
||||
lightcraft-segment = { path = "crates/segment" }
|
||||
|
||||
# foundation
|
||||
|
|
|
|||
33
README.md
33
README.md
|
|
@ -258,8 +258,8 @@ lightcraft --control 7980 ~/Pictures/trip
|
|||
## Fast, native, private
|
||||
|
||||
- **Pure Rust, no C.** Our own RAW decoders (DNG, Canon CR2, Sony ARW, Nikon NEF, Fujifilm RAF incl. X-Trans,
|
||||
Panasonic RW2, Pentax PEF, Olympus ORF), our own colour science, our own pipeline. JPEG, PNG, TIFF, WebP, PSD
|
||||
composites and JPEG XL open today.
|
||||
Panasonic RW2 / Leica RWL, Pentax PEF, Olympus ORF), our own colour science, our own pipeline. JPEG, PNG, TIFF, WebP,
|
||||
PSD composites and JPEG XL open today.
|
||||
- **Scene-referred & wide-gamut.** Linear Rec.2020 float internally, Bradford-adapted white balance, gamut mapping
|
||||
instead of clipping, a filmic shoulder for raw and pixel-exact pass-through for JPEGs you haven't touched.
|
||||
- **Resolution-independent edits.** Radii and brush sizes are relative to the image, so a 400 px preview, your
|
||||
|
|
@ -301,9 +301,9 @@ LightCraft is young and moving fast. **Where we honestly stand** (details in the
|
|||
| Masking: brush, linear, radial, luminance/colour range, add/subtract/intersect | ✅ (AI subject/sky use classical heuristics for now) |
|
||||
| Crop, straighten tool + auto straighten, flip, rotate, aspect ratios, overlays | ✅ |
|
||||
| Profiles (Color, Neutral, Vivid, Landscape, Portrait, Monochrome: our own looks), presets, versions, history, copy/paste/sync settings | ✅ |
|
||||
| Camera colour: DNG files use their own matrices | ✅ · our own calibration for other raws ⬜ (top priority; neutral fallback today) |
|
||||
| Camera colour: DNG files use their own matrices | ✅ · our own calibration for other raws ⬜ (top priority; ARW, NEF and RW2 start from a look fitted to their own JPEG, other raws from a neutral fallback) |
|
||||
| Native macOS menu bar (generated from the command registry), control channel + every widget addressable, headless UI snapshots | ✅ |
|
||||
| RAW: DNG, CR2, ARW, NEF (uncompressed + lossless/lossy compressed), Fujifilm RAF (uncompressed, Bayer + X-Trans), Panasonic RW2, Pentax PEF, Olympus ORF (uncompressed); embedded previews for every format incl. CR3 | ✅ · CR3, compressed RAF/ORF decode ⬜ |
|
||||
| RAW: DNG, CR2, ARW, NEF (uncompressed + lossless/lossy compressed), Fujifilm RAF (uncompressed, Bayer + X-Trans), Panasonic RW2 / Leica RWL / Panasonic RAW (every raw format, DMC-LX1 to DC-S1RM2), Pentax PEF, Olympus ORF (uncompressed); embedded previews for every format incl. CR3 | ✅ · CR3, compressed RAF/ORF decode ⬜ |
|
||||
| Detail: sharpening, luminance + colour noise reduction | ✅ · AI Denoise, Super Resolution ⬜ |
|
||||
| Remove / Heal / Clone spots (auto source), Visualize Spots, Red Eye and Pet Eye (auto pupil detection, catchlight) | ✅ · content-aware fill, spot pin editing 🚧 |
|
||||
| Export: JPEG / PNG / TIFF / WebP / AVIF / DNG / original, sizing, file-size limit, output sharpening, naming templates, batch, metadata policy, text or image watermark | ✅ · HDR export ⬜ |
|
||||
|
|
@ -357,6 +357,31 @@ Language**; the choice applies immediately and persists. See [docs/localization.
|
|||
The web build needs the `wasm32-unknown-unknown` target and the matching `wasm-bindgen` CLI
|
||||
(`cargo xtask web` prints the exact install command); see [docs/web.md](docs/web.md).
|
||||
|
||||
**Nix** builds the desktop app and `lightcraft-cli` (the Nix build always includes the craft-fonts input, so Japanese
|
||||
text has glyphs):
|
||||
|
||||
```sh
|
||||
nix run github:storytold/lightcraft # the desktop app
|
||||
nix build github:storytold/lightcraft # → ./result/bin/{lightcraft,lightcraft-cli}
|
||||
nix develop github:storytold/lightcraft # rust toolchain + native deps + fonts
|
||||
```
|
||||
|
||||
In a flake configuration (NixOS, home-manager, nix-darwin):
|
||||
|
||||
```nix
|
||||
# flake.nix
|
||||
inputs.lightcraft.url = "github:storytold/lightcraft";
|
||||
# optional: build against your own nixpkgs instead of the one LightCraft pins
|
||||
# inputs.lightcraft.inputs.nixpkgs.follows = "nixpkgs";
|
||||
|
||||
# then, in a NixOS or home-manager module (where `inputs` is in scope):
|
||||
nixpkgs.overlays = [ inputs.lightcraft.overlays.default ]; # makes `pkgs.lightcraft` available
|
||||
environment.systemPackages = [ pkgs.lightcraft ]; # home-manager: home.packages = [ pkgs.lightcraft ];
|
||||
```
|
||||
|
||||
`nix build` installs the same desktop file, hicolor icons and AppStream metadata as the .deb/.rpm, and runs
|
||||
`cargo test --workspace` as its check phase (skip it with `pkgs.lightcraft.overrideAttrs { doCheck = false; }`).
|
||||
|
||||
**Keyboard:** <kbd>G</kbd> grid · <kbd>D</kbd> detail · <kbd>E</kbd> edit · <kbd>C</kbd> crop · <kbd>M</kbd> masking ·
|
||||
<kbd>Shift</kbd>+<kbd>P</kbd> presets · <kbd>\\</kbd> original · <kbd>Y</kbd> before/after · <kbd>Z</kbd> zoom ·
|
||||
<kbd>J</kbd> clipping · <kbd>⌘Z</kbd> undo · <kbd>⌘/</kbd> all shortcuts.
|
||||
|
|
|
|||
30
ROADMAP.md
30
ROADMAP.md
|
|
@ -23,8 +23,8 @@ duplicate Local entries #22, black GPU exports on an Intel iGPU #78).
|
|||
| Dimension | Estimate | What's true today | Biggest gaps |
|
||||
|---|---:|---|---|
|
||||
| **Feature checklist** | 79% | P0 core and P1 nearly complete: import (Add / Copy / Move, templates, devices), library, grid/loupe/compare/survey, every Edit slider, curves, colour grading, masking tools, crop/Upright, heal/clone, presets/profiles, versions/history, sync, export, menus, shortcuts | P1: lens-profile database, content-aware fill (patch synthesis), video playback/trim |
|
||||
| **RAW coverage** (formats people shoot) | ~50% | DNG (all kinds), CR2, ARW, NEF (uncompressed + Huffman lossless/lossy), uncompressed RAF/ORF, packed RW2, PEF; every container's embedded preview (incl. CR3) | **CR3** (every Canon since ~2018), compressed RAF/ORF, RW2 v4, Nikon lossy-after-split, Canon sRAW, HEIC/AVIF. NEFs labelled compressed but stored uncompressed (Z 6 packed 14-bit, D850 12-bit uncompressed). Per-model verification is thin (~40 corpus files vs >1,000 models) |
|
||||
| **Colour & image quality** | ~55–65% | Pipeline is complete and fast; GPU path CPU-exact within 1/255 | **No measured camera calibration database**: ARW and NEF have a guarded per-file embedded-JPEG colour estimate (docs/camera-preview-colour.md); other non-DNG raws and rejected estimates use a neutral matrix. Colour fidelity remains incomplete. No lens-profile database. No measured fidelity against Lightroom (tone, highlights, texture/clarity, NR, sharpening are tuned by eye) |
|
||||
| **RAW coverage** (formats people shoot) | ~55% | DNG (all kinds), CR2, ARW, NEF (uncompressed + Huffman lossless/lossy), uncompressed RAF/ORF, RW2 / RWL / Panasonic RAW (every raw format, checked on 178 files from 118 Panasonic and Leica bodies), PEF; every container's embedded preview (incl. CR3) | **CR3** (every Canon since ~2018), compressed RAF/ORF, Nikon lossy-after-split, Canon sRAW, HEIC/AVIF. NEFs labelled compressed but stored uncompressed (Z 6 packed 14-bit, D850 12-bit uncompressed). Per-model verification is thin (~55 corpus files vs >1,000 models) |
|
||||
| **Colour & image quality** | ~55–65% | Pipeline is complete and fast; GPU path CPU-exact within 1/255 | **No measured camera calibration database**: ARW, NEF and RW2 have a guarded per-file embedded-JPEG colour estimate (docs/camera-preview-colour.md); other non-DNG raws and rejected estimates use a neutral matrix. Colour fidelity remains incomplete. No lens-profile database. No measured fidelity against Lightroom (tone, highlights, texture/clarity, NR, sharpening are tuned by eye) |
|
||||
| **AI & computational** | ~15–20% | Assisted culling (focus, bursts), auto tone, HDR/panorama merge; subject/sky/background masks as classical heuristics | Real segmentation masks (subject, sky, people, objects, landscape, depth), AI denoise, super resolution, lens blur, generative remove, faces/people, natural-language search. **Blocked on a model strategy** (licensable weights or our own training; pure-Rust inference is feasible) |
|
||||
| **Workflow & library** | ~85% (single machine) | Robust catalog (journal + snapshots, background compaction, crash-tested), 85k-photo libraries stay responsive, Local browsing with automatic cleanup, XMP interop, keywords, smart albums, Move import | Opening an 85k library takes 1.7–4.7 s; no cloud sync (out of scope), no tablet companion (#74, roadmap), shared albums, publish services, tethering |
|
||||
| **Classic modules** | ~30% | Geotagging from GPX track logs, soft proofing (partial), slideshow (basic) | **Map view, Book, Print, Slideshow module, Web, publish services**: ~40 tracker rows ⬜ |
|
||||
|
|
@ -36,7 +36,7 @@ duplicate Local entries #22, black GPU exports on an Intel iGPU #78).
|
|||
| User | Readiness | What blocks them |
|
||||
|---|---:|---|
|
||||
| JPEG / DNG shooter, single machine | ~85% | Fidelity polish, AI masks |
|
||||
| Nikon / Sony / older-Canon raw shooter | ~65% | Camera colour fidelity and coverage (ARW and NEF preview estimates are only a starting point) |
|
||||
| Nikon / Sony / Panasonic / older-Canon raw shooter | ~65% | Camera colour fidelity and coverage (ARW, NEF and RW2 preview estimates are only a starting point); Panasonic compacts and kit zooms also miss their embedded distortion correction |
|
||||
| Canon CR3 / Fujifilm / Olympus shooter | ~35% | Their raws open as embedded previews only (CR3: the full-size JPEG with full metadata) |
|
||||
| Lightroom Classic power user | ~45% | Print, Book, Map, publish, tethering |
|
||||
| Relies on AI (masks, denoise) | ~25% | Object / Describe masks via optional SAM 3 (download not yet hosted); no AI denoise |
|
||||
|
|
@ -46,9 +46,9 @@ duplicate Local entries #22, black GPU exports on an Intel iGPU #78).
|
|||
Priorities, in order. Each points at tracker rows in [`docs/parity.md`](docs/parity.md) → *Top gaps*.
|
||||
|
||||
1. **Camera colour calibration of our own** (LR-PROF-CAMERACOLOR, P0): fit each camera to its own embedded JPEG, use
|
||||
matrices the files carry themselves, then chart shots. Sony ARW and Nikon NEF have the file-local fit (matrix + tone
|
||||
curve from their own JPEG); generalise it to the other makes' raws (RW2, PEF, ORF…), then validate fidelity.
|
||||
2. **Raw formats, clean-room** (LR-IMP-FORMATS, P0): **CR3** first, then compressed RAF / ORF, RW2 v4, NEF
|
||||
matrices the files carry themselves, then chart shots. Sony ARW, Nikon NEF and Panasonic RW2 have the file-local fit
|
||||
(matrix + tone curve from their own JPEG); generalise it to the other makes' raws (PEF, ORF…), then validate fidelity.
|
||||
2. **Raw formats, clean-room** (LR-IMP-FORMATS, P0): **CR3** first, then compressed RAF / ORF, NEF
|
||||
lossy-after-split, sRAW. Decided 2026-10-05: write our own decoders from prose descriptions (never decoder source,
|
||||
no LGPL dependency); compressed NEF (#86) is the template.
|
||||
3. **Verified camera coverage** (LR-IMP-CAMERA-COVERAGE, P0): a CC0 sample per model in the corpus, each decoded and
|
||||
|
|
@ -84,7 +84,7 @@ hardening (#78), copyright metadata (#51), GPX geotagging (#60), import tag help
|
|||
| M8 | Heal / Remove | content-aware remove (PatchMatch), heal, clone, brush spots, visualize spots, red/pet eye | 6–10 | 🚧 (heal, clone, auto source, visualize spots, red/pet eye ✅; PatchMatch remove ⬜) |
|
||||
| M9 | Presets, profiles, versions, sync | preset browser + amount, create/import presets, profile browser, versions, history, copy/paste/sync settings | 5–8 | ✅ |
|
||||
| M10 | Export & share | export dialog (JPEG/PNG/TIFF/DNG/AVIF/JXL/original), sizing, sharpening, metadata, watermark, naming, batch jobs, XMP sidecars, HDR export | 6–10 | 🚧 (all formats incl. DNG/original, sizing, presets, background jobs ✅; JXL encode, HDR export ⬜) |
|
||||
| M11 | RAW II | CR3, RAF (X-Trans), ORF, RW2, PEF, SRW, 3FR, IIQ + long tail; camera calibration DB; HEIC/AVIF/JXL import | 20–35 | 🚧 (RAF uncompressed, RW2 packed, PEF, ORF uncompressed ✅; **camera colour calibration** 🚧 (guarded ARW and NEF preview fitting; measured database still missing), CR3, compressed ORF/RAF ⬜) |
|
||||
| M11 | RAW II | CR3, RAF (X-Trans), ORF, RW2, PEF, SRW, 3FR, IIQ + long tail; camera calibration DB; HEIC/AVIF/JXL import | 20–35 | 🚧 (RAF uncompressed, RW2 (every raw format), PEF, ORF uncompressed ✅; **camera colour calibration** 🚧 (guarded ARW, NEF and RW2 preview fitting; measured database still missing), CR3, compressed ORF/RAF ⬜) |
|
||||
| M12 | AI & smart features | subject/sky/background/people/object masks, semantic search, faces/People (permissively licensed models, pure-Rust inference) | 20–40 | ⬜ |
|
||||
| M13 | Merge | HDR merge (deghost), panorama (projections, boundary warp, fill edges), HDR panorama | 10–15 | ✅ |
|
||||
| M14 | Video | import/playback/trim via FilmCraft crates, global edits + presets on video, video export | 6–10 | ⬜ |
|
||||
|
|
@ -118,7 +118,7 @@ trip, slideshow, auto import…) in ≈ 12 h on 2026-10-02; four to six parallel
|
|||
| Work package | Tracker rows | Agent-hours | Risk |
|
||||
|---|---|---:|---|
|
||||
| Remaining P0/P1 UI and library features (folder rename/move, keyword painter, people view…) | ≈ 8 | 5–10 | low |
|
||||
| Raw codecs: CR3 (CRX), compressed ORF / RAF, NEF lossy-after-split, RW2 v4, HEIC/AVIF decode, JPEG XL DNG | LR-IMP-FORMATS | 40–80 | **high** — clean-room black-box analysis, no permissive specs |
|
||||
| Raw codecs: CR3 (CRX), compressed ORF / RAF, NEF lossy-after-split, HEIC/AVIF decode, JPEG XL DNG | LR-IMP-FORMATS | 40–80 | **high** — clean-room black-box analysis, no permissive specs |
|
||||
| Lens-profile database of our own (calibration targets, fitting, data) | LR-EDIT-OPTICS-PROFILE | 15–30 | data collection |
|
||||
| Video: playback, trim, edits, export (pure-Rust decode, ideally shared with FilmCraft) | R. Video | 20–40 | medium |
|
||||
| AI: subject / sky / background / people / object masks, object-aware remove, AI denoise, super resolution, lens blur, people & faces, natural-language search, culling | ≈ 30 | 80–150 | **high** — permissively licensed weights, pure-Rust inference, maybe training |
|
||||
|
|
@ -165,7 +165,9 @@ The milestone estimates in the table above were made before work started and are
|
|||
Decoded (CC0 corpus from raw.pixls.us, `cargo xtask corpus --download`, `crates/raw/tests/corpus.rs`): DNG (uncompressed,
|
||||
LJ92, lossy JPEG / Smart Previews, Deflate, float, linear), CR2, ARW (uncompressed, ARW2, LJ92; as-shot white balance and black level of pre-2017 bodies from the enciphered
|
||||
maker-note `Tag2010` and the encrypted `SR2SubIFD`, both recovered by black-box analysis, `crates/raw/src/vendor/arw.rs`), NEF/NRW uncompressed and Huffman-compressed (lossless, lossy type 1/2, 12/14-bit), RAF uncompressed (Bayer and
|
||||
X-Trans), RW2 packed 12/14-bit, PEF (uncompressed and Huffman), ORF uncompressed (16-bit and 12-bit packed). Every
|
||||
X-Trans), RW2 / Leica RWL / Panasonic RAW in every raw format (compressed formats 4 and 6, the prefix-coded strips of format 8,
|
||||
packed formats 2, 5 and 7, the 16-bit words of the 2005–2007 bodies; all recovered by black-box analysis of 178 CC0 files from
|
||||
118 bodies, `crates/raw/src/vendor/rw2.rs`), PEF (uncompressed and Huffman), ORF uncompressed (16-bit and 12-bit packed). Every
|
||||
supported container also yields its embedded JPEG preview (CR3 too), and the engine shows that preview for raw variants
|
||||
it can't decode yet.
|
||||
|
||||
|
|
@ -174,14 +176,11 @@ Not decoded yet — preview only (no permissively licensed description; black-bo
|
|||
files): rows above the split decode with the regular lossy table; from the split row on a different code is used
|
||||
that our black-box analysis has not recovered yet (none of the four regular tables fits, also not with byte
|
||||
alignment or reset predictors at the split row). The other Nikon Huffman variants are decoded (`crates/raw/src/vendor/nefc.rs` documents the analysis).
|
||||
- **Panasonic RW2 raw format 4** (quantised): the block layout is known (0x4000-byte chunks rotated by 0x1ff8; 128-bit
|
||||
blocks of two 12-bit seeds plus four groups of a 2-bit scale and three 8-bit codes; scales 0/1 are ×1/×2 differences
|
||||
from the same-colour pixel two to the left), the reconstruction rule for scales 2/3 is not established.
|
||||
- **Olympus compressed ORF**, **Fujifilm compressed RAF**, **Canon CR3/CRX** (M11.1), **Canon sRAW/mRAW**, lossy DNG.
|
||||
|
||||
**Camera colour matrices:** ARW files can use guarded, separate chromaticity and tone estimates from their own embedded JPEG (see `docs/camera-preview-colour.md`); this is a per-file camera-look estimate with relative WB, not measured calibration or absolute-Kelvin WB. Other non-DNG raws and rejected fits use the documented neutral fallback (camera RGB ≈ linear sRGB, flagged
|
||||
**Camera colour matrices:** ARW, NEF and RW2 files can use guarded, separate chromaticity and tone estimates from their own embedded JPEG (see `docs/camera-preview-colour.md`); this is a per-file camera-look estimate with relative WB, not measured calibration or absolute-Kelvin WB. Other non-DNG raws and rejected fits use the documented neutral fallback (camera RGB ≈ linear sRGB, flagged
|
||||
`matrix_is_fallback`) with the file's as-shot white-balance multipliers. Clean sources to evaluate next: manufacturer
|
||||
matrices stored in the files themselves (Olympus ImageProcessing `ColorMatrix`, Pentax/Panasonic equivalents) and our
|
||||
matrices stored in the files themselves (Olympus ImageProcessing `ColorMatrix`, Pentax equivalents; Panasonic RW2 files carry none) and our
|
||||
own chart-based calibration (M11.4). Adobe matrices are never used.
|
||||
|
||||
## Log
|
||||
|
|
@ -193,6 +192,9 @@ own chart-based calibration (M11.4). Adobe matrices are never used.
|
|||
Local roots and cleanup, 85k-photo grid and catalog performance, background compaction, failed-save errors, GPU
|
||||
export hardening, clippy 1.99. Added the honest *Where we stand* assessment and *Where we're going* priorities; added
|
||||
tracker rows for camera colour, camera coverage and render fidelity.
|
||||
- 2026-10-07: Panasonic RW2 / Leica RWL / Panasonic RAW decode in every raw format (compressed 4/6/8, packed 2/5/7,
|
||||
16-bit words), black levels, in-camera aspect crops and mapped-out defects established on 178 CC0 files from 118
|
||||
bodies; RW2 gets the file-local camera look (accepted on 140 of 174 files with a preview).
|
||||
|
||||
## Chinese and Japanese interfaces, and text watermarks
|
||||
|
||||
|
|
|
|||
|
|
@ -292,7 +292,9 @@ fn devices_are_listed_and_imported_from() {
|
|||
assert_eq!(dev["name"], "CARD", "{line}");
|
||||
// the import review of its DCIM folder finds the photo
|
||||
let dcim = dev["path"].as_str().unwrap().to_string();
|
||||
let o = Command::new(BIN).args(["run", "library.importPreview", &format!("paths={}", serde_json::json!([dcim]))]).output().unwrap();
|
||||
// JSON-escaped: a Windows path has backslashes
|
||||
let paths = serde_json::to_string(&[&dcim]).unwrap();
|
||||
let o = Command::new(BIN).args(["run", "library.importPreview", &format!("paths={paths}")]).output().unwrap();
|
||||
let line: Value = serde_json::from_slice(o.stdout.split(|b| *b == b'\n').next().unwrap()).unwrap();
|
||||
assert_eq!(line["result"]["candidates"].as_array().map(Vec::len), Some(1), "{line}");
|
||||
}
|
||||
|
|
|
|||
|
|
@ -428,8 +428,9 @@ impl WebApp {
|
|||
let cores = window().map_or(1, |w| w.navigator().hardware_concurrency() as usize);
|
||||
cores.saturating_sub(1).clamp(1, 4)
|
||||
});
|
||||
let cache = app.session.media.rendered.clone();
|
||||
let workers =
|
||||
(n > 0).then(|| Workers::start(n, backend.as_ref().map_or("memory", |b| b.kind()), backend.clone(), index, cc.egui_ctx.clone()));
|
||||
(n > 0).then(|| Workers::start(n, backend.as_ref().map_or("memory", |b| b.kind()), backend.clone(), index, &cache, cc.egui_ctx.clone()));
|
||||
if let Some(w) = &workers {
|
||||
app.renderer.set_offload(Box::new(w.clone()));
|
||||
}
|
||||
|
|
|
|||
|
|
@ -11,13 +11,13 @@
|
|||
//! ([`ThumbIndex`]).
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::sync::Arc;
|
||||
use std::sync::{Arc, Weak};
|
||||
|
||||
use lightcraft_engine::catalog::Source;
|
||||
use lightcraft_engine::develop::{DevelopSettings, EmbeddedLens};
|
||||
use lightcraft_engine::media::{DecodedSource, MediaCache, RenderJob, SourceLevel, SourceRef};
|
||||
use lightcraft_engine::pipeline::{Quality, RenderRequest, Rendered, SourceInfo, StageCache};
|
||||
use lightcraft_preview::{Hash128, Lru};
|
||||
use lightcraft_preview::{Hash128, Lru, PreviewCache};
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
use crate::store::hash_of_path;
|
||||
|
|
@ -299,6 +299,33 @@ impl ThumbIndex {
|
|||
}
|
||||
}
|
||||
|
||||
/// Which rendered-thumbnail cache, at which generation, the stored thumbnails ([`ThumbIndex`])
|
||||
/// belong to (main thread). Clearing the previews bumps the cache's generation, and opening
|
||||
/// another library replaces the cache: either makes every stored thumbnail obsolete.
|
||||
pub struct CacheWatch(Option<(Weak<PreviewCache>, u64)>);
|
||||
|
||||
impl CacheWatch {
|
||||
/// Watch `active`, the session's cache when the workers start: the persisted index belongs
|
||||
/// to it at its current generation. (Without this baseline, the first observation after a
|
||||
/// clear would only record the cleared generation and keep the obsolete thumbnails.)
|
||||
pub fn new(active: &Arc<PreviewCache>) -> CacheWatch {
|
||||
CacheWatch(Some((Arc::downgrade(active), active.generation())))
|
||||
}
|
||||
|
||||
/// The watched cache, while it's alive.
|
||||
pub fn cache(&self) -> Option<Arc<PreviewCache>> {
|
||||
self.0.as_ref().and_then(|(cache, _)| cache.upgrade())
|
||||
}
|
||||
|
||||
/// `cache` at `generation` is the active one now. If it was cleared or replaced since the
|
||||
/// last observation, invalidate `index` and return the keys whose files should be deleted.
|
||||
pub fn observe(&mut self, cache: &Arc<PreviewCache>, generation: u64, index: &mut ThumbIndex) -> Vec<String> {
|
||||
let changed = self.0.as_ref().is_some_and(|(old, g)| old.as_ptr() != Arc::as_ptr(cache) || *g != generation);
|
||||
self.0 = Some((Arc::downgrade(cache), generation));
|
||||
if changed { index.invalidate() } else { Vec::new() }
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
|
@ -381,6 +408,62 @@ mod tests {
|
|||
assert_eq!(core.needs_original(&refreshed).as_deref(), Some(hash.as_str()), "explicit refresh must not reuse the old decoded source");
|
||||
}
|
||||
|
||||
/// The persisted index as a restarted page loads it: one stored thumbnail.
|
||||
fn persisted_index(key: Hash128) -> (ThumbIndex, String) {
|
||||
let mut index = ThumbIndex::default();
|
||||
let hex = index.cache_key(key);
|
||||
index.insert(&hex, 100);
|
||||
(ThumbIndex::from_json(&index.to_json()), hex)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn clearing_previews_before_the_first_request_invalidates_stored_thumbnails() {
|
||||
let key = lightcraft_preview::hash_bytes(b"thumbnail");
|
||||
let (mut index, old) = persisted_index(key);
|
||||
let cache = Arc::new(PreviewCache::memory(1 << 20));
|
||||
let mut watch = CacheWatch::new(&cache);
|
||||
let mut gone = Vec::new();
|
||||
// each frame (`Workers::finished`) observes the watched cache, even with no requests
|
||||
let mut frame = |watch: &mut CacheWatch, index: &mut ThumbIndex| {
|
||||
if let Some(c) = watch.cache() {
|
||||
gone.extend(watch.observe(&c, c.generation(), index));
|
||||
}
|
||||
};
|
||||
frame(&mut watch, &mut index); // an empty filtered view: no render request yet
|
||||
cache.clear(); // File ▸ Clear Preview Cache
|
||||
frame(&mut watch, &mut index);
|
||||
// the filter is removed: the first thumbnail request (`Workers::try_start`)
|
||||
gone.extend(watch.observe(&cache, cache.generation(), &mut index));
|
||||
let disk_key = index.cache_key(key);
|
||||
assert!(!index.touch(&disk_key), "the pre-clear thumbnail must not be read back");
|
||||
assert_eq!(gone, vec![old.clone()], "the stored file is deleted");
|
||||
assert_ne!(disk_key, old, "a fresh namespace: a late old write can't come back");
|
||||
assert!(index.dirty, "the invalidated index is saved");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn replacing_the_cache_before_the_first_request_invalidates_stored_thumbnails() {
|
||||
let key = lightcraft_preview::hash_bytes(b"thumbnail");
|
||||
let (mut index, old) = persisted_index(key);
|
||||
let first = Arc::new(PreviewCache::memory(1 << 20));
|
||||
let mut watch = CacheWatch::new(&first);
|
||||
let other = Arc::new(PreviewCache::memory(1 << 20)); // another library, generation 0 again
|
||||
assert_eq!(watch.observe(&other, other.generation(), &mut index), vec![old]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_normal_start_keeps_stored_thumbnails() {
|
||||
let key = lightcraft_preview::hash_bytes(b"thumbnail");
|
||||
let (mut index, old) = persisted_index(key);
|
||||
let cache = Arc::new(PreviewCache::memory(1 << 20));
|
||||
let mut watch = CacheWatch::new(&cache);
|
||||
for _ in 0..3 {
|
||||
assert!(watch.observe(&cache, cache.generation(), &mut index).is_empty());
|
||||
}
|
||||
assert_eq!(index.cache_key(key), old);
|
||||
assert!(index.touch(&old), "persisted thumbnails survive a restart");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn thumb_index_prunes_lru() {
|
||||
let mut i = ThumbIndex::default();
|
||||
|
|
|
|||
|
|
@ -27,7 +27,7 @@ use wasm_bindgen::JsCast;
|
|||
use wasm_bindgen::prelude::*;
|
||||
|
||||
use crate::backend::Backend;
|
||||
use crate::wire::{ThumbIndex, WireJob, WorkerCore, thumb_storage_key};
|
||||
use crate::wire::{CacheWatch, ThumbIndex, WireJob, WorkerCore, thumb_storage_key};
|
||||
|
||||
/// Storage key of the thumbnail index.
|
||||
pub const THUMB_INDEX: &str = "thumbs/index.json";
|
||||
|
|
@ -175,7 +175,8 @@ struct W {
|
|||
}
|
||||
|
||||
struct Inner {
|
||||
preview_generation: Option<(std::sync::Weak<PreviewCache>, u64)>,
|
||||
/// The rendered-thumbnail cache the stored thumbnails belong to.
|
||||
preview_generation: CacheWatch,
|
||||
workers: Vec<W>,
|
||||
done: Vec<(Slot, RenderResult, f64)>,
|
||||
next_id: u32,
|
||||
|
|
@ -192,17 +193,16 @@ struct Inner {
|
|||
|
||||
impl Inner {
|
||||
fn observe_cache(&mut self, cache: &Arc<PreviewCache>, generation: u64) {
|
||||
if self.preview_generation.as_ref().is_some_and(|(old, g)| old.as_ptr() != Arc::as_ptr(cache) || *g != generation) {
|
||||
let gone = self.index.invalidate();
|
||||
if let Some(backend) = self.backend.clone() {
|
||||
wasm_bindgen_futures::spawn_local(async move {
|
||||
for key in gone {
|
||||
let _ = backend.remove(&thumb_storage_key(&key)).await;
|
||||
}
|
||||
});
|
||||
}
|
||||
let gone = self.preview_generation.observe(cache, generation, &mut self.index);
|
||||
if !gone.is_empty()
|
||||
&& let Some(backend) = self.backend.clone()
|
||||
{
|
||||
wasm_bindgen_futures::spawn_local(async move {
|
||||
for key in gone {
|
||||
let _ = backend.remove(&thumb_storage_key(&key)).await;
|
||||
}
|
||||
});
|
||||
}
|
||||
self.preview_generation = Some((Arc::downgrade(cache), generation));
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -212,9 +212,11 @@ pub struct Workers(Rc<RefCell<Inner>>);
|
|||
|
||||
impl Workers {
|
||||
/// Start `n` workers (none: every job runs inline). `store` is the backend kind for workers.
|
||||
pub fn start(n: usize, store: &str, backend: Option<Backend>, index: ThumbIndex, ctx: egui::Context) -> Workers {
|
||||
/// `index` lists the stored thumbnails of `cache`, the session's rendered-thumbnail cache
|
||||
/// now: watching it from the start, a clear before the first render request still counts.
|
||||
pub fn start(n: usize, store: &str, backend: Option<Backend>, index: ThumbIndex, cache: &Arc<PreviewCache>, ctx: egui::Context) -> Workers {
|
||||
let w = Workers(Rc::new(RefCell::new(Inner {
|
||||
preview_generation: None,
|
||||
preview_generation: CacheWatch::new(cache),
|
||||
workers: Vec::new(),
|
||||
done: Vec::new(),
|
||||
next_id: 0,
|
||||
|
|
@ -471,7 +473,7 @@ impl RenderOffload for Workers {
|
|||
fn finished(&mut self) -> Vec<(Slot, RenderResult, f64)> {
|
||||
let mut g = self.0.borrow_mut();
|
||||
// Persist invalidation even when the cleared library has no visible render requests.
|
||||
if let Some(cache) = g.preview_generation.as_ref().and_then(|(cache, _)| cache.upgrade()) {
|
||||
if let Some(cache) = g.preview_generation.cache() {
|
||||
g.observe_cache(&cache, cache.generation());
|
||||
}
|
||||
g.inline_this_frame = false;
|
||||
|
|
|
|||
|
|
@ -286,8 +286,8 @@ fn services() -> Services {
|
|||
.add_filter(
|
||||
lightcraft_ui_egui::i18n::tr("Photos"),
|
||||
&[
|
||||
"jpg", "jpeg", "png", "tif", "tiff", "webp", "dng", "cr2", "cr3", "nef", "nrw", "arw", "raf", "orf", "rw2", "pef", "psd",
|
||||
"jxl", "gif", "bmp",
|
||||
"jpg", "jpeg", "png", "tif", "tiff", "webp", "dng", "cr2", "cr3", "nef", "nrw", "arw", "raf", "orf", "rw2", "rwl", "raw",
|
||||
"pef", "psd", "jxl", "gif", "bmp",
|
||||
],
|
||||
)
|
||||
.pick_files()
|
||||
|
|
|
|||
|
|
@ -336,11 +336,12 @@ impl Photo {
|
|||
pub fn develops_raw(&self) -> bool {
|
||||
self.kind == MediaKind::Raw && self.preview_only.is_none()
|
||||
}
|
||||
/// The current ARW and NEF readers have vendor WB multipliers but no measured camera
|
||||
/// The current ARW, NEF and RW2 readers have vendor WB multipliers but no measured camera
|
||||
/// illuminant. Use adjustments relative to the camera's as-shot look, as for rendered
|
||||
/// photographs (the engine's `camera_preview::file_local_look` covers the same formats).
|
||||
/// photographs (the engine's `camera_preview::file_local_look` covers the same formats; RWL and
|
||||
/// RAW are Leica's and the oldest Panasonic bodies' names for RW2 files).
|
||||
pub fn relative_wb(&self) -> bool {
|
||||
self.develops_raw() && ["ARW", "NEF", "NRW"].iter().any(|f| self.format.eq_ignore_ascii_case(f))
|
||||
self.develops_raw() && ["ARW", "NEF", "NRW", "RW2", "RWL", "RAW"].iter().any(|f| self.format.eq_ignore_ascii_case(f))
|
||||
}
|
||||
/// The develop settings import gave this photo: [`Photo::camera_defaults`], or the user's
|
||||
/// default preset applied on top of them ([`Photo::import_look`]).
|
||||
|
|
@ -430,7 +431,15 @@ mod edited_tests {
|
|||
|
||||
#[test]
|
||||
fn sony_and_nikon_raws_use_relative_white_balance() {
|
||||
for (name, format, relative) in [("a.arw", "ARW", true), ("a.nef", "NEF", true), ("a.nrw", "nrw", true), ("a.dng", "DNG", false)] {
|
||||
for (name, format, relative) in [
|
||||
("a.arw", "ARW", true),
|
||||
("a.nef", "NEF", true),
|
||||
("a.nrw", "nrw", true),
|
||||
("a.rw2", "RW2", true),
|
||||
("a.rwl", "RWL", true),
|
||||
("a.raw", "RAW", true),
|
||||
("a.dng", "DNG", false),
|
||||
] {
|
||||
let mut p = Photo::new(PhotoId(1), Source::Demo { scene: 0 }, name, format, 10, 10, "2026-10-01T00:00:00");
|
||||
p.kind = MediaKind::Raw;
|
||||
p.as_shot_wb = Some((5200.0, 4.0));
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
//! Estimate the starting look of a raw without a camera colour matrix (Sony ARW, Nikon NEF) from its
|
||||
//! own JPEG. Colour and luminance are fitted separately; the JPEG supplies correspondences only,
|
||||
//! Estimate the starting look of a raw without a camera colour matrix (Sony ARW, Nikon NEF, Panasonic
|
||||
//! RW2) from its own JPEG. Colour and luminance are fitted separately; the JPEG supplies correspondences only,
|
||||
//! never output pixels or a replacement for RAW editing.
|
||||
//! A global matrix can't follow the camera's hue-dependent rendering (the best matrix rendered a
|
||||
//! lime shirt olive that the camera kept lime): a hue/saturation table fitted to the residuals
|
||||
|
|
@ -31,7 +31,7 @@ pub(crate) const PROFILE_PROXY: usize = 192;
|
|||
/// as-shot look (`docs/camera-preview-colour.md`). The catalog's `Photo::relative_wb` matches the
|
||||
/// same formats by file extension.
|
||||
pub(crate) fn file_local_look(format: RawFormat) -> bool {
|
||||
matches!(format, RawFormat::Arw | RawFormat::Nef | RawFormat::Nrw)
|
||||
matches!(format, RawFormat::Arw | RawFormat::Nef | RawFormat::Nrw | RawFormat::Rw2)
|
||||
}
|
||||
|
||||
pub(crate) fn fit_preview(raw: &RawImage, bytes: &[u8], transform: &CameraTransform) -> Option<CameraLook> {
|
||||
|
|
@ -63,14 +63,24 @@ fn proxies(raw: &RawImage, bytes: &[u8], transform: &CameraTransform, size: usiz
|
|||
let jpeg = lightcraft_raw::embedded_preview(bytes)?;
|
||||
let edge = (2 * size).max(384) as u32;
|
||||
let decoded = lightcraft_codecs::decode(&jpeg, lightcraft_codecs::DecodeOptions { max_size: Some((edge, edge)), max_pixels: 64_000_000 }).ok()?;
|
||||
let reference = decoded.to_working();
|
||||
let crop = raw.crop.clipped(raw.active_area.width, raw.active_area.height);
|
||||
let mut reference = decoded.to_working();
|
||||
let (a, crop) = (raw.active_area, raw.crop.clipped(raw.active_area.width, raw.active_area.height));
|
||||
if crop.width == 0 || crop.height == 0 || reference.width == 0 || reference.height == 0 {
|
||||
return None;
|
||||
}
|
||||
let aspect = crop.width as f64 / crop.height as f64;
|
||||
if (reference.width as f64 / reference.height as f64 / aspect - 1.0).abs() > 0.02 {
|
||||
return None;
|
||||
let matches = |w: usize, h: usize, rw: usize, rh: usize| (rw as f64 / rh as f64 / (w as f64 / h as f64) - 1.0).abs() <= 0.02;
|
||||
if !matches(crop.width, crop.height, reference.width, reference.height) {
|
||||
// Panasonic previews show the whole active area while the default crop is the in-camera aspect ratio
|
||||
if !matches(a.width, a.height, reference.width, reference.height) {
|
||||
return None;
|
||||
}
|
||||
let (sx, sy) = (reference.width as f64 / a.width as f64, reference.height as f64 / a.height as f64);
|
||||
let (x, y) = ((crop.x as f64 * sx).round() as usize, (crop.y as f64 * sy).round() as usize);
|
||||
let (w, h) = ((crop.width as f64 * sx).round() as usize, (crop.height as f64 * sy).round() as usize);
|
||||
if w < 16 || h < 16 || x + w > reference.width || y + h > reference.height {
|
||||
return None;
|
||||
}
|
||||
reference = reference.into_crop(x, y, w, h);
|
||||
}
|
||||
// Fixed, bounded proxy: the selected look cannot depend on thumbnail/export resolution.
|
||||
let k = (crop.width.max(crop.height).div_ceil(edge as usize).max(2)).div_ceil(2) * 2;
|
||||
|
|
@ -731,8 +741,8 @@ mod tests {
|
|||
}
|
||||
|
||||
#[test]
|
||||
fn sony_and_nikon_raws_get_a_file_local_look() {
|
||||
assert!([RawFormat::Arw, RawFormat::Nef, RawFormat::Nrw].into_iter().all(file_local_look));
|
||||
fn sony_nikon_and_panasonic_raws_get_a_file_local_look() {
|
||||
assert!([RawFormat::Arw, RawFormat::Nef, RawFormat::Nrw, RawFormat::Rw2].into_iter().all(file_local_look));
|
||||
assert!(![RawFormat::Dng, RawFormat::Cr2, RawFormat::Raf].into_iter().any(file_local_look));
|
||||
}
|
||||
|
||||
|
|
@ -753,6 +763,24 @@ mod tests {
|
|||
assert!(info.relative_wb && info.as_shot_temp == 6500.0 && info.as_shot_tint == 0.0);
|
||||
}
|
||||
|
||||
/// A public DC-FZ1000 II RW2 shot at 4:3 on its 3:2 sensor (skipped without the corpus): the default crop is 4:3
|
||||
/// while the embedded JPEG shows the whole sensor; the look is still fitted, against the matching part of it.
|
||||
#[test]
|
||||
fn corpus_rw2_with_an_in_camera_crop_gets_a_camera_look() {
|
||||
let path = std::env::var_os("LIGHTCRAFT_CORPUS")
|
||||
.map(std::path::PathBuf::from)
|
||||
.unwrap_or_else(|| std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("../../corpus"))
|
||||
.join("raw/rw2-panasonic-fz1000m2-4x3.rw2");
|
||||
let Ok(bytes) = std::fs::read(&path) else {
|
||||
eprintln!("skip: {} absent", path.display());
|
||||
return;
|
||||
};
|
||||
let (img, info) = crate::files::load_bytes(&bytes, 400).unwrap();
|
||||
assert_eq!((img.width, img.height), (400, 300), "framed in the in-camera aspect ratio");
|
||||
assert!(info.camera_tone.is_some(), "no camera look fitted");
|
||||
assert!(info.relative_wb && info.as_shot_temp == 6500.0 && info.as_shot_tint == 0.0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_monochrome_invalid_and_unrelated_previews() {
|
||||
let mut sensor = Rgb32f::new(32, 32);
|
||||
|
|
|
|||
|
|
@ -32,8 +32,8 @@ use crate::media::ProbeInfo;
|
|||
|
||||
/// File extensions LightCraft imports (lower case).
|
||||
pub const EXTENSIONS: &[&str] = &[
|
||||
"jpg", "jpeg", "png", "tif", "tiff", "webp", "dng", "cr2", "cr3", "nef", "nrw", "arw", "raf", "orf", "rw2", "pef", "psd", "jxl", "gif", "bmp",
|
||||
"heic", "avif",
|
||||
"jpg", "jpeg", "png", "tif", "tiff", "webp", "dng", "cr2", "cr3", "nef", "nrw", "arw", "raf", "orf", "rw2", "rwl", "raw", "pef", "psd", "jxl",
|
||||
"gif", "bmp", "heic", "avif",
|
||||
];
|
||||
|
||||
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
|
||||
|
|
|
|||
|
|
@ -244,8 +244,11 @@ fn source_bytes(img: &Rgb32f) -> usize {
|
|||
}
|
||||
|
||||
impl MediaCache {
|
||||
/// Keep rendered thumbnails on disk in `dir` as well.
|
||||
/// Keep rendered thumbnails on disk in `dir` as well. The cache this replaces is retired
|
||||
/// (its files stay): render jobs still holding it can't write through it any more, even
|
||||
/// into a directory a later clear emptied (`dir` may be the same one).
|
||||
pub fn attach_disk_cache(&mut self, dir: &std::path::Path, disk_bytes: u64) {
|
||||
self.rendered.retire();
|
||||
self.rendered = Arc::new(PreviewCache::with_disk(rendered_budget(self.budget), dir, disk_bytes));
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -774,12 +774,13 @@ mod tests {
|
|||
fail: RefCell::new(Vec::new()),
|
||||
}
|
||||
}
|
||||
/// The model volume spells paths with `/`; the code under test joins them with the host's separator.
|
||||
fn norm(p: &Path) -> String {
|
||||
p.to_string_lossy().replace('\\', "/")
|
||||
}
|
||||
fn idx(&self, p: &Path) -> Option<usize> {
|
||||
let p = p.to_string_lossy();
|
||||
self.files
|
||||
.borrow()
|
||||
.iter()
|
||||
.position(|(f, _)| if self.ci { folded_path(f) == folded_path(&p) } else { Path::new(f) == Path::new(p.as_ref()) })
|
||||
let p = Self::norm(p);
|
||||
self.files.borrow().iter().position(|(f, _)| if self.ci { folded_path(f) == folded_path(&p) } else { *f == p })
|
||||
}
|
||||
/// The listing: (exact path, contents), sorted.
|
||||
pub fn listing(&self) -> Vec<(String, String)> {
|
||||
|
|
@ -800,14 +801,14 @@ mod tests {
|
|||
self.idx(a).is_some() && self.idx(a) == self.idx(b)
|
||||
}
|
||||
fn rename_no_replace(&self, a: &Path, b: &Path) -> std::io::Result<()> {
|
||||
if self.fail.borrow().iter().any(|f| a.to_string_lossy().contains(f.as_str())) {
|
||||
if self.fail.borrow().iter().any(|f| Self::norm(a).contains(f.as_str())) {
|
||||
return Err(std::io::Error::other("injected failure"));
|
||||
}
|
||||
if self.exists(b) {
|
||||
return Err(std::io::Error::new(std::io::ErrorKind::AlreadyExists, "exists"));
|
||||
}
|
||||
let i = self.idx(a).ok_or_else(|| std::io::Error::new(std::io::ErrorKind::NotFound, "no such file"))?;
|
||||
self.files.borrow_mut()[i].0 = b.to_string_lossy().to_string();
|
||||
self.files.borrow_mut()[i].0 = Self::norm(b);
|
||||
Ok(())
|
||||
}
|
||||
fn copy_no_replace(&self, a: &Path, b: &Path) -> std::io::Result<()> {
|
||||
|
|
@ -816,7 +817,7 @@ mod tests {
|
|||
}
|
||||
let i = self.idx(a).ok_or_else(|| std::io::Error::new(std::io::ErrorKind::NotFound, "no such file"))?;
|
||||
let c = self.files.borrow()[i].1.clone();
|
||||
self.files.borrow_mut().push((b.to_string_lossy().to_string(), c));
|
||||
self.files.borrow_mut().push((Self::norm(b), c));
|
||||
Ok(())
|
||||
}
|
||||
fn remove_file(&self, p: &Path) -> std::io::Result<()> {
|
||||
|
|
@ -825,17 +826,19 @@ mod tests {
|
|||
Ok(())
|
||||
}
|
||||
fn stem_sibling(&self, sidecar: &Path) -> Option<String> {
|
||||
let stem = sidecar.with_extension("").to_string_lossy().to_string();
|
||||
let stem = Self::norm(&sidecar.with_extension(""));
|
||||
self.files.borrow().iter().map(|(f, _)| f.clone()).find(|f| {
|
||||
let p = Path::new(f);
|
||||
p.with_extension("").to_string_lossy() == stem && !p.extension().is_some_and(|e| e.eq_ignore_ascii_case("xmp"))
|
||||
Self::norm(&p.with_extension("")) == stem && !p.extension().is_some_and(|e| e.eq_ignore_ascii_case("xmp"))
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
fn file_photo(id: u64, path: &str) -> Photo {
|
||||
let name = Path::new(path).file_name().map(|n| n.to_string_lossy().to_string()).unwrap_or_default();
|
||||
Photo::new(PhotoId(id), Source::File { path: path.into() }, &name, "JPG", 1, 1, "2026-01-01T00:00:00")
|
||||
// the host's separator, so `dir.join(name)` in the planner spells the same path the same way
|
||||
let path = path.replace('/', std::path::MAIN_SEPARATOR_STR);
|
||||
Photo::new(PhotoId(id), Source::File { path }, &name, "JPG", 1, 1, "2026-01-01T00:00:00")
|
||||
}
|
||||
|
||||
#[cfg(windows)]
|
||||
|
|
@ -845,8 +848,11 @@ mod tests {
|
|||
// Any attempt to move the unchanged photo (including rollback) must fail.
|
||||
fs.fail.borrow_mut().push("IMG".into());
|
||||
let mut s = Session::new();
|
||||
// spelled with `/` on purpose (not `file_photo`, which uses the host's separator)
|
||||
for (id, path) in [(1, "N:/photos/IMG.jpg"), (2, "N:/photos/other.jpg")] {
|
||||
s.catalog.apply(Op::AddPhoto { photo: Box::new(file_photo(id, path)) }).unwrap();
|
||||
let name = Path::new(path).file_name().map(|n| n.to_string_lossy().to_string()).unwrap_or_default();
|
||||
let photo = Photo::new(PhotoId(id), Source::File { path: path.into() }, &name, "JPG", 1, 1, "2026-01-01T00:00:00");
|
||||
s.catalog.apply(Op::AddPhoto { photo: Box::new(photo) }).unwrap();
|
||||
}
|
||||
let before = fs.listing();
|
||||
let plans = s.plan_rename_with(&fs, &[PhotoId(1)], "{name}", 1);
|
||||
|
|
|
|||
|
|
@ -260,7 +260,7 @@ impl Segmenter {
|
|||
{
|
||||
let mirrors = self.mirrors();
|
||||
if mirrors.is_empty() {
|
||||
return Err(lightcraft_segment::fetch::DownloadError::NoMirrors.to_string());
|
||||
return Err(lightcraft_segment::fetch::no_mirrors_message());
|
||||
}
|
||||
self.download.start(lightcraft_segment::fetch::SAM3_FILES, mirrors, dir, lightcraft_segment::fetch::Options::default())
|
||||
}
|
||||
|
|
|
|||
|
|
@ -273,7 +273,7 @@ fn import_review_preview_and_options() {
|
|||
let mut s = Session::new().with_fs();
|
||||
s.open_library(&lib, false).unwrap();
|
||||
// c.png is already in the library
|
||||
s.execute("library.import", &json!({"paths": [src.join("sub/c.png").to_string_lossy()]})).unwrap();
|
||||
s.execute("library.import", &json!({"paths": [src.join("sub").join("c.png").to_string_lossy()]})).unwrap();
|
||||
let n0 = s.catalog.len();
|
||||
let undo0 = s.undo.len();
|
||||
let pv = s.execute("library.importPreview", &json!({"paths": [src.to_string_lossy()]})).unwrap();
|
||||
|
|
|
|||
|
|
@ -288,3 +288,32 @@ fn resizing_the_disk_cache_keeps_the_cache_and_its_thumbnails() {
|
|||
drop(s);
|
||||
let _ = std::fs::remove_dir_all(&lib);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn reopening_a_library_retires_the_old_cache_so_its_jobs_cannot_write_after_a_clear() {
|
||||
let lib = temp_dir("reopen-writer");
|
||||
let mut s = Session::new().with_fs();
|
||||
s.open_library(&lib, true).unwrap();
|
||||
let id = s.catalog.photos().next().unwrap().id;
|
||||
// a thumbnail job still holding the first open's cache object
|
||||
let job = s.thumb_job(id, 128).unwrap();
|
||||
let (old, key) = job.cache.clone().unwrap();
|
||||
// a thumbnail written before the reopen is still valid afterwards
|
||||
let kept = lightcraft_preview::hash_bytes(b"kept thumbnail");
|
||||
let img = lightcraft_raster::Rgba8 { width: 4, height: 4, data: vec![[10, 20, 30, 255]; 16] };
|
||||
old.put(kept, Arc::new(img.clone()));
|
||||
s.open_library(&lib, false).unwrap();
|
||||
assert!(!Arc::ptr_eq(&old, &s.media.rendered));
|
||||
assert!(s.media.rendered.get(kept).is_some(), "a plain reopen keeps the disk cache");
|
||||
s.execute("library.clearPreviews", &json!({})).unwrap();
|
||||
// the old job finishes: its render must not land in the cleared directory
|
||||
let r = job.run();
|
||||
assert!(r.rendered.is_ok());
|
||||
old.put(kept, Arc::new(img));
|
||||
let fresh = lightcraft_preview::PreviewCache::with_disk(1 << 20, &lib.join("thumbs"), 1 << 30);
|
||||
assert!(fresh.get(key).is_none(), "stale thumbnail written through the replaced cache");
|
||||
assert!(fresh.get(kept).is_none(), "stale put through the replaced cache");
|
||||
assert!(s.media.rendered.get(key).is_none());
|
||||
drop(s);
|
||||
let _ = std::fs::remove_dir_all(&lib);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -0,0 +1,17 @@
|
|||
[package]
|
||||
name = "lightcraft-fetch"
|
||||
version.workspace = true
|
||||
edition.workspace = true
|
||||
license = "Apache-2.0"
|
||||
description = "LightCraft model downloads: mirrors, resume, pinned SHA-256, over HTTPS in pure Rust (L0)"
|
||||
|
||||
[lints]
|
||||
workspace = true
|
||||
|
||||
# downloads run natively only (the web build fetches no models)
|
||||
[target.'cfg(not(target_arch = "wasm32"))'.dependencies]
|
||||
log = { workspace = true }
|
||||
rustls = { version = "0.23", default-features = false, features = ["std", "tls12"] }
|
||||
rustls-rustcrypto = "0.0.2-alpha"
|
||||
webpki-roots = "1"
|
||||
sha2 = { version = "0.10", default-features = false, features = ["std"] }
|
||||
|
|
@ -0,0 +1,394 @@
|
|||
//! Downloading a model's files, only when the user asked for it: from an ordered list of mirrors
|
||||
//! (the next one when a mirror fails), resuming partial files, with connect and stall timeouts,
|
||||
//! a size cap, progress and cancellation, and checked (exact size, SHA-256 where pinned) before
|
||||
//! each file is moved into the model folder. Runs on whatever thread calls [`download`] (the
|
||||
//! engine uses a background thread; the UI thread never waits for the network). Pure Rust: the
|
||||
//! HTTPS client is rustls with the RustCrypto provider (see [`http`]).
|
||||
//!
|
||||
//! Files are written to `<name>.part` next to their final place and renamed when verified, so a
|
||||
//! model folder never holds a half-written file under a real name, and an interrupted download
|
||||
//! resumes from where it stopped. A file that fails its check is deleted, never kept.
|
||||
//!
|
||||
//! What to download and where from belongs to the caller: it passes the [`FileSpec`]s and the
|
||||
//! default mirrors (see [`mirrors`]). `lightcraft-segment` does so for the SAM 3 weights, which
|
||||
//! are never part of LightCraft (SAM License, see docs/ai-masks.md).
|
||||
//!
|
||||
//! Native only: on wasm32 this crate is empty (the web build downloads no models).
|
||||
|
||||
#![cfg(not(target_arch = "wasm32"))]
|
||||
#![forbid(unsafe_code)]
|
||||
#![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable)]
|
||||
|
||||
pub mod http;
|
||||
|
||||
use std::fs::{File, OpenOptions};
|
||||
use std::io::{Read, Write};
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
use std::time::Duration;
|
||||
|
||||
use sha2::{Digest, Sha256};
|
||||
|
||||
use http::{HttpError, Limits, Url};
|
||||
|
||||
/// One file of the model.
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct FileSpec {
|
||||
pub name: &'static str,
|
||||
/// Exact size in bytes, when pinned (then the download resumes and is checked against it).
|
||||
pub size: Option<u64>,
|
||||
/// SHA-256 (lower-case hex), when pinned.
|
||||
pub sha256: Option<&'static str>,
|
||||
/// Largest size accepted when `size` is not pinned.
|
||||
pub max: u64,
|
||||
}
|
||||
|
||||
/// Most mirrors used (a hostile list can't make a download loop forever).
|
||||
const MAX_MIRRORS: usize = 16;
|
||||
/// Redirects followed per request.
|
||||
const MAX_REDIRECTS: usize = 8;
|
||||
|
||||
/// The mirrors to try, in order: `env`'s (base URLs separated by commas, spaces or newlines), then
|
||||
/// the mirrors file's (one base URL per line, `#` comments), then `defaults`; duplicates and
|
||||
/// unusable URLs dropped, at most 16.
|
||||
pub fn mirrors(env: Option<&str>, file: Option<&Path>, defaults: &[&str]) -> Vec<String> {
|
||||
let mut out: Vec<String> = Vec::new();
|
||||
let mut add = |s: &str| {
|
||||
let s = s.trim().trim_end_matches('/');
|
||||
if !s.is_empty() && Url::parse(s).is_ok() && !out.iter().any(|m| m == s) && out.len() < MAX_MIRRORS {
|
||||
out.push(s.to_string());
|
||||
}
|
||||
};
|
||||
if let Some(env) = env {
|
||||
env.split([',', ' ', '\n', '\t']).for_each(&mut add);
|
||||
}
|
||||
// a small text file; anything larger isn't a mirror list
|
||||
if let Some(text) = file.and_then(|f| std::fs::metadata(f).ok().filter(|m| m.len() < 64 * 1024).and_then(|_| std::fs::read_to_string(f).ok())) {
|
||||
text.lines().map(|l| l.split('#').next().unwrap_or_default()).for_each(&mut add);
|
||||
}
|
||||
defaults.iter().for_each(|m| add(m));
|
||||
out
|
||||
}
|
||||
|
||||
/// Download settings.
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct Options {
|
||||
/// Longest wait to connect to a mirror.
|
||||
pub connect_timeout: Duration,
|
||||
/// Longest wait for the next bytes once connected.
|
||||
pub stall_timeout: Duration,
|
||||
/// Tries per mirror and file (each resumes where the last one stopped).
|
||||
pub attempts: u32,
|
||||
}
|
||||
|
||||
impl Default for Options {
|
||||
fn default() -> Self {
|
||||
Options { connect_timeout: Duration::from_secs(15), stall_timeout: Duration::from_secs(30), attempts: 3 }
|
||||
}
|
||||
}
|
||||
|
||||
/// Where a download is.
|
||||
#[derive(Clone, Debug, Default, PartialEq, Eq)]
|
||||
pub struct Progress {
|
||||
pub file: String,
|
||||
/// Bytes of all files on disk so far, and the total expected (pinned sizes + announced ones).
|
||||
pub done: u64,
|
||||
pub total: u64,
|
||||
/// The mirror in use (without query).
|
||||
pub mirror: String,
|
||||
}
|
||||
|
||||
/// Why a download failed.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub enum DownloadError {
|
||||
/// No mirror is configured.
|
||||
NoMirrors,
|
||||
Cancelled,
|
||||
/// A local file problem (folder not writable, disk full…).
|
||||
Disk(String),
|
||||
/// Every mirror failed for `file`; the reason for each.
|
||||
AllMirrorsFailed {
|
||||
file: String,
|
||||
errors: Vec<String>,
|
||||
},
|
||||
}
|
||||
|
||||
impl std::fmt::Display for DownloadError {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
match self {
|
||||
DownloadError::NoMirrors => write!(f, "no download location is configured for this model in this build"),
|
||||
DownloadError::Cancelled => write!(f, "the download was cancelled"),
|
||||
DownloadError::Disk(e) => write!(f, "{e}"),
|
||||
DownloadError::AllMirrorsFailed { file, errors } => write!(f, "could not download {file}: {}", errors.join("; ")),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// One mirror's failure for one file; `Retry` resumes on the same mirror.
|
||||
enum Fail {
|
||||
Retry(String),
|
||||
NextMirror(String),
|
||||
Cancelled,
|
||||
Disk(String),
|
||||
}
|
||||
|
||||
/// Download `files` into `dir` from `mirrors` (in order; see [`mirrors`]). Files already there
|
||||
/// and verified are kept. `progress` is called as bytes arrive (often: throttle in it).
|
||||
pub fn download(
|
||||
files: &[FileSpec],
|
||||
mirrors: &[String],
|
||||
dir: &Path,
|
||||
opts: &Options,
|
||||
cancel: &AtomicBool,
|
||||
progress: &mut dyn FnMut(&Progress),
|
||||
) -> Result<(), DownloadError> {
|
||||
std::fs::create_dir_all(dir).map_err(|e| DownloadError::Disk(format!("can't create {}: {e}", dir.display())))?;
|
||||
// what's left to fetch
|
||||
let mut todo = Vec::new();
|
||||
for f in files {
|
||||
if !verified(f, &dir.join(f.name))? {
|
||||
todo.push(f);
|
||||
}
|
||||
}
|
||||
if todo.is_empty() {
|
||||
return Ok(());
|
||||
}
|
||||
if mirrors.is_empty() {
|
||||
return Err(DownloadError::NoMirrors);
|
||||
}
|
||||
let total: u64 = files.iter().map(|f| f.size.unwrap_or(0)).sum();
|
||||
let mut before: u64 = files.iter().filter(|f| !todo.contains(f)).filter_map(|f| f.size).sum();
|
||||
for f in todo {
|
||||
let mut errors = Vec::new();
|
||||
let mut ok = false;
|
||||
'mirrors: for m in mirrors {
|
||||
let url = format!("{m}/{}", f.name);
|
||||
for _ in 0..opts.attempts.max(1) {
|
||||
if cancel.load(Ordering::Relaxed) {
|
||||
return Err(DownloadError::Cancelled);
|
||||
}
|
||||
let mut report = |n: u64, announced: u64| {
|
||||
let mirror = Url::parse(m).map(|u| u.display()).unwrap_or_default();
|
||||
progress(&Progress {
|
||||
file: f.name.to_string(),
|
||||
done: before.saturating_add(n),
|
||||
total: total.max(before.saturating_add(announced)),
|
||||
mirror,
|
||||
})
|
||||
};
|
||||
match fetch_file(f, &url, dir, opts, cancel, &mut report) {
|
||||
Ok(()) => {
|
||||
ok = true;
|
||||
break 'mirrors;
|
||||
}
|
||||
Err(Fail::Cancelled) => return Err(DownloadError::Cancelled),
|
||||
Err(Fail::Disk(e)) => return Err(DownloadError::Disk(e)),
|
||||
Err(Fail::Retry(e)) => {
|
||||
log::warn!("download of {url}: {e} (retrying)");
|
||||
errors.push(format!("{}: {e}", short(m)));
|
||||
}
|
||||
Err(Fail::NextMirror(e)) => {
|
||||
log::warn!("download of {url}: {e}");
|
||||
errors.push(format!("{}: {e}", short(m)));
|
||||
continue 'mirrors;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if !ok {
|
||||
return Err(DownloadError::AllMirrorsFailed { file: f.name.to_string(), errors });
|
||||
}
|
||||
before = before.saturating_add(std::fs::metadata(dir.join(f.name)).map(|m| m.len()).unwrap_or(0));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// A mirror for messages (no query string: it may hold a token).
|
||||
fn short(m: &str) -> String {
|
||||
Url::parse(m).map(|u| u.display()).unwrap_or_else(|_| "mirror".into())
|
||||
}
|
||||
|
||||
/// Whether `path` is already the right file (exact size and hash when pinned).
|
||||
fn verified(f: &FileSpec, path: &Path) -> Result<bool, DownloadError> {
|
||||
let Ok(meta) = std::fs::metadata(path) else { return Ok(false) };
|
||||
if !meta.is_file() || meta.len() == 0 {
|
||||
return Ok(false);
|
||||
}
|
||||
if f.size.is_some_and(|s| s != meta.len()) || meta.len() > f.max {
|
||||
return Ok(false);
|
||||
}
|
||||
match f.sha256 {
|
||||
Some(want) => Ok(sha256_file(path).map_err(DownloadError::Disk)? == want),
|
||||
None => Ok(true),
|
||||
}
|
||||
}
|
||||
|
||||
fn sha256_file(path: &Path) -> Result<String, String> {
|
||||
let mut file = File::open(path).map_err(|e| format!("{}: {e}", path.display()))?;
|
||||
let mut h = Sha256::new();
|
||||
let mut buf = vec![0u8; 1 << 20];
|
||||
loop {
|
||||
let n = file.read(&mut buf).map_err(|e| format!("{}: {e}", path.display()))?;
|
||||
if n == 0 {
|
||||
break;
|
||||
}
|
||||
h.update(buf.get(..n).unwrap_or_default());
|
||||
}
|
||||
Ok(h.finalize().iter().map(|b| format!("{b:02x}")).collect())
|
||||
}
|
||||
|
||||
fn part_path(dir: &Path, name: &str) -> PathBuf {
|
||||
dir.join(format!("{name}.part"))
|
||||
}
|
||||
|
||||
/// Fetch one file from one mirror into `<name>.part` (resuming it when its size is pinned),
|
||||
/// check it and move it into place.
|
||||
fn fetch_file(f: &FileSpec, url: &str, dir: &Path, opts: &Options, cancel: &AtomicBool, report: &mut dyn FnMut(u64, u64)) -> Result<(), Fail> {
|
||||
let part = part_path(dir, f.name);
|
||||
let limit = f.size.unwrap_or(f.max);
|
||||
let disk = |e: std::io::Error| Fail::Disk(format!("{}: {e}", part.display()));
|
||||
// resume only a pinned file (an unpinned one can't be checked after resuming)
|
||||
let mut have = match std::fs::metadata(&part) {
|
||||
Ok(m) if f.size.is_some() && m.len() <= limit => m.len(),
|
||||
Ok(_) => {
|
||||
std::fs::remove_file(&part).map_err(disk)?;
|
||||
0
|
||||
}
|
||||
Err(_) => 0,
|
||||
};
|
||||
let fetched = f.size != Some(have) || have == 0;
|
||||
if fetched {
|
||||
let limits = Limits { connect: opts.connect_timeout, stall: opts.stall_timeout, cancel };
|
||||
let mut url = Url::parse(url).map_err(|e| Fail::NextMirror(e.to_string()))?;
|
||||
let origin = url.host.clone();
|
||||
let mut redirects = 0;
|
||||
let mut resp = loop {
|
||||
let mut headers = vec![];
|
||||
if have > 0 {
|
||||
headers.push(("Range", format!("bytes={have}-")));
|
||||
}
|
||||
// a token for the configured host only (never sent on to a redirect's host)
|
||||
if let Ok(t) = std::env::var("LIGHTCRAFT_SAM3_TOKEN")
|
||||
&& !t.trim().is_empty()
|
||||
&& url.tls
|
||||
&& url.host == origin
|
||||
{
|
||||
headers.push(("Authorization", format!("Bearer {}", t.trim())));
|
||||
}
|
||||
let r = http::get(&url, &headers, &limits).map_err(http_fail)?;
|
||||
match r.status {
|
||||
301 | 302 | 303 | 307 | 308 => {
|
||||
redirects += 1;
|
||||
if redirects > MAX_REDIRECTS {
|
||||
return Err(Fail::NextMirror("too many redirects".into()));
|
||||
}
|
||||
let loc = r.header("location").ok_or_else(|| Fail::NextMirror("redirect without a location".into()))?;
|
||||
url = url.join(loc).map_err(|e| Fail::NextMirror(e.to_string()))?;
|
||||
}
|
||||
_ => break r,
|
||||
}
|
||||
};
|
||||
let mut file = match resp.status {
|
||||
200 => {
|
||||
// the whole file (from the start, also when a range was asked for)
|
||||
if resp.content_length().is_some_and(|n| n > limit || f.size.is_some_and(|s| s != n)) {
|
||||
return Err(Fail::NextMirror(format!("the server's file has the wrong size ({:?} bytes)", resp.content_length())));
|
||||
}
|
||||
have = 0;
|
||||
File::create(&part).map_err(disk)?
|
||||
}
|
||||
206 if have > 0 => {
|
||||
let range = resp.header("content-range").unwrap_or_default();
|
||||
let (start, total) = parse_content_range(range).ok_or_else(|| Fail::NextMirror(format!("bad Content-Range `{range}`")))?;
|
||||
if start != have || total.is_some_and(|t| Some(t) != f.size) {
|
||||
return Err(Fail::NextMirror(format!("the server resumed at the wrong place ({range})")));
|
||||
}
|
||||
OpenOptions::new().append(true).open(&part).map_err(disk)?
|
||||
}
|
||||
416 => {
|
||||
// the partial file doesn't fit what the server has: start over
|
||||
std::fs::remove_file(&part).map_err(disk)?;
|
||||
return Err(Fail::Retry("the partial download didn't match; starting over".into()));
|
||||
}
|
||||
401 | 403 => return Err(Fail::NextMirror(format!("access denied (HTTP {})", resp.status))),
|
||||
404 | 410 => return Err(Fail::NextMirror(format!("not found (HTTP {})", resp.status))),
|
||||
s if s >= 500 => return Err(Fail::Retry(format!("server error (HTTP {s})"))),
|
||||
s => return Err(Fail::NextMirror(format!("HTTP {s}"))),
|
||||
};
|
||||
let announced = f.size.unwrap_or_else(|| resp.content_length().unwrap_or(0));
|
||||
report(have, announced);
|
||||
let mut buf = vec![0u8; 256 * 1024];
|
||||
loop {
|
||||
let n = match resp.read(&mut buf, &limits) {
|
||||
Ok(n) => n,
|
||||
Err(e) => {
|
||||
let _ = file.flush();
|
||||
if f.size.is_none() {
|
||||
drop(file);
|
||||
let _ = std::fs::remove_file(&part);
|
||||
}
|
||||
return Err(http_fail(e));
|
||||
}
|
||||
};
|
||||
if n == 0 {
|
||||
break;
|
||||
}
|
||||
if have.saturating_add(n as u64) > limit {
|
||||
drop(file);
|
||||
let _ = std::fs::remove_file(&part);
|
||||
return Err(Fail::NextMirror(format!("the server sent more than the expected {limit} bytes")));
|
||||
}
|
||||
file.write_all(buf.get(..n).unwrap_or_default()).map_err(disk)?;
|
||||
have = have.saturating_add(n as u64);
|
||||
report(have, announced);
|
||||
}
|
||||
file.sync_all().map_err(disk)?;
|
||||
}
|
||||
// checks: exact size, hash
|
||||
let len = std::fs::metadata(&part).map_err(disk)?.len();
|
||||
if f.size.is_some_and(|s| s != len) {
|
||||
return Err(Fail::Retry(format!("incomplete ({len} of {} bytes)", f.size.unwrap_or(0))));
|
||||
}
|
||||
if len == 0 {
|
||||
let _ = std::fs::remove_file(&part);
|
||||
return Err(Fail::NextMirror("the server sent an empty file".into()));
|
||||
}
|
||||
if let Some(want) = f.sha256 {
|
||||
let got = sha256_file(&part).map_err(Fail::Disk)?;
|
||||
if got != want {
|
||||
std::fs::remove_file(&part).map_err(disk)?;
|
||||
let why = format!("the file is damaged or not the official one (SHA-256 {got})");
|
||||
// a damaged partial file from an earlier run isn't this mirror's fault
|
||||
return Err(if fetched { Fail::NextMirror(why) } else { Fail::Retry(why) });
|
||||
}
|
||||
}
|
||||
std::fs::rename(&part, dir.join(f.name)).map_err(disk)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn http_fail(e: HttpError) -> Fail {
|
||||
match e {
|
||||
HttpError::Cancelled => Fail::Cancelled,
|
||||
HttpError::Stalled | HttpError::Io(_) | HttpError::Protocol(_) => Fail::Retry(e.to_string()),
|
||||
HttpError::BadUrl(_) | HttpError::Connect(_) | HttpError::Tls(_) => Fail::NextMirror(e.to_string()),
|
||||
}
|
||||
}
|
||||
|
||||
/// `bytes a-b/total` → (a, total) (`*` total → None).
|
||||
fn parse_content_range(v: &str) -> Option<(u64, Option<u64>)> {
|
||||
let v = v.trim().strip_prefix("bytes")?.trim();
|
||||
let (range, total) = v.split_once('/')?;
|
||||
let (a, b) = range.split_once('-')?;
|
||||
let (a, b) = (a.trim().parse::<u64>().ok()?, b.trim().parse::<u64>().ok()?);
|
||||
if b < a {
|
||||
return None;
|
||||
}
|
||||
let total = match total.trim() {
|
||||
"*" => None,
|
||||
t => Some(t.parse::<u64>().ok()?),
|
||||
};
|
||||
Some((a, total))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests;
|
||||
|
|
@ -294,13 +294,15 @@ fn no_mirrors_is_a_clear_error_and_mirror_lists_are_parsed() {
|
|||
let dir = tmp("nomirror");
|
||||
let e = run(&[pinned()], &[], &dir, &AtomicBool::new(false)).0.unwrap_err();
|
||||
assert_eq!(e, DownloadError::NoMirrors);
|
||||
assert!(e.to_string().contains(MIRRORS_ENV));
|
||||
assert!(e.to_string().contains("no download location"));
|
||||
let file = std::env::temp_dir().join(format!("lc-sam3-mirrors-{}.txt", std::process::id()));
|
||||
std::fs::write(&file, "# mine\nhttps://b.example/m/ # second\n\nnot a url\nhttps://a.example/x\n").unwrap();
|
||||
let m = mirrors(Some("https://a.example/x/, ftp://no"), Some(&file));
|
||||
assert_eq!(m[..2], ["https://a.example/x".to_string(), "https://b.example/m".to_string()]);
|
||||
assert_eq!(m.len(), 2 + DEFAULT_MIRRORS.len());
|
||||
assert_eq!(mirrors(None, Some(Path::new("/nonexistent/mirrors.txt"))).len(), DEFAULT_MIRRORS.len());
|
||||
let defaults = ["https://c.example/d", "https://a.example/x"];
|
||||
let m = mirrors(Some("https://a.example/x/, ftp://no"), Some(&file), &defaults);
|
||||
// the environment's, then the file's, then the defaults; a repeated one is kept once
|
||||
assert_eq!(m, ["https://a.example/x", "https://b.example/m", "https://c.example/d"]);
|
||||
assert_eq!(mirrors(None, Some(Path::new("/nonexistent/mirrors.txt")), &defaults).len(), 2);
|
||||
assert!(mirrors(None, None, &[]).is_empty());
|
||||
let _ = std::fs::remove_file(&file);
|
||||
let _ = std::fs::remove_dir_all(&dir);
|
||||
}
|
||||
|
|
@ -73,7 +73,23 @@ fn check(name: &str, src: &Arc<Rgb32f>, info: &SourceInfo, s: &DevelopSettings,
|
|||
let gpu = lightcraft_gpu::render(src, info, s, req, None).expect("gpu render");
|
||||
let (mean, max, over) = diff(&cpu, &gpu.image);
|
||||
eprintln!("{name:<28} {}x{} mean {mean:.4} max {max} >1: {:.4}%", cpu.width, cpu.height, over * 100.0);
|
||||
assert!(mean < MEAN_LSB && max <= MAX_LSB, "{name}: mean {mean:.4} LSB, max {max} LSB");
|
||||
// A pixel on the very edge of the picture can fall either side of the sampling boundary (the
|
||||
// lens warp with a shifted red plane lands exactly outside it): the CPU and a GPU's rounding
|
||||
// may then disagree on one edge pixel. Tolerate a couple of those, nothing in the interior.
|
||||
let (w, h) = (cpu.width, cpu.height);
|
||||
let strays: Vec<(usize, usize)> = cpu
|
||||
.data
|
||||
.iter()
|
||||
.zip(&gpu.image.data)
|
||||
.enumerate()
|
||||
.filter(|(_, (p, q))| (0..3).any(|c| p[c].abs_diff(q[c]) > MAX_LSB))
|
||||
.map(|(i, _)| (i % w, i / w))
|
||||
.collect();
|
||||
let on_the_edge = |&(x, y): &(usize, usize)| x == 0 || y == 0 || x + 1 == w || y + 1 == h;
|
||||
assert!(
|
||||
mean < MEAN_LSB && (max <= MAX_LSB || (strays.len() <= 2 && strays.iter().all(on_the_edge))),
|
||||
"{name}: mean {mean:.4} LSB, max {max} LSB, pixels over {MAX_LSB}: {strays:?}"
|
||||
);
|
||||
(mean, max)
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -11,8 +11,10 @@ use serde_json::{Value, json};
|
|||
use crate::backend::Backend;
|
||||
|
||||
/// File extensions recognised as photos when expanding folders.
|
||||
pub const PHOTO_EXTENSIONS: &[&str] =
|
||||
&["jpg", "jpeg", "png", "tif", "tiff", "webp", "dng", "cr2", "cr3", "nef", "arw", "raf", "orf", "rw2", "pef", "psd", "jxl", "gif", "bmp", "avif"];
|
||||
pub const PHOTO_EXTENSIONS: &[&str] = &[
|
||||
"jpg", "jpeg", "png", "tif", "tiff", "webp", "dng", "cr2", "cr3", "nef", "arw", "raf", "orf", "rw2", "rwl", "raw", "pef", "psd", "jxl", "gif",
|
||||
"bmp", "avif",
|
||||
];
|
||||
|
||||
/// Headless backend: a [`Session`] with filesystem hooks.
|
||||
pub struct Headless {
|
||||
|
|
|
|||
|
|
@ -28,7 +28,8 @@ pub use pool::JobPool;
|
|||
|
||||
/// Rendered-thumbnail cache: memory LRU over an optional disk cache.
|
||||
pub struct PreviewCache {
|
||||
generation: RwLock<u64>,
|
||||
/// Current generation; `None` once [`Self::retire`]d (nothing is read or written any more).
|
||||
generation: RwLock<Option<u64>>,
|
||||
mem: Mutex<Lru<Hash128, Arc<Rgba8>>>,
|
||||
disk: Option<DiskCache>,
|
||||
}
|
||||
|
|
@ -36,12 +37,12 @@ pub struct PreviewCache {
|
|||
impl PreviewCache {
|
||||
/// Memory-only cache of at most `mem_bytes`.
|
||||
pub fn memory(mem_bytes: usize) -> PreviewCache {
|
||||
PreviewCache { generation: RwLock::new(0), mem: Mutex::new(Lru::new(mem_bytes)), disk: None }
|
||||
PreviewCache { generation: RwLock::new(Some(0)), mem: Mutex::new(Lru::new(mem_bytes)), disk: None }
|
||||
}
|
||||
|
||||
/// Memory LRU + disk cache in `dir` (at most `disk_bytes`).
|
||||
pub fn with_disk(mem_bytes: usize, dir: &Path, disk_bytes: u64) -> PreviewCache {
|
||||
PreviewCache { generation: RwLock::new(0), mem: Mutex::new(Lru::new(mem_bytes)), disk: Some(DiskCache::new(dir, disk_bytes)) }
|
||||
PreviewCache { generation: RwLock::new(Some(0)), mem: Mutex::new(Lru::new(mem_bytes)), disk: Some(DiskCache::new(dir, disk_bytes)) }
|
||||
}
|
||||
|
||||
pub fn disk(&self) -> Option<&DiskCache> {
|
||||
|
|
@ -53,13 +54,14 @@ impl PreviewCache {
|
|||
}
|
||||
|
||||
/// Changes when explicitly cleared; jobs captured before that clear cannot repopulate it.
|
||||
/// A retired cache reports `u64::MAX`, which no job's generation matches.
|
||||
pub fn generation(&self) -> u64 {
|
||||
*self.generation.read().unwrap_or_else(std::sync::PoisonError::into_inner)
|
||||
self.generation.read().unwrap_or_else(std::sync::PoisonError::into_inner).unwrap_or(u64::MAX)
|
||||
}
|
||||
|
||||
pub fn get_at(&self, generation: u64, key: Hash128) -> Option<Arc<Rgba8>> {
|
||||
let current = self.generation.read().unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
if *current != generation {
|
||||
if *current != Some(generation) {
|
||||
return None;
|
||||
}
|
||||
if let Some(v) = self.mem.lock().unwrap_or_else(|e| e.into_inner()).get(&key) {
|
||||
|
|
@ -76,7 +78,7 @@ impl PreviewCache {
|
|||
|
||||
pub fn put_at(&self, generation: u64, key: Hash128, img: Arc<Rgba8>) {
|
||||
let current = self.generation.read().unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
if *current != generation {
|
||||
if *current != Some(generation) {
|
||||
return;
|
||||
}
|
||||
if let Some(d) = &self.disk {
|
||||
|
|
@ -94,7 +96,7 @@ impl PreviewCache {
|
|||
|
||||
pub fn put_deferred_at(self: &Arc<Self>, generation: u64, key: Hash128, img: Arc<Rgba8>) {
|
||||
let current = self.generation.read().unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
if *current != generation {
|
||||
if *current != Some(generation) {
|
||||
return;
|
||||
}
|
||||
let c = cost(&img);
|
||||
|
|
@ -106,7 +108,7 @@ impl PreviewCache {
|
|||
let me = self.clone();
|
||||
let write = move || {
|
||||
let current = me.generation.read().unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
if *current != generation {
|
||||
if *current != Some(generation) {
|
||||
return;
|
||||
}
|
||||
if let Some(d) = &me.disk {
|
||||
|
|
@ -121,16 +123,30 @@ impl PreviewCache {
|
|||
write();
|
||||
}
|
||||
|
||||
/// Drop everything (memory and disk).
|
||||
/// Drop everything (memory and disk). Does nothing once [`Self::retire`]d.
|
||||
pub fn clear(&self) {
|
||||
let mut generation = self.generation.write().unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
*generation = generation.wrapping_add(1);
|
||||
// a retired cache no longer owns its directory (its replacement may)
|
||||
let Some(g) = generation.as_mut() else { return };
|
||||
// (never reaches u64::MAX, which stands for "retired")
|
||||
*g = g.wrapping_add(1) % u64::MAX;
|
||||
self.mem.lock().unwrap_or_else(|e| e.into_inner()).clear();
|
||||
if let Some(d) = &self.disk {
|
||||
d.clear();
|
||||
}
|
||||
}
|
||||
|
||||
/// Stop this cache object for good, leaving its disk files alone: called when another cache
|
||||
/// object replaces it (a library reopened or another one opened). Jobs still holding it then
|
||||
/// can neither read nor write through it, so a later clear of the replacement's directory
|
||||
/// (possibly the same one) can't be repopulated by them. Waits for writes in progress, like
|
||||
/// [`Self::clear`].
|
||||
pub fn retire(&self) {
|
||||
let mut generation = self.generation.write().unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
*generation = None;
|
||||
self.mem.lock().unwrap_or_else(|e| e.into_inner()).clear();
|
||||
}
|
||||
|
||||
/// Use tick ([`next_tick`]) of the least recently used image in memory.
|
||||
pub fn oldest_tick(&self) -> Option<u64> {
|
||||
self.mem.lock().unwrap_or_else(|e| e.into_inner()).oldest_tick()
|
||||
|
|
|
|||
|
|
@ -26,6 +26,30 @@ fn cleared_generation_cannot_read_or_repopulate_memory_or_disk() {
|
|||
cache.clear();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn retired_cache_keeps_its_files_but_refuses_every_read_and_write() {
|
||||
let dir = temp_dir("retire");
|
||||
let old = Arc::new(PreviewCache::with_disk(1 << 20, &dir, 1 << 20));
|
||||
let (kept, late) = (hash_bytes(b"kept"), hash_bytes(b"late"));
|
||||
let img = Arc::new(gradient(8, 8, 7));
|
||||
let generation = old.generation();
|
||||
old.put_at(generation, kept, img.clone());
|
||||
old.retire();
|
||||
assert_ne!(old.generation(), generation);
|
||||
assert!(old.get_at(generation, kept).is_none());
|
||||
assert!(old.get(kept).is_none());
|
||||
old.put_at(generation, late, img.clone());
|
||||
old.put_deferred_at(generation, late, img.clone());
|
||||
old.put(late, img.clone());
|
||||
old.put_deferred(late, img.clone());
|
||||
old.clear();
|
||||
// a replacement for the same directory still finds the valid file, and only that
|
||||
let new = PreviewCache::with_disk(1 << 20, &dir, 1 << 20);
|
||||
assert_eq!(new.get(kept).map(|i| (i.width, i.height)), Some((8, 8)), "the disk file stays (JPEG: lossy)");
|
||||
assert!(new.get(late).is_none());
|
||||
new.clear();
|
||||
}
|
||||
|
||||
fn temp_dir(tag: &str) -> std::path::PathBuf {
|
||||
let d = std::env::temp_dir().join(format!("lc-preview-{tag}-{}", std::process::id()));
|
||||
let _ = std::fs::remove_dir_all(&d);
|
||||
|
|
|
|||
|
|
@ -177,6 +177,9 @@ pub fn write_dng(raw: &RawImage, opts: &DngWriteOptions) -> Result<Vec<u8>> {
|
|||
ifd.set(t::AS_SHOT_NEUTRAL, rat_vec(&v));
|
||||
} else if let Some(xy) = c.as_shot_white_xy {
|
||||
ifd.set(t::AS_SHOT_WHITE_XY, rat_vec(&[xy.x, xy.y]));
|
||||
} else if let Some(m) = raw.wb_multipliers.filter(|m| m.iter().all(|v| *v > 0.0 && v.is_finite())) {
|
||||
// vendor raws (NEF, ARW…) carry white balance as multipliers: the DNG neutral is their reciprocal
|
||||
ifd.set(t::AS_SHOT_NEUTRAL, rat_vec(&m.map(|v| 1.0 / v as f64)));
|
||||
}
|
||||
ifd.set(t::BASELINE_EXPOSURE, Value::SRational(vec![srational(c.baseline_exposure)]));
|
||||
// the source's own profile look travels with its data (conversions, smart previews, merges)
|
||||
|
|
|
|||
|
|
@ -13,9 +13,10 @@
|
|||
//!
|
||||
//! Formats: DNG (uncompressed, lossless JPEG, lossy JPEG (Smart Previews), Deflate incl. floating point, tiled/stripped, CFA and LinearRaw),
|
||||
//! Canon CR2, Nikon NEF/NRW (uncompressed, Huffman lossless / lossy compressed), Sony ARW (uncompressed, ARW2, lossless), Fujifilm RAF (uncompressed Bayer
|
||||
//! and X-Trans), Panasonic RW2 (packed 12/14-bit), Pentax PEF (uncompressed, Huffman), Olympus ORF (uncompressed).
|
||||
//! and X-Trans), Panasonic RW2 / Leica RWL / Panasonic RAW (every raw format: compressed 4 and 6, the prefix-coded strips of 8,
|
||||
//! packed 2/5/7, the 16-bit words of the oldest bodies), Pentax PEF (uncompressed, Huffman), Olympus ORF (uncompressed).
|
||||
//! [`embedded_preview`] covers all of them plus CR3. Variants we can't decode yet (Nikon "lossy after split" NEF,
|
||||
//! Panasonic quantised RW2, compressed ORF/RAF, CR3) return [`RawError::Unsupported`]; each vendor module documents its sources
|
||||
//! compressed ORF/RAF, CR3) return [`RawError::Unsupported`]; each vendor module documents its sources
|
||||
//! (public specifications, tag-name documentation, black-box analysis of CC0 samples) and gaps. Non-DNG files carry no
|
||||
//! colour matrix: [`color`] falls back to a documented neutral model. The decoders never panic on malformed input.
|
||||
#![forbid(unsafe_code)]
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load Diff
|
|
@ -19,9 +19,6 @@ const KNOWN_UNSUPPORTED: &[&str] = &[
|
|||
"arw-sony-a7m4-lossless-m", // Sony lossless compressed M/S: subsampled (YCbCr) lossless JPEG
|
||||
"arw-sony-a7m4-lossless-s", // "
|
||||
"raf-fuji-xt20-compressed", // Fujifilm compressed RAF
|
||||
"rw2-panasonic-gh5.", // Panasonic raw format 4 (quantised)
|
||||
"rw2-panasonic-gx80", // "
|
||||
"rw2-panasonic-g9-b", // "
|
||||
"orf-olympus-em", // Olympus compressed ORF
|
||||
"sraw", // Canon sRAW / mRAW
|
||||
];
|
||||
|
|
@ -46,11 +43,12 @@ fn corpus_raw_decodes() {
|
|||
let t0 = Instant::now();
|
||||
let preview = embedded_preview(&bytes);
|
||||
let tp = t0.elapsed().as_secs_f64() * 1e3;
|
||||
// DNG previews are optional (and some carry only an uncompressed RGB thumbnail); vendor raws always embed a JPEG
|
||||
// DNG previews are optional (and some carry only an uncompressed RGB thumbnail); vendor raws embed a JPEG,
|
||||
// except the Panasonic `.RAW` files of 2005–2007
|
||||
if let Some(p) = &preview {
|
||||
assert!(p.starts_with(&[0xff, 0xd8]) && p.ends_with(&[0xff, 0xd9]), "{name}: preview is not a JPEG");
|
||||
} else {
|
||||
assert_eq!(fmt, RawFormat::Dng, "{name}: no embedded preview");
|
||||
assert!(fmt == RawFormat::Dng || name.starts_with("raw-panasonic-"), "{name}: no embedded preview");
|
||||
}
|
||||
let preview_kb = preview.as_ref().map_or(0, |p| p.len() / 1024);
|
||||
let t1 = Instant::now();
|
||||
|
|
@ -289,3 +287,145 @@ fn corpus_sony_pre2017_colour_metadata() {
|
|||
}
|
||||
eprintln!("pre-2017 Sony ARW colour metadata checked on {seen} files");
|
||||
}
|
||||
|
||||
/// The embedded JPEG as linear RGB, reduced to `gw × gh` cells.
|
||||
fn jpeg_cells(jpeg: &[u8], gw: usize, gh: usize) -> Vec<[f64; 3]> {
|
||||
use zune_core::{bytestream::ZCursor, colorspace::ColorSpace, options::DecoderOptions};
|
||||
let opts = DecoderOptions::default().jpeg_set_out_colorspace(ColorSpace::RGB);
|
||||
let mut d = zune_jpeg::JpegDecoder::new_with_options(ZCursor::new(jpeg), opts);
|
||||
let px = d.decode().unwrap();
|
||||
let info = d.info().unwrap();
|
||||
let (w, h) = (info.width as usize, info.height as usize);
|
||||
let (mut sum, mut n) = (vec![[0f64; 3]; gw * gh], vec![0f64; gw * gh]);
|
||||
for y in 0..h {
|
||||
for x in 0..w {
|
||||
let i = (y * gh / h) * gw + x * gw / w;
|
||||
for c in 0..3 {
|
||||
let v = px[(y * w + x) * 3 + c] as f64 / 255.0;
|
||||
sum[i][c] += if v <= 0.04045 { v / 12.92 } else { ((v + 0.055) / 1.055).powf(2.4) };
|
||||
}
|
||||
n[i] += 1.0;
|
||||
}
|
||||
}
|
||||
sum.iter().zip(&n).map(|(s, n)| s.map(|v| v / n)).collect()
|
||||
}
|
||||
|
||||
/// Black-subtracted means of the four sites of the 2×2 cells (anchored at sample (0, 0)) of the active area,
|
||||
/// reduced to `gw × gh` cells.
|
||||
fn raw_sites(img: &lightcraft_raw::RawImage, gw: usize, gh: usize) -> Vec<[f64; 4]> {
|
||||
let lightcraft_raw::RawData::U16(d) = &img.data else { panic!("float data") };
|
||||
let (w, a, black) = (img.width, img.active_area, img.black.mean() as f64);
|
||||
let (x0, y0) = ((a.x + 1) & !1, (a.y + 1) & !1);
|
||||
let (cw, ch) = ((a.x + a.width - x0) / 2, (a.y + a.height - y0) / 2);
|
||||
let (mut sum, mut n) = (vec![[0f64; 4]; gw * gh], vec![0f64; gw * gh]);
|
||||
for cy in 0..ch {
|
||||
for cx in 0..cw {
|
||||
let i = (cy * gh / ch) * gw + cx * gw / cw;
|
||||
for site in 0..4 {
|
||||
sum[i][site] += d[(y0 + 2 * cy + site / 2) * w + x0 + 2 * cx + site % 2] as f64 - black;
|
||||
}
|
||||
n[i] += 1.0;
|
||||
}
|
||||
}
|
||||
sum.iter().zip(&n).map(|(s, n)| s.map(|v| v / n)).collect()
|
||||
}
|
||||
|
||||
fn ranks(v: &[f64]) -> Vec<f64> {
|
||||
let mut idx: Vec<usize> = (0..v.len()).collect();
|
||||
idx.sort_by(|&a, &b| v[a].total_cmp(&v[b]));
|
||||
let mut r = vec![0f64; v.len()];
|
||||
for (rank, &i) in idx.iter().enumerate() {
|
||||
r[i] = rank as f64;
|
||||
}
|
||||
r
|
||||
}
|
||||
|
||||
/// How well the raw's chromaticities (log r/g, log b/g per cell) follow the JPEG's when the mosaic is read with
|
||||
/// `layout` (colour of each 2×2 site at sample (0, 0)).
|
||||
fn chroma_agreement(sites: &[[f64; 4]], jpeg: &[[f64; 3]], layout: &[u8]) -> f64 {
|
||||
let (mut x, mut y) = ([vec![], vec![]], [vec![], vec![]]);
|
||||
for (s, j) in sites.iter().zip(jpeg) {
|
||||
let (mut rgb, mut n) = ([0f64; 3], [0f64; 3]);
|
||||
for (site, &c) in layout.iter().enumerate() {
|
||||
rgb[c as usize] += s[site];
|
||||
n[c as usize] += 1.0;
|
||||
}
|
||||
let rgb = [rgb[0] / n[0], rgb[1] / n[1], rgb[2] / n[2]];
|
||||
if rgb.iter().all(|v| *v > 1.0) && j.iter().all(|v| (0.002..0.9).contains(v)) {
|
||||
for (k, c) in [0, 2].into_iter().enumerate() {
|
||||
x[k].push((rgb[c] / rgb[1]).ln());
|
||||
y[k].push((j[c] / j[1]).ln());
|
||||
}
|
||||
}
|
||||
}
|
||||
(correlation(&x[0], &y[0]) + correlation(&x[1], &y[1])) / 2.0
|
||||
}
|
||||
|
||||
/// Panasonic RW2 / RWL / RAW (`crates/raw/src/vendor/rw2.rs`): one file per raw encoding (formats 2, 4, 5, 6, 7 and
|
||||
/// 8 — at 12, 14 and 16 bits — and the 16-bit words of the oldest bodies) decodes to the image of the camera's own
|
||||
/// JPEG, with the black levels, crops and colour-filter layouts the module docs derive and no defect markers left.
|
||||
#[test]
|
||||
fn corpus_panasonic_encodings() {
|
||||
let dir = corpus_root().join("raw");
|
||||
// (file, bits, mean black level, crop width × height, CFA layout)
|
||||
let cases = [
|
||||
("raw-panasonic-fz50.raw", 12, 15.0, (3648, 2736), "BGGR"),
|
||||
("raw-panasonic-fz8.raw", 12, 15.0, (3072, 2304), "RGGB"),
|
||||
("rw2-panasonic-fz1000m2-4x3.rw2", 12, 143.0, (4864, 3648), "GBRG"),
|
||||
("rw2-panasonic-g9-b.rw2", 12, 143.0, (5184, 3888), "RGGB"),
|
||||
("rw2-panasonic-g9.rw2", 12, 127.5, (5184, 3888), "RGGB"),
|
||||
("rw2-panasonic-gh1.rw2", 12, 15.0, (4000, 3000), "GBRG"),
|
||||
("rw2-panasonic-gh5.rw2", 12, 143.5, (5184, 3888), "RGGB"),
|
||||
("rw2-panasonic-gh5m2.rw2", 12, 144.5, (5184, 3888), "RGGB"),
|
||||
("rw2-panasonic-gh5s.rw2", 14, 509.5, (3680, 2760), "RGGB"),
|
||||
("rw2-panasonic-gh6.rw2", 16, 2048.0, (5776, 4336), "RGGB"),
|
||||
("rw2-panasonic-gx80.rw2", 12, 143.0, (4592, 3448), "BGGR"),
|
||||
("rw2-panasonic-s1.rw2", 14, 526.0, (6000, 4000), "RGGB"),
|
||||
("rw2-panasonic-s5-format7.rw2", 14, 512.0, (6000, 4000), "RGGB"),
|
||||
("rw2-panasonic-s5m2.rw2", 14, 512.0, (6000, 4000), "RGGB"),
|
||||
("rw2-panasonic-s9.rw2", 12, 128.0, (6000, 4000), "RGGB"),
|
||||
("rwl-leica-dlux7.rwl", 12, 143.0, (4736, 3552), "BGGR"),
|
||||
];
|
||||
let mut seen = 0;
|
||||
for (name, bits, black, (cw, ch), cfa) in cases {
|
||||
let Ok(bytes) = std::fs::read(dir.join(name)) else {
|
||||
eprintln!("skip: {name} absent");
|
||||
continue;
|
||||
};
|
||||
let img = decode(&bytes).unwrap_or_else(|e| panic!("{name}: {e}"));
|
||||
assert_eq!(img.bits, bits, "{name}: bits");
|
||||
assert_eq!(img.black.mean(), black, "{name}: black level");
|
||||
assert_eq!((img.crop.width, img.crop.height), (cw, ch), "{name}: crop");
|
||||
let layout = img.cfa.clone().unwrap_or_else(|| panic!("{name}: no CFA"));
|
||||
assert_eq!(layout.name(), cfa, "{name}: CFA layout");
|
||||
let lightcraft_raw::RawData::U16(d) = &img.data else { panic!("float data") };
|
||||
let a = img.active_area;
|
||||
let zeros =
|
||||
(a.y..a.y + a.height).map(|y| d[y * img.width + a.x..y * img.width + a.x + a.width].iter().filter(|&&v| v == 0).count()).sum::<usize>();
|
||||
assert_eq!(zeros, 0, "{name}: defect markers left in the active area");
|
||||
let Some(jpeg) = embedded_preview(&bytes) else {
|
||||
// the 2005–2007 `.RAW` files have no preview; their flat scenes make the green diagonal unambiguous
|
||||
assert_eq!(green_on_main_diagonal(&img), cfa.starts_with('G'), "{name}: mosaic statistics disagree with {cfa}");
|
||||
seen += 1;
|
||||
continue;
|
||||
};
|
||||
// the camera's JPEG shows the whole active area (whatever the crop): the raw's green must rank like it, and
|
||||
// its colours must follow the JPEG's best with the decoded colour-filter layout
|
||||
let (gw, gh) = (24, 18);
|
||||
let (sites, jpeg) = (raw_sites(&img, gw, gh), jpeg_cells(&jpeg, gw, gh));
|
||||
let green: Vec<f64> = sites.iter().map(|s| (0..4).filter(|&i| layout.pattern[i] == 1).map(|i| s[i]).sum::<f64>()).collect();
|
||||
let jg: Vec<f64> = jpeg.iter().map(|j| j[1]).collect();
|
||||
let rho = correlation(&ranks(&green), &ranks(&jg));
|
||||
let agree = |l: &str| chroma_agreement(&sites, &jpeg, &lightcraft_raw::Cfa::bayer(l).unwrap().pattern);
|
||||
let best_other = ["RGGB", "GRBG", "GBRG", "BGGR"].into_iter().filter(|l| *l != cfa).map(agree).fold(f64::MIN, f64::max);
|
||||
let own = agree(cfa);
|
||||
eprintln!(
|
||||
"{name:32} {}x{} {bits}-bit: green rank correlation {rho:.3}, colour agreement {own:.3} (others ≤ {best_other:.3})",
|
||||
img.width, img.height
|
||||
);
|
||||
assert!(rho > 0.9, "{name}: rank correlation with the camera JPEG {rho}");
|
||||
assert!(own > 0.5 && own > best_other, "{name}: colours follow the JPEG better with another layout ({own} vs {best_other})");
|
||||
seen += 1;
|
||||
}
|
||||
eprintln!("Panasonic encodings checked on {seen} files");
|
||||
}
|
||||
|
|
|
|||
|
|
@ -453,6 +453,22 @@ fn colour_transform_from_decoded_dng() {
|
|||
assert!(o.iter().all(|v| (v - 1.0).abs() < 1e-9));
|
||||
}
|
||||
|
||||
/// Issue #212: a NEF converted to DNG came out dark and green. NEF white balance is vendor multipliers
|
||||
/// (no colour matrix, no `AsShotNeutral`); the DNG must carry them as its as-shot neutral.
|
||||
#[test]
|
||||
fn vendor_white_balance_survives_dng_conversion() {
|
||||
let mut raw = synthetic(16, 16, Some(Cfa::bayer("RGGB").unwrap()), 1);
|
||||
raw.color = ColorData::default();
|
||||
raw.wb_multipliers = Some([2.25, 1.0, 1.7421875]);
|
||||
let before = lightcraft_raw::color::camera_transform(&raw, lightcraft_raw::color::as_shot_white_xy(&raw));
|
||||
let back = decode(&write_dng(&raw, &DngWriteOptions::default()).unwrap()).unwrap();
|
||||
let after = lightcraft_raw::color::camera_transform(&back, lightcraft_raw::color::as_shot_white_xy(&back));
|
||||
for c in 0..3 {
|
||||
assert!((before.wb[c] - after.wb[c]).abs() < 1e-3, "{:?} vs {:?}", before.wb, after.wb);
|
||||
}
|
||||
assert!((after.wb[0] - 2.25).abs() < 1e-3 && (after.wb[2] - 1.7421875).abs() < 1e-3, "{:?}", after.wb);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_unsupported_and_broken() {
|
||||
let (w, h) = (8, 8);
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
//! Malformed vendor raw files (NEF, ARW, PEF, ORF, RW2, CR2, RAF) must decode to an error, never panic.
|
||||
//! Malformed vendor raw files (NEF, ARW, PEF, ORF, RW2 of every raw format, CR2, RAF) must decode to an error, never panic.
|
||||
|
||||
use lightcraft_tiff::tags as t;
|
||||
use lightcraft_tiff::{ByteOrder, IfdBuilder, ImageData, TiffWriter, Value};
|
||||
|
|
@ -68,6 +68,66 @@ fn rw2() -> Vec<u8> {
|
|||
b
|
||||
}
|
||||
|
||||
/// A Panasonic RW2 of raw format `format` (or, with `format` 0, the oldest 16-bit-word kind): sensor size, borders,
|
||||
/// bit depth, black levels, an in-camera crop and `n` bytes of payload at the end, at `RawDataOffset`. Format 8 also
|
||||
/// gets two strips covering the payload and a code table.
|
||||
fn rw2_of(format: u16, bits: u16, n: usize) -> Vec<u8> {
|
||||
let u16s = |v: &[u16]| Value::Undefined(v.iter().flat_map(|x| x.to_le_bytes()).collect());
|
||||
let build = |off: u32| {
|
||||
let mut ifd = IfdBuilder::new();
|
||||
for (tag, v) in [(2, W as u16), (3, H as u16), (4, 2), (5, 4), (6, H as u16), (7, W as u16), (9, 2), (0x0a, bits), (0x1c, 128), (0x1d, 128)] {
|
||||
ifd.set(tag, Value::Short(vec![v]));
|
||||
}
|
||||
for (tag, v) in [(0x2f, 2u16), (0x30, 6), (0x31, H as u16 - 2), (0x32, W as u16 - 2)] {
|
||||
ifd.set(tag, Value::Short(vec![v]));
|
||||
}
|
||||
if format == 0 {
|
||||
ifd.set(0x000b, Value::Short(vec![34828]));
|
||||
ifd.set(t::STRIP_OFFSETS, Value::Long(vec![off]));
|
||||
} else {
|
||||
ifd.set(0x002d, Value::Short(vec![format]));
|
||||
ifd.set(0x0118, Value::Long(vec![off]));
|
||||
}
|
||||
if format == 8 {
|
||||
let table = [
|
||||
(6, 62),
|
||||
(7, 126),
|
||||
(6, 61),
|
||||
(5, 28),
|
||||
(4, 12),
|
||||
(3, 4),
|
||||
(3, 2),
|
||||
(2, 0),
|
||||
(3, 3),
|
||||
(3, 5),
|
||||
(4, 13),
|
||||
(5, 29),
|
||||
(6, 60),
|
||||
(8, 254),
|
||||
(8, 255),
|
||||
(12, 4094),
|
||||
(12, 4095),
|
||||
];
|
||||
ifd.set(0x0040, u16s(&std::iter::once(17).chain(table.iter().flat_map(|&(l, c)| [l, c])).collect::<Vec<u16>>()));
|
||||
let half = (n / 2) as u32;
|
||||
let (a, b) = (off, off + half);
|
||||
ifd.set(0x0044, u16s(&[2, a as u16, (a >> 16) as u16, b as u16, (b >> 16) as u16]));
|
||||
ifd.set(0x0045, u16s(&[2, 0, 0, W as u16 / 2, 0]));
|
||||
ifd.set(0x0046, u16s(&[2, (half * 8) as u16, ((half * 8) >> 16) as u16, (half * 8) as u16, ((half * 8) >> 16) as u16]));
|
||||
ifd.set(0x0047, u16s(&[2, W as u16 / 2, W as u16 / 2]));
|
||||
ifd.set(0x0048, u16s(&[2, H as u16, H as u16]));
|
||||
}
|
||||
ifd.set(t::MAKE, Value::Ascii("Panasonic".into()));
|
||||
let mut b = TiffWriter::new(ByteOrder::Little, false).write(&[ifd]).unwrap();
|
||||
b[..4].copy_from_slice(b"IIU\0");
|
||||
b
|
||||
};
|
||||
let off = build(0).len();
|
||||
let mut b = build(off as u32);
|
||||
b.extend((0..n).map(|i| (i * 89 % 253) as u8));
|
||||
b
|
||||
}
|
||||
|
||||
/// A Pentax PEF: the raw image in IFD0, wide enough for the dark-border trim.
|
||||
fn pef(compression: u16) -> Vec<u8> {
|
||||
let (w, h) = (264u32, 4u32);
|
||||
|
|
@ -181,6 +241,20 @@ fn samples() -> Vec<Vec<u8>> {
|
|||
cr2(),
|
||||
raf(),
|
||||
]
|
||||
.into_iter()
|
||||
.chain(rw2_formats())
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// One RW2 per encoding with a sample decoder of its own (formats 4, 6 at 14 and 12 bits, 8, and 16-bit words).
|
||||
fn rw2_formats() -> Vec<Vec<u8>> {
|
||||
vec![
|
||||
rw2_of(4, 12, (W * H).div_ceil(14) as usize * 16),
|
||||
rw2_of(6, 14, (W * H).div_ceil(11) as usize * 16),
|
||||
rw2_of(6, 12, (W * H).div_ceil(14) as usize * 16),
|
||||
rw2_of(8, 14, (W * H) as usize * 2),
|
||||
rw2_of(0, 12, (W * H) as usize * 2),
|
||||
]
|
||||
}
|
||||
|
||||
fn exercise(bytes: &[u8]) {
|
||||
|
|
@ -208,7 +282,7 @@ proptest! {
|
|||
#![proptest_config(ProptestConfig { cases: 2000, .. ProptestConfig::default() })]
|
||||
|
||||
#[test]
|
||||
fn mutated_vendor_files_never_panic(kind in 0usize..12, flips in proptest::collection::vec((any::<usize>(), any::<u8>()), 1..16), cut in any::<usize>()) {
|
||||
fn mutated_vendor_files_never_panic(kind in 0usize..17, flips in proptest::collection::vec((any::<usize>(), any::<u8>()), 1..16), cut in any::<usize>()) {
|
||||
let mut data = samples().swap_remove(kind);
|
||||
let n = data.len();
|
||||
for (i, v) in flips {
|
||||
|
|
@ -218,3 +292,12 @@ proptest! {
|
|||
exercise(&data[..keep]);
|
||||
}
|
||||
}
|
||||
|
||||
/// The RW2 samples decode as they are, so the mutations above reach every format's sample decoder.
|
||||
#[test]
|
||||
fn rw2_samples_decode_unmutated() {
|
||||
for s in &rw2_formats() {
|
||||
let r = lightcraft_raw::decode(s).unwrap();
|
||||
assert_eq!((r.width, r.height), (W as usize, H as usize));
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -18,10 +18,7 @@ log = { workspace = true }
|
|||
[target.'cfg(not(target_arch = "wasm32"))'.dependencies]
|
||||
candle-core = { workspace = true }
|
||||
candle-nn = { workspace = true }
|
||||
rustls = { version = "0.23", default-features = false, features = ["std", "tls12"] }
|
||||
rustls-rustcrypto = "0.0.2-alpha"
|
||||
webpki-roots = "1"
|
||||
sha2 = { version = "0.10", default-features = false, features = ["std"] }
|
||||
lightcraft-fetch = { workspace = true }
|
||||
|
||||
[target.'cfg(target_os = "macos")'.dependencies]
|
||||
candle-core = { workspace = true, features = ["metal"] }
|
||||
|
|
|
|||
|
|
@ -1,39 +1,13 @@
|
|||
//! Downloading the SAM 3 model, only when the user asked for it: from an ordered list of mirrors
|
||||
//! (the next one when a mirror fails), resuming partial files, with connect and stall timeouts,
|
||||
//! a size cap, progress and cancellation, and checked (exact size, SHA-256 where pinned) before
|
||||
//! each file is moved into the model folder. Runs on whatever thread calls [`download`] (the
|
||||
//! engine uses a background thread; the UI thread never waits for the network).
|
||||
//!
|
||||
//! Files are written to `<name>.part` next to their final place and renamed when verified, so a
|
||||
//! model folder never holds a half-written file under a real name, and an interrupted download
|
||||
//! resumes from where it stopped. A file that fails its check is deleted, never kept.
|
||||
//! Downloading the SAM 3 model, only when the user asked for it. The downloader itself (mirrors,
|
||||
//! resuming, checks, the pure-Rust HTTPS client) is the `lightcraft-fetch` crate; this module
|
||||
//! says what SAM 3 needs and where it may come from.
|
||||
//!
|
||||
//! The weights are never part of LightCraft (SAM License, see docs/ai-masks.md); where they are
|
||||
//! downloaded from is configured by [`DEFAULT_MIRRORS`] and the user's own list.
|
||||
|
||||
pub mod http;
|
||||
pub use lightcraft_fetch::*;
|
||||
|
||||
use std::fs::{File, OpenOptions};
|
||||
use std::io::{Read, Write};
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
use std::time::Duration;
|
||||
|
||||
use sha2::{Digest, Sha256};
|
||||
|
||||
use http::{HttpError, Limits, Url};
|
||||
|
||||
/// One file of the model.
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct FileSpec {
|
||||
pub name: &'static str,
|
||||
/// Exact size in bytes, when pinned (then the download resumes and is checked against it).
|
||||
pub size: Option<u64>,
|
||||
/// SHA-256 (lower-case hex), when pinned.
|
||||
pub sha256: Option<&'static str>,
|
||||
/// Largest size accepted when `size` is not pinned.
|
||||
pub max: u64,
|
||||
}
|
||||
use std::path::Path;
|
||||
|
||||
/// The `facebook/sam3` checkpoint files [`crate::Sam3::load`] needs.
|
||||
///
|
||||
|
|
@ -71,354 +45,27 @@ pub const DEFAULT_MIRRORS: &[&str] = &[];
|
|||
/// tried before the defaults.
|
||||
pub const MIRRORS_ENV: &str = "LIGHTCRAFT_SAM3_MIRRORS";
|
||||
|
||||
/// Most mirrors used (a hostile list can't make a download loop forever).
|
||||
const MAX_MIRRORS: usize = 16;
|
||||
/// Redirects followed per request.
|
||||
const MAX_REDIRECTS: usize = 8;
|
||||
|
||||
/// The mirrors to try, in order: the environment variable's, then the mirrors file's (one base
|
||||
/// URL per line, `#` comments), then [`DEFAULT_MIRRORS`]; duplicates and unusable URLs dropped.
|
||||
/// The mirrors to try for SAM 3, in order: the environment variable's, then the mirrors file's
|
||||
/// (one base URL per line, `#` comments), then [`DEFAULT_MIRRORS`].
|
||||
pub fn mirrors(env: Option<&str>, file: Option<&Path>) -> Vec<String> {
|
||||
let mut out: Vec<String> = Vec::new();
|
||||
let mut add = |s: &str| {
|
||||
let s = s.trim().trim_end_matches('/');
|
||||
if !s.is_empty() && Url::parse(s).is_ok() && !out.iter().any(|m| m == s) && out.len() < MAX_MIRRORS {
|
||||
out.push(s.to_string());
|
||||
}
|
||||
};
|
||||
if let Some(env) = env {
|
||||
env.split([',', ' ', '\n', '\t']).for_each(&mut add);
|
||||
}
|
||||
// a small text file; anything larger isn't a mirror list
|
||||
if let Some(text) = file.and_then(|f| std::fs::metadata(f).ok().filter(|m| m.len() < 64 * 1024).and_then(|_| std::fs::read_to_string(f).ok())) {
|
||||
text.lines().map(|l| l.split('#').next().unwrap_or_default()).for_each(&mut add);
|
||||
}
|
||||
DEFAULT_MIRRORS.iter().for_each(|m| add(m));
|
||||
out
|
||||
lightcraft_fetch::mirrors(env, file, DEFAULT_MIRRORS)
|
||||
}
|
||||
|
||||
/// Download settings.
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct Options {
|
||||
/// Longest wait to connect to a mirror.
|
||||
pub connect_timeout: Duration,
|
||||
/// Longest wait for the next bytes once connected.
|
||||
pub stall_timeout: Duration,
|
||||
/// Tries per mirror and file (each resumes where the last one stopped).
|
||||
pub attempts: u32,
|
||||
}
|
||||
|
||||
impl Default for Options {
|
||||
fn default() -> Self {
|
||||
Options { connect_timeout: Duration::from_secs(15), stall_timeout: Duration::from_secs(30), attempts: 3 }
|
||||
}
|
||||
}
|
||||
|
||||
/// Where a download is.
|
||||
#[derive(Clone, Debug, Default, PartialEq, Eq)]
|
||||
pub struct Progress {
|
||||
pub file: String,
|
||||
/// Bytes of all files on disk so far, and the total expected (pinned sizes + announced ones).
|
||||
pub done: u64,
|
||||
pub total: u64,
|
||||
/// The mirror in use (without query).
|
||||
pub mirror: String,
|
||||
}
|
||||
|
||||
/// Why a download failed.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub enum DownloadError {
|
||||
/// No mirror is configured.
|
||||
NoMirrors,
|
||||
Cancelled,
|
||||
/// A local file problem (folder not writable, disk full…).
|
||||
Disk(String),
|
||||
/// Every mirror failed for `file`; the reason for each.
|
||||
AllMirrorsFailed {
|
||||
file: String,
|
||||
errors: Vec<String>,
|
||||
},
|
||||
}
|
||||
|
||||
impl std::fmt::Display for DownloadError {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
match self {
|
||||
DownloadError::NoMirrors => write!(
|
||||
f,
|
||||
"no download location is configured for the SAM 3 model in this build (set {MIRRORS_ENV}, or put the files in the model folder yourself; see docs/ai-masks.md)"
|
||||
),
|
||||
DownloadError::Cancelled => write!(f, "the download was cancelled"),
|
||||
DownloadError::Disk(e) => write!(f, "{e}"),
|
||||
DownloadError::AllMirrorsFailed { file, errors } => write!(f, "could not download {file}: {}", errors.join("; ")),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// One mirror's failure for one file; `Retry` resumes on the same mirror.
|
||||
enum Fail {
|
||||
Retry(String),
|
||||
NextMirror(String),
|
||||
Cancelled,
|
||||
Disk(String),
|
||||
}
|
||||
|
||||
/// Download `files` into `dir` from `mirrors` (in order; see [`mirrors`]). Files already there
|
||||
/// and verified are kept. `progress` is called as bytes arrive (often: throttle in it).
|
||||
pub fn download(
|
||||
files: &[FileSpec],
|
||||
mirrors: &[String],
|
||||
dir: &Path,
|
||||
opts: &Options,
|
||||
cancel: &AtomicBool,
|
||||
progress: &mut dyn FnMut(&Progress),
|
||||
) -> Result<(), DownloadError> {
|
||||
std::fs::create_dir_all(dir).map_err(|e| DownloadError::Disk(format!("can't create {}: {e}", dir.display())))?;
|
||||
// what's left to fetch
|
||||
let mut todo = Vec::new();
|
||||
for f in files {
|
||||
if !verified(f, &dir.join(f.name))? {
|
||||
todo.push(f);
|
||||
}
|
||||
}
|
||||
if todo.is_empty() {
|
||||
return Ok(());
|
||||
}
|
||||
if mirrors.is_empty() {
|
||||
return Err(DownloadError::NoMirrors);
|
||||
}
|
||||
let total: u64 = files.iter().map(|f| f.size.unwrap_or(0)).sum();
|
||||
let mut before: u64 = files.iter().filter(|f| !todo.contains(f)).filter_map(|f| f.size).sum();
|
||||
for f in todo {
|
||||
let mut errors = Vec::new();
|
||||
let mut ok = false;
|
||||
'mirrors: for m in mirrors {
|
||||
let url = format!("{m}/{}", f.name);
|
||||
for _ in 0..opts.attempts.max(1) {
|
||||
if cancel.load(Ordering::Relaxed) {
|
||||
return Err(DownloadError::Cancelled);
|
||||
}
|
||||
let mut report = |n: u64, announced: u64| {
|
||||
let mirror = Url::parse(m).map(|u| u.display()).unwrap_or_default();
|
||||
progress(&Progress {
|
||||
file: f.name.to_string(),
|
||||
done: before.saturating_add(n),
|
||||
total: total.max(before.saturating_add(announced)),
|
||||
mirror,
|
||||
})
|
||||
};
|
||||
match fetch_file(f, &url, dir, opts, cancel, &mut report) {
|
||||
Ok(()) => {
|
||||
ok = true;
|
||||
break 'mirrors;
|
||||
}
|
||||
Err(Fail::Cancelled) => return Err(DownloadError::Cancelled),
|
||||
Err(Fail::Disk(e)) => return Err(DownloadError::Disk(e)),
|
||||
Err(Fail::Retry(e)) => {
|
||||
log::warn!("SAM 3 download of {url}: {e} (retrying)");
|
||||
errors.push(format!("{}: {e}", short(m)));
|
||||
}
|
||||
Err(Fail::NextMirror(e)) => {
|
||||
log::warn!("SAM 3 download of {url}: {e}");
|
||||
errors.push(format!("{}: {e}", short(m)));
|
||||
continue 'mirrors;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if !ok {
|
||||
return Err(DownloadError::AllMirrorsFailed { file: f.name.to_string(), errors });
|
||||
}
|
||||
before = before.saturating_add(std::fs::metadata(dir.join(f.name)).map(|m| m.len()).unwrap_or(0));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// A mirror for messages (no query string: it may hold a token).
|
||||
fn short(m: &str) -> String {
|
||||
Url::parse(m).map(|u| u.display()).unwrap_or_else(|_| "mirror".into())
|
||||
}
|
||||
|
||||
/// Whether `path` is already the right file (exact size and hash when pinned).
|
||||
fn verified(f: &FileSpec, path: &Path) -> Result<bool, DownloadError> {
|
||||
let Ok(meta) = std::fs::metadata(path) else { return Ok(false) };
|
||||
if !meta.is_file() || meta.len() == 0 {
|
||||
return Ok(false);
|
||||
}
|
||||
if f.size.is_some_and(|s| s != meta.len()) || meta.len() > f.max {
|
||||
return Ok(false);
|
||||
}
|
||||
match f.sha256 {
|
||||
Some(want) => Ok(sha256_file(path).map_err(DownloadError::Disk)? == want),
|
||||
None => Ok(true),
|
||||
}
|
||||
}
|
||||
|
||||
fn sha256_file(path: &Path) -> Result<String, String> {
|
||||
let mut file = File::open(path).map_err(|e| format!("{}: {e}", path.display()))?;
|
||||
let mut h = Sha256::new();
|
||||
let mut buf = vec![0u8; 1 << 20];
|
||||
loop {
|
||||
let n = file.read(&mut buf).map_err(|e| format!("{}: {e}", path.display()))?;
|
||||
if n == 0 {
|
||||
break;
|
||||
}
|
||||
h.update(buf.get(..n).unwrap_or_default());
|
||||
}
|
||||
Ok(h.finalize().iter().map(|b| format!("{b:02x}")).collect())
|
||||
}
|
||||
|
||||
fn part_path(dir: &Path, name: &str) -> PathBuf {
|
||||
dir.join(format!("{name}.part"))
|
||||
}
|
||||
|
||||
/// Fetch one file from one mirror into `<name>.part` (resuming it when its size is pinned),
|
||||
/// check it and move it into place.
|
||||
fn fetch_file(f: &FileSpec, url: &str, dir: &Path, opts: &Options, cancel: &AtomicBool, report: &mut dyn FnMut(u64, u64)) -> Result<(), Fail> {
|
||||
let part = part_path(dir, f.name);
|
||||
let limit = f.size.unwrap_or(f.max);
|
||||
let disk = |e: std::io::Error| Fail::Disk(format!("{}: {e}", part.display()));
|
||||
// resume only a pinned file (an unpinned one can't be checked after resuming)
|
||||
let mut have = match std::fs::metadata(&part) {
|
||||
Ok(m) if f.size.is_some() && m.len() <= limit => m.len(),
|
||||
Ok(_) => {
|
||||
std::fs::remove_file(&part).map_err(disk)?;
|
||||
0
|
||||
}
|
||||
Err(_) => 0,
|
||||
};
|
||||
let fetched = f.size != Some(have) || have == 0;
|
||||
if fetched {
|
||||
let limits = Limits { connect: opts.connect_timeout, stall: opts.stall_timeout, cancel };
|
||||
let mut url = Url::parse(url).map_err(|e| Fail::NextMirror(e.to_string()))?;
|
||||
let origin = url.host.clone();
|
||||
let mut redirects = 0;
|
||||
let mut resp = loop {
|
||||
let mut headers = vec![];
|
||||
if have > 0 {
|
||||
headers.push(("Range", format!("bytes={have}-")));
|
||||
}
|
||||
// a token for the configured host only (never sent on to a redirect's host)
|
||||
if let Ok(t) = std::env::var("LIGHTCRAFT_SAM3_TOKEN")
|
||||
&& !t.trim().is_empty()
|
||||
&& url.tls
|
||||
&& url.host == origin
|
||||
{
|
||||
headers.push(("Authorization", format!("Bearer {}", t.trim())));
|
||||
}
|
||||
let r = http::get(&url, &headers, &limits).map_err(http_fail)?;
|
||||
match r.status {
|
||||
301 | 302 | 303 | 307 | 308 => {
|
||||
redirects += 1;
|
||||
if redirects > MAX_REDIRECTS {
|
||||
return Err(Fail::NextMirror("too many redirects".into()));
|
||||
}
|
||||
let loc = r.header("location").ok_or_else(|| Fail::NextMirror("redirect without a location".into()))?;
|
||||
url = url.join(loc).map_err(|e| Fail::NextMirror(e.to_string()))?;
|
||||
}
|
||||
_ => break r,
|
||||
}
|
||||
};
|
||||
let mut file = match resp.status {
|
||||
200 => {
|
||||
// the whole file (from the start, also when a range was asked for)
|
||||
if resp.content_length().is_some_and(|n| n > limit || f.size.is_some_and(|s| s != n)) {
|
||||
return Err(Fail::NextMirror(format!("the server's file has the wrong size ({:?} bytes)", resp.content_length())));
|
||||
}
|
||||
have = 0;
|
||||
File::create(&part).map_err(disk)?
|
||||
}
|
||||
206 if have > 0 => {
|
||||
let range = resp.header("content-range").unwrap_or_default();
|
||||
let (start, total) = parse_content_range(range).ok_or_else(|| Fail::NextMirror(format!("bad Content-Range `{range}`")))?;
|
||||
if start != have || total.is_some_and(|t| Some(t) != f.size) {
|
||||
return Err(Fail::NextMirror(format!("the server resumed at the wrong place ({range})")));
|
||||
}
|
||||
OpenOptions::new().append(true).open(&part).map_err(disk)?
|
||||
}
|
||||
416 => {
|
||||
// the partial file doesn't fit what the server has: start over
|
||||
std::fs::remove_file(&part).map_err(disk)?;
|
||||
return Err(Fail::Retry("the partial download didn't match; starting over".into()));
|
||||
}
|
||||
401 | 403 => return Err(Fail::NextMirror(format!("access denied (HTTP {})", resp.status))),
|
||||
404 | 410 => return Err(Fail::NextMirror(format!("not found (HTTP {})", resp.status))),
|
||||
s if s >= 500 => return Err(Fail::Retry(format!("server error (HTTP {s})"))),
|
||||
s => return Err(Fail::NextMirror(format!("HTTP {s}"))),
|
||||
};
|
||||
let announced = f.size.unwrap_or_else(|| resp.content_length().unwrap_or(0));
|
||||
report(have, announced);
|
||||
let mut buf = vec![0u8; 256 * 1024];
|
||||
loop {
|
||||
let n = match resp.read(&mut buf, &limits) {
|
||||
Ok(n) => n,
|
||||
Err(e) => {
|
||||
let _ = file.flush();
|
||||
if f.size.is_none() {
|
||||
drop(file);
|
||||
let _ = std::fs::remove_file(&part);
|
||||
}
|
||||
return Err(http_fail(e));
|
||||
}
|
||||
};
|
||||
if n == 0 {
|
||||
break;
|
||||
}
|
||||
if have.saturating_add(n as u64) > limit {
|
||||
drop(file);
|
||||
let _ = std::fs::remove_file(&part);
|
||||
return Err(Fail::NextMirror(format!("the server sent more than the expected {limit} bytes")));
|
||||
}
|
||||
file.write_all(buf.get(..n).unwrap_or_default()).map_err(disk)?;
|
||||
have = have.saturating_add(n as u64);
|
||||
report(have, announced);
|
||||
}
|
||||
file.sync_all().map_err(disk)?;
|
||||
}
|
||||
// checks: exact size, hash
|
||||
let len = std::fs::metadata(&part).map_err(disk)?.len();
|
||||
if f.size.is_some_and(|s| s != len) {
|
||||
return Err(Fail::Retry(format!("incomplete ({len} of {} bytes)", f.size.unwrap_or(0))));
|
||||
}
|
||||
if len == 0 {
|
||||
let _ = std::fs::remove_file(&part);
|
||||
return Err(Fail::NextMirror("the server sent an empty file".into()));
|
||||
}
|
||||
if let Some(want) = f.sha256 {
|
||||
let got = sha256_file(&part).map_err(Fail::Disk)?;
|
||||
if got != want {
|
||||
std::fs::remove_file(&part).map_err(disk)?;
|
||||
let why = format!("the file is damaged or not the official one (SHA-256 {got})");
|
||||
// a damaged partial file from an earlier run isn't this mirror's fault
|
||||
return Err(if fetched { Fail::NextMirror(why) } else { Fail::Retry(why) });
|
||||
}
|
||||
}
|
||||
std::fs::rename(&part, dir.join(f.name)).map_err(disk)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn http_fail(e: HttpError) -> Fail {
|
||||
match e {
|
||||
HttpError::Cancelled => Fail::Cancelled,
|
||||
HttpError::Stalled | HttpError::Io(_) | HttpError::Protocol(_) => Fail::Retry(e.to_string()),
|
||||
HttpError::BadUrl(_) | HttpError::Connect(_) | HttpError::Tls(_) => Fail::NextMirror(e.to_string()),
|
||||
}
|
||||
}
|
||||
|
||||
/// `bytes a-b/total` → (a, total) (`*` total → None).
|
||||
fn parse_content_range(v: &str) -> Option<(u64, Option<u64>)> {
|
||||
let v = v.trim().strip_prefix("bytes")?.trim();
|
||||
let (range, total) = v.split_once('/')?;
|
||||
let (a, b) = range.split_once('-')?;
|
||||
let (a, b) = (a.trim().parse::<u64>().ok()?, b.trim().parse::<u64>().ok()?);
|
||||
if b < a {
|
||||
return None;
|
||||
}
|
||||
let total = match total.trim() {
|
||||
"*" => None,
|
||||
t => Some(t.parse::<u64>().ok()?),
|
||||
};
|
||||
Some((a, total))
|
||||
/// What to tell the user when there is no mirror for the SAM 3 model.
|
||||
pub fn no_mirrors_message() -> String {
|
||||
format!(
|
||||
"no download location is configured for the SAM 3 model in this build (set {MIRRORS_ENV}, or put the files in the model folder yourself; see docs/ai-masks.md)"
|
||||
)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests;
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn sam3_mirrors_come_from_the_environment_the_file_and_the_defaults() {
|
||||
let m = mirrors(Some("https://a.example/x/"), Some(Path::new("/nonexistent/mirrors.txt")));
|
||||
assert_eq!(m.len(), 1 + DEFAULT_MIRRORS.len());
|
||||
assert!(no_mirrors_message().contains(MIRRORS_ENV));
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -231,18 +231,16 @@ impl Headless {
|
|||
self.paint()
|
||||
}
|
||||
|
||||
/// Wait for a fixture widget whose background directory listing may still be loading.
|
||||
/// Tests that browse a folder in the temp directory expect it to be a Local location of its
|
||||
/// own. Where the temp directory lies inside the home folder (Windows), the sidebar lists it
|
||||
/// inside Home's tree instead, so hide Home for the test; elsewhere this does nothing.
|
||||
#[cfg(test)]
|
||||
pub(crate) fn wait_for_widget(&mut self, id: &str, timeout: Duration) -> bool {
|
||||
let start = Instant::now();
|
||||
while let Some(remaining) = timeout.checked_sub(start.elapsed()) {
|
||||
let reply = self.request("ui.widgets", json!({"filter": id}), remaining);
|
||||
if reply["result"].as_array().is_some_and(|widgets| widgets.iter().any(|w| w["id"].as_str() == Some(id))) {
|
||||
return true;
|
||||
}
|
||||
std::thread::sleep(Duration::from_millis(1));
|
||||
pub(crate) fn hide_home_above(&mut self, path: &std::path::Path) {
|
||||
let home = std::env::var("HOME").or_else(|_| std::env::var("USERPROFILE")).unwrap_or_default();
|
||||
if !home.is_empty() && lightcraft_catalog::query::folder_within(&path.to_string_lossy(), &home) {
|
||||
let r = self.request("engine.execute", json!({"command": "local.hide", "params": {"path": home}}), Duration::from_secs(10));
|
||||
assert_eq!(r["ok"], true, "{r}");
|
||||
}
|
||||
false
|
||||
}
|
||||
|
||||
/// Send a control-protocol request (see [`crate::control`]) and run frames until it is
|
||||
|
|
@ -294,6 +292,8 @@ mod tests {
|
|||
fn demo_grid_snapshot_has_ui_pixels() {
|
||||
let t0 = Instant::now();
|
||||
let mut h = demo([1200.0, 760.0]);
|
||||
// settle() can see a quiet spell before the first thumbnails land on a loaded machine
|
||||
h.step_until(SETTLE, |h| h.app.renderer.thumb_textures() > 0);
|
||||
let img = h.snapshot(SETTLE);
|
||||
eprintln!("headless snapshot: {:?} in {:?} ({} frames)", img.size, t0.elapsed(), h.frames());
|
||||
assert_eq!(img.size, [1200, 760]);
|
||||
|
|
@ -398,6 +398,7 @@ mod tests {
|
|||
let id = h.app.session.active().unwrap();
|
||||
let photo = |h: &Headless| h.app.session.catalog.photo(id).unwrap().clone();
|
||||
let before = photo(&h);
|
||||
h.step_until(SETTLE, |h| h.app.renderer.variant_textures() >= 6);
|
||||
assert!(h.app.renderer.variant_textures() >= 6, "variant thumbnails rendered: {}", h.app.renderer.variant_textures());
|
||||
// hover: the loupe shows the look, nothing is committed
|
||||
let r = h.request("ui.hoverWidget", json!({"id": "profileCell:lc.vivid"}), t);
|
||||
|
|
@ -900,6 +901,7 @@ mod tests {
|
|||
};
|
||||
let ids = |h: &mut Headless| -> Vec<String> { rects(h).into_iter().map(|(id, _)| id).collect() };
|
||||
h.request("ui.set", json!({"leftPanel": true}), t);
|
||||
h.hide_home_above(&dir);
|
||||
// Browse Folder… browses the picked folder and keeps it in Local within one frame; a
|
||||
// request runs frames, so keep it first (no frame sees it browsed but not yet kept)
|
||||
let r = h.request("engine.execute", json!({"command": "local.addRoot", "params": {"path": path}}), t);
|
||||
|
|
@ -907,8 +909,11 @@ mod tests {
|
|||
let r = h.request("engine.execute", json!({"command": "library.browse", "params": {"path": path}}), t);
|
||||
assert_eq!(r["ok"], true, "{r}");
|
||||
h.settle(SETTLE);
|
||||
assert!(h.wait_for_widget(&format!("source:local:{path}"), t), "the browsed folder is listed: {:?}", ids(&mut h));
|
||||
assert!(!ids(&mut h).iter().any(|i| i == "source:local:restoreHidden"));
|
||||
let row = format!("source:local:{path}");
|
||||
assert!(h.step_until(t, |h| h.app.widgets.iter().any(|(w, _)| *w == row)), "the browsed folder is listed: {:?}", ids(&mut h));
|
||||
// the "Show hidden locations" row appears only while something is hidden
|
||||
let hidden_before = !h.app.ui.hidden_locations.is_empty();
|
||||
assert_eq!(ids(&mut h).iter().any(|i| i == "source:local:restoreHidden"), hidden_before);
|
||||
// hide it while it is still being browsed
|
||||
let r = h.request("engine.execute", json!({"command": "local.hide", "params": {"path": path}}), t);
|
||||
assert_eq!(r["ok"], true, "{r}");
|
||||
|
|
@ -930,7 +935,11 @@ mod tests {
|
|||
assert_eq!(h.request("ui.clickWidget", json!({"id": "source:local:restoreHidden"}), t)["ok"], true);
|
||||
h.settle(SETTLE);
|
||||
assert!(h.app.ui.hidden_locations.is_empty());
|
||||
assert!(ids(&mut h).contains(&format!("source:local:{path}")));
|
||||
assert!(!ids(&mut h).iter().any(|i| i == "source:local:restoreHidden"), "nothing left to restore");
|
||||
if !hidden_before {
|
||||
// (with Home hidden for this test, the folder is back inside Home's tree instead)
|
||||
assert!(ids(&mut h).contains(&format!("source:local:{path}")));
|
||||
}
|
||||
let _ = std::fs::remove_dir_all(&dir);
|
||||
}
|
||||
|
||||
|
|
@ -947,12 +956,10 @@ mod tests {
|
|||
std::fs::create_dir_all(base.join(d)).unwrap();
|
||||
}
|
||||
let s = |p: std::path::PathBuf| p.to_string_lossy().to_string();
|
||||
let (photos, day1, day2, other) = (
|
||||
s(base.join("Photos")),
|
||||
s(base.join("Photos").join("2026").join("20260101")),
|
||||
s(base.join("Photos").join("2026").join("20260114")),
|
||||
s(base.join("Other")),
|
||||
);
|
||||
// joined by component: the sidebar's ids spell paths with the platform's separator
|
||||
let year = base.join("Photos").join("2026");
|
||||
let (photos, day1, day2, other) = (s(base.join("Photos")), s(year.join("20260101")), s(year.join("20260114")), s(base.join("Other")));
|
||||
h.hide_home_above(&base);
|
||||
let exec = |h: &mut Headless, c: &str, p: Value| h.request("engine.execute", json!({"command": c, "params": p}), t);
|
||||
let rects = |h: &mut Headless| -> std::collections::HashMap<String, f64> {
|
||||
let w = h.request("ui.widgets", json!({"filter": "lc-ui-roots-"}), t);
|
||||
|
|
@ -969,7 +976,8 @@ mod tests {
|
|||
exec(&mut h, "library.browse", json!({"path": day1}));
|
||||
h.settle(SETTLE);
|
||||
h.step();
|
||||
assert!(h.wait_for_widget(&format!("source:local:{day2}"), t), "the sibling listing did not arrive");
|
||||
let row = format!("source:local:{day2}");
|
||||
assert!(h.step_until(t, |h| h.app.widgets.iter().any(|(w, _)| *w == row)), "the sibling listing did not arrive");
|
||||
let r = rects(&mut h);
|
||||
assert!(r.contains_key(&format!("source:local:{photos}")), "the kept root stays: {r:?}");
|
||||
assert!(r.contains_key(&format!("source:local:{day2}")), "the sibling stays reachable: {r:?}");
|
||||
|
|
@ -982,7 +990,8 @@ mod tests {
|
|||
assert_eq!(exec(&mut h, "library.browse", json!({"path": other}))["ok"], true);
|
||||
h.settle(SETTLE);
|
||||
h.step();
|
||||
assert!(h.wait_for_widget(&format!("source:local:{other}"), t), "the other location did not arrive");
|
||||
let row = format!("source:local:{other}");
|
||||
assert!(h.step_until(t, |h| h.app.widgets.iter().any(|(w, _)| *w == row)), "the other location did not arrive");
|
||||
let r = rects(&mut h);
|
||||
assert!(r.contains_key(&format!("source:local:{photos}")) && r.contains_key(&format!("source:local:{other}")), "{r:?}");
|
||||
// restart: the kept roots come back with the saved UI state
|
||||
|
|
@ -1094,6 +1103,7 @@ mod tests {
|
|||
let r = h.request("engine.execute", json!({"command": "file.addFromDevice", "params": {"path": sub.to_string_lossy()}}), t);
|
||||
assert_eq!(r["ok"], true, "{r}");
|
||||
h.settle(SETTLE);
|
||||
h.step_until(SETTLE, |h| matches!(h.app.ui.dialog, Some(crate::state::Dialog::Import { .. })));
|
||||
let Some(crate::state::Dialog::Import { opts }) = &h.app.ui.dialog else { panic!("no import review") };
|
||||
assert!(opts.copy);
|
||||
let _ = std::fs::remove_dir_all(&dir);
|
||||
|
|
@ -1308,6 +1318,8 @@ mod tests {
|
|||
let r = h.request("engine.execute", json!({"command": "file.addPhotos", "params": {"paths": [dir.to_string_lossy()]}}), t);
|
||||
assert_eq!(r["result"]["scanning"], true, "{r}");
|
||||
h.settle(SETTLE);
|
||||
h.step_until(SETTLE, |h| matches!(h.app.ui.dialog, Some(crate::state::Dialog::Import { .. })));
|
||||
h.step_until(SETTLE, |h| h.app.renderer.textures.keys().filter(|s| matches!(s, crate::render::Slot::Import(_))).count() >= 5);
|
||||
let Some(crate::state::Dialog::Import { opts }) = &h.app.ui.dialog else { panic!("no import review") };
|
||||
assert_eq!(opts.candidates.len(), 6);
|
||||
assert_eq!(opts.candidates.iter().filter(|c| c.duplicate.is_some()).count(), 1);
|
||||
|
|
@ -1440,6 +1452,7 @@ mod tests {
|
|||
let r = h.request("engine.execute", json!({"command": "file.addPhotos", "params": {"paths": [src.to_string_lossy()]}}), t);
|
||||
assert_eq!(r["result"]["scanning"], true, "{r}");
|
||||
h.settle(SETTLE);
|
||||
h.step_until(SETTLE, |h| matches!(h.app.ui.dialog, Some(crate::state::Dialog::Import { .. })));
|
||||
let Some(crate::state::Dialog::Import { opts }) = &h.app.ui.dialog else { panic!("no import review") };
|
||||
assert_eq!(opts.candidates.len(), 10);
|
||||
for id in ["button:importCopy", "button:importDest"] {
|
||||
|
|
|
|||
|
|
@ -366,6 +366,7 @@ fn local_folder_tree_expands_and_browses() {
|
|||
std::fs::create_dir_all(base.join(".hidden")).unwrap();
|
||||
let mut h = detail("panel.edit");
|
||||
exec(&mut h, "view.leftPanel", json!({"show": true}));
|
||||
h.hide_home_above(&base);
|
||||
exec(&mut h, "local.addRoot", json!({"path": base.to_string_lossy()}));
|
||||
exec(&mut h, "library.browse", json!({"path": base.to_string_lossy()}));
|
||||
h.settle(SETTLE);
|
||||
|
|
|
|||
|
|
@ -170,7 +170,7 @@ wait and return the result (when built with the `sam` feature; the default CLI h
|
|||
with 2-D RoPE and windowed attention, the feature pyramids, the SAM 2-style prompt encoder and
|
||||
two-way mask decoder for clicks, and the CLIP text encoder, DETR encoder/decoder (box
|
||||
relative-position bias, presence token) and pixel decoder for text. The CLIP tokenizer is a
|
||||
small BPE in `tokenizer.rs`. `fetch/` is the downloader (original code).
|
||||
small BPE in `tokenizer.rs`. The downloader (original code) is the `lightcraft-fetch` crate, shared by any model that is downloaded; `fetch/` here says what SAM 3 needs.
|
||||
- candle is pinned at 0.9.2: later releases make `candle-core` depend on `tokenizers` with the
|
||||
Oniguruma C library, and the product is pure Rust. Weights are read with positional reads (no
|
||||
memory map: `unsafe` stays in `lightcraft-sysmem`), only the tensors a path needs, and every
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# Sony ARW and Nikon NEF starting look
|
||||
# Sony ARW, Nikon NEF and Panasonic RW2 starting look
|
||||
|
||||
ARW and NEF decoding supply a Bayer mosaic and camera white-balance multipliers, but currently no measured camera colour matrix. Treating camera RGB as linear sRGB produces a dark, muted default rendering. The loader now estimates a file-local starting look from that file's embedded camera JPEG, only when a usable camera matrix is absent. All output pixels still come from the RAW mosaic; there is no JPEG replacement or uniform saturation boost.
|
||||
|
||||
|
|
@ -40,6 +40,12 @@ Measuring this found a decoder bug: 12-bit NEFs store maker note `0x003d` BlackL
|
|||
|
||||
On one sample, Exposure +1 EV, Saturation +100, Contrast +50, warm/cool WB, tint, highlights and shadows all changed the output. The same matrix and curve were selected for 400-pixel and full-resolution renders. Headless UI checks wait until the loupe reports `source: render` and no jobs remain; the camera JPEG stand-in does not count as verification.
|
||||
|
||||
### Panasonic RW2
|
||||
|
||||
The same fit, gates and relative WB apply to Panasonic RW2, Leica RWL and the older Panasonic RAW files (all decode as `RawFormat::Rw2`; white balance from tags `0x0024`–`0x0026`). Their embedded JPEG (`JpgFromRaw`, 1920 px wide) always shows the whole active area, while the default crop is the aspect ratio set in the camera (4:3 on the 3:2 sensors, 1:1, 16:9, 2.71 on the S1R II…); when the preview matches the active area rather than the crop, the reference is cropped to the default crop before fitting.
|
||||
|
||||
Public CC0 samples from raw.pixls.us (178 files from 118 Panasonic and Leica bodies; the four from 2005–2007 carry no preview), rendered at 1200 px and compared with each file's embedded JPEG (mean CIE76 ΔE at 200 px): 140 of 174 fits accepted, from 93 of the 114 bodies with a preview, median ΔE 5.0 (1.2–21.3) with matching mean lightness (median |ΔL*| 0.4); 124 of them use a hue/saturation table. (Fitted with the matrix and tone curve alone, 137 were accepted at median ΔE 6.1; every one of those is as close or closer with the table and chroma curve, by 0.75 on average.) On the 14 corpus files with a preview the neutral fallback scored ΔE 9.6–27 and rendered up to 16 L* too dark; 13 fits were accepted, ΔE 2.8–8.9, lightness within 2 L*. Of the 34 files left on the neutral fallback, 21 fail the gates, almost all compacts and kit zooms (TZ/ZS, FZ2500, D-Lux, V-Lux, GF2, GX80 with its 12–32 mm): their camera JPEG is distortion-corrected, the pixel pairs no longer line up and the held-out RMS stays above 0.10 (GX80: 0.138 → 0.107); two S1 II shots just miss it (0.103, 0.107). The other 13 never reach the gates, mostly nearly colourless scenes (grey walls, concrete: under 5% coloured pixels) or dark ones (a night sky, a near-black frame). Panasonic's embedded distortion data (tag `0x0119`) is not used yet, and `lightcraft-cli calibrate` doesn't collect RW2 files: the misaligned pairs of distortion-corrected JPEGs would distort a pooled profile.
|
||||
|
||||
This is a per-file camera-look estimate, **not measured spectral calibration or Lightroom parity**. Camera local tone, picture styles, noise reduction and lens warps cannot all be reproduced by a global matrix/curve; same-aspect crops may evade the aspect check. Skin, mixed-light scenes and saturated highlights can still differ from a supplied JPEG. Broader Sony and Nikon model/lighting coverage, chart-based calibration, absolute WB and other manufacturers remain unverified. Rejected fits and other RAW formats retain their existing behaviour.
|
||||
|
||||
## Camera profiles (pooled per model)
|
||||
|
|
|
|||
|
|
@ -59,8 +59,8 @@ Ordered by user impact, then tier, then effort. The checklist above counts featu
|
|||
that decide whether a photographer can switch (see the honest assessment in [ROADMAP.md](../ROADMAP.md#where-we-stand)).
|
||||
Take the first one nobody is working on.
|
||||
|
||||
1. **LR-PROF-CAMERACOLOR** (P0): our own camera colour calibration. Sony ARW and Nikon NEF now get a guarded file-local fit to their own embedded JPEG (colour matrix + hue/saturation/value table + tone and chroma curves, relative WB; per-model profiles pooled from many photos via `lightcraft-cli calibrate`; ARW: 7 of 10 public samples accepted, mean ΔE vs the camera JPEG 17–26 → 3–10; NEF: 13 of 13 decodable samples from 6 bodies accepted, ΔE 13–46 → 3–8, one mixed-light scene 13 → 12); measured calibration and fidelity remain missing. Other non-DNG raws and rejected fits still use a neutral matrix. Expand validated preview fitting and use matrices the files carry themselves; never Adobe data.
|
||||
2. **LR-IMP-FORMATS** (P0): **CR3** first (every Canon body since ~2018), then compressed RAF / ORF, RW2 v4, Nikon
|
||||
1. **LR-PROF-CAMERACOLOR** (P0): our own camera colour calibration. Sony ARW, Nikon NEF and Panasonic RW2 now get a guarded file-local fit to their own embedded JPEG (colour matrix + hue/saturation/value table + tone and chroma curves, relative WB; per-model profiles pooled from many photos via `lightcraft-cli calibrate` for ARW and NEF; ARW: 7 of 10 public samples accepted, mean ΔE vs the camera JPEG 17–26 → 3–10; NEF: 13 of 13 decodable samples from 6 bodies accepted, ΔE 13–46 → 3–8, one mixed-light scene 13 → 12; RW2: 140 of 174 public samples accepted, from 93 of the 114 bodies with a preview, median ΔE 5.0); measured calibration and fidelity remain missing. Other non-DNG raws and rejected fits still use a neutral matrix. Expand validated preview fitting and use matrices the files carry themselves; never Adobe data.
|
||||
2. **LR-IMP-FORMATS** (P0): **CR3** first (every Canon body since ~2018), then compressed RAF / ORF, Nikon
|
||||
"lossy after split" NEF, Canon sRAW; HEIC/AVIF decode. Clean-room, from prose descriptions only (see
|
||||
`crates/raw/src/vendor/nefc.rs` for how compressed NEF was done). Until decoded, such photos are `preview_only`.
|
||||
3. **LR-IMP-CAMERA-COVERAGE** (P0): per-model verification; grow the CC0 corpus and fix per-model bugs (like the CR2
|
||||
|
|
@ -124,8 +124,8 @@ under the UI command that wraps it (a no-op error); the UI command now wins.
|
|||
| LR-IMP-PROFILES | Import profiles & presets | P1 | ✅ | `cmd:file.importPresets`, `cmd:preset.import`, `cmd:profile.import`, `crates/engine/src/preset_import.rs`, `crates/engine/src/preset_luminar.rs` | presets: .lcpreset, XMP, classic .lrtemplate, photos carrying edits ("DNG presets"), Luminar looks (.lmp, .mplumpack collections; sliders with a counterpart), .zip bundles, folders (folder → group), drag & drop; masks carried over; unmapped settings reported; profiles: .cube 3D LUTs; Adobe profile formats deliberately unsupported |
|
||||
| LR-IMP-LOCAL | Work on files in place | P0 | ✅ | `cmd:library.browse`, `cmd:photo.addToLibrary`, `cmd:library.import` (mode add), `cmd:local.addRoot`, `cmd:local.hide`, `cmd:local.restoreHidden`, `crates/engine/src/cmd/browse.rs`, `crates/ui-egui/src/panels/left.rs` (`local_section`), `crates/ui-egui/src/panels/grid.rs` (`folder_header`) | Local: browse Pictures / Desktop / Downloads / Home or any folder without adding it (breadcrumb, Include subfolders, Add N to My Photos); browsed photos stay out of All Photos, albums and counts; folders picked with Browse Folder… (or Keep in Local / Add to Local on a folder row) stay listed across restarts, and browsing a folder inside a listed one highlights it inside that tree (opened down to it, siblings reachable) instead of adding a root; edits go to XMP sidecars; importing promotes them; right-click a top-level location → Remove from Local hides the shortcut only (nothing on disk or in the catalog changes), saved with the UI state, and “Show N hidden locations” (or Browse Folder… on it) brings it back; a folder is one location however its path is spelled (`/` or `\`, trailing separator, `.`/`..`, drive-letter case; `folder_key` in `crates/catalog/src/query.rs`); browsed photos nobody changed are forgotten once their folder has not been browsed for 30 days (Settings → Performance → Local folders, 0 = never; checked when the library opens, or `cmd:library.forgetLocal` with `dryRun`; files and sidecars stay, browsing again brings them back; see `crates/catalog/src/local.rs` for what counts as a change) |
|
||||
| LR-IMP-SIDECAR-SPLIT | Separate XMP sidecar variants | P2 | ⬜ | `cmd:library.xmpPreferences` | sidecar naming option exists (stem/full), no split sidecars |
|
||||
| LR-IMP-FORMATS | Supported formats | P0 | 🟡 | `crates/codecs/src/lib.rs`, `crates/raw/src/lib.rs` | JPEG, PNG, TIFF, WebP, JXL, PSD, GIF, BMP; DNG (incl. lossy / Smart Preview DNG), CR2 (colour-filter layout from the file's `CR2CFAPattern` tag, issue #85), ARW (pre-2017 bodies: white balance and black level from enciphered maker-note / `SR2SubIFD` data, issue #148), NEF (uncompressed, lossless and lossy compressed 12/14-bit, `crates/raw/src/vendor/nefc.rs`), RAF, RW2, PEF, ORF. 12-bit NEF black level: maker note `0x003d` is in 14-bit units (D750, D780, D850, D7500, Z 50 checked). CR3 opens from its full-size embedded JPEG with full metadata (Exif, GPS, XMP from the `CMT1`/`CMT2`/`CMT4` boxes, `crates/meta/src/cr3.rs`) but its raw data is not decoded yet. Missing: CR3 raw decoding (CRX), compressed RAF/ORF, RW2 v4, NEFs labelled compressed (34713) but stored uncompressed without a `0x0096` table (Z 6 packed 14-bit, D850 12-bit uncompressed), NEF "lossy after split" (preview only: imported with `preview_only` = the decoder's reason, rendered as a rendered JPEG, flagged by a grid badge and loupe/Edit/Info notices, `previewOnly` in `catalog.query`; Reload clears it once the file decodes), HEIC/AVIF decode |
|
||||
| LR-IMP-CAMERA-COVERAGE | Camera coverage, verified per model | P0 | 🟡 | `crates/raw/tests/corpus.rs`, `xtask/src/main.rs` | a supported container is not the same as every camera that writes it decoding correctly: ~40 CC0 corpus files (plus ~60 NEFs checked by hand) against the >1,000 models Lightroom lists; per-model differences still surface (e.g. the CR2 colour-filter layout differed by model until it was read from the file's own tag, issue #85, checked on 35 CR2 bodies; pre-2017 Sony ARWs opened green until their scrambled white balance / black level were read, issue #148, checked on 16 ARWs from 15 Sony bodies). Grow the CC0 corpus per model and test each sample decodes with plausible colour |
|
||||
| LR-IMP-FORMATS | Supported formats | P0 | 🟡 | `crates/codecs/src/lib.rs`, `crates/raw/src/lib.rs` | JPEG, PNG, TIFF, WebP, JXL, PSD, GIF, BMP; DNG (incl. lossy / Smart Preview DNG), CR2 (colour-filter layout from the file's `CR2CFAPattern` tag, issue #85), ARW (pre-2017 bodies: white balance and black level from enciphered maker-note / `SR2SubIFD` data, issue #148), NEF (uncompressed, lossless and lossy compressed 12/14-bit, `crates/raw/src/vendor/nefc.rs`), RAF, RW2 / Leica RWL / Panasonic RAW in every raw format (compressed 4 and 6, the prefix-coded strips of format 8 from the GH6 on, packed 2/5/7, the 16-bit words of the 2005–2007 bodies; black levels, in-camera aspect crops and mapped-out defect pixels; 178 CC0 files from 118 bodies checked, `crates/raw/src/vendor/rw2.rs`), PEF, ORF. 12-bit NEF black level: maker note `0x003d` is in 14-bit units (D750, D780, D850, D7500, Z 50 checked). CR3 opens from its full-size embedded JPEG with full metadata (Exif, GPS, XMP from the `CMT1`/`CMT2`/`CMT4` boxes, `crates/meta/src/cr3.rs`) but its raw data is not decoded yet. Missing: CR3 raw decoding (CRX), compressed RAF/ORF, NEFs labelled compressed (34713) but stored uncompressed without a `0x0096` table (Z 6 packed 14-bit, D850 12-bit uncompressed), NEF "lossy after split" (preview only: imported with `preview_only` = the decoder's reason, rendered as a rendered JPEG, flagged by a grid badge and loupe/Edit/Info notices, `previewOnly` in `catalog.query`; Reload clears it once the file decodes), HEIC/AVIF decode |
|
||||
| LR-IMP-CAMERA-COVERAGE | Camera coverage, verified per model | P0 | 🟡 | `crates/raw/tests/corpus.rs`, `xtask/src/main.rs` | a supported container is not the same as every camera that writes it decoding correctly: ~55 CC0 corpus files (plus ~60 NEFs checked by hand and 178 Panasonic / Leica RW2, RWL and RAW files from 118 bodies, checked against each file's own JPEG) against the >1,000 models Lightroom lists; per-model differences still surface (e.g. the CR2 colour-filter layout differed by model until it was read from the file's own tag, issue #85, checked on 35 CR2 bodies; pre-2017 Sony ARWs opened green until their scrambled white balance / black level were read, issue #148, checked on 16 ARWs from 15 Sony bodies). Grow the CC0 corpus per model and test each sample decodes with plausible colour |
|
||||
| LR-IMP-CULL-AT-IMPORT | Culling analysis at import | P2 | 🟡 | `cmd:photo.analyze` | run Assisted Culling on the imported photos (they're selected after an import); not automatic |
|
||||
| LR-IMP-MOVE | Move on import [Classic] | P1 | ✅ | `cmd:library.import` (`mode` move), `crates/engine/src/import_move.rs`, `crates/engine/src/tests_import_move.rs`, `crates/ui-egui/src/import.rs` | Transfer ▸ Move with Copy's destination, folders (day / month / one folder / custom template), rename template and example destination; XMP sidecars (both namings) move along; each source is removed only after its destination is written (hard link on the same volume, else copied, synced and compared byte for byte) and its catalog record is saved; failed, duplicate, unchecked files keep their sources, taken names get -1, -2…, never overwritten; files already in the destination/library are added in place; sources that can't be removed (read-only card) are kept and reported (`kept`). Undo removes the photos from the library but leaves the moved files at the destination. No Copy as DNG while moving |
|
||||
| LR-IMP-DNG-CONVERT | Convert to DNG on import [Classic] | P2 | ✅ | `cmd:library.import` (`dng`), `crates/ui-egui/src/import.rs` | copy imports: Raw files ▸ Copy as DNG (lossless; the card is untouched); the raw copy is removed only once its DNG is verified and on disk (issue #106) |
|
||||
|
|
@ -276,7 +276,7 @@ under the UI command that wraps it (a no-op error); the UI command now wins.
|
|||
| LR-PROF-ADOBE | Standard raw looks (own equivalents) | P0 | ✅ | `crates/engine/src/presets.rs` (`PROFILES`), `crates/pipeline/src/profiles.rs` | six own looks: Color, Neutral, Vivid, Landscape, Portrait, Monochrome |
|
||||
| LR-PROF-ADAPTIVE | Adaptive profiles | P2 | ⬜ | | |
|
||||
| LR-PROF-CAMERA | Camera-matching looks | P2 | ⬜ | | |
|
||||
| LR-PROF-CAMERACOLOR | Camera colour calibration (own) | P0 | 🟡 | `crates/raw/src/color.rs`, `crates/raw/src/profile.rs`, `crates/engine/src/camera_preview.rs` | DNG files use the colour matrices and the camera-profile look they carry (`ProfileHueSatMap`, `ProfileLookTable`, `ProfileToneCurve`, read from the file at run time per DNG spec ch. 6 — Lightroom-converted DNGs rendered flat and muted without them, issue #138); Sony ARW and Nikon NEF get a guarded file-local fit to their own embedded JPEG — chromaticity matrix + hue/saturation/value table + tone and chroma curves, relative WB — or, with a local per-model profile pooled from many photos (`lightcraft-cli calibrate`, `crates/engine/src/camera_profiles.rs`), the profile's colour and only their own tone/chroma curves (on 29 held-out ILCE-7M4 photos mean ΔE vs the camera JPEG 3.54 → 3.10), kept with smart previews (see docs/camera-preview-colour.md); ARW: on 10 public raw.pixls.us samples from 8 bodies 7 fits were accepted, mean ΔE vs the camera JPEG 17–26 → 3–10; NEF: 13 of 13 decodable samples (D750, D780, D850, D7500, Z 50 and local D7500 shots) accepted, ΔE 13–46 → 3–8 (issue #150); this is not measured camera calibration; other raws and rejected fits use a neutral fallback (camera RGB ≈ linear sRGB, `matrix_is_fallback`) with as-shot white balance, so colours are muted and not accurate. Needs our own per-camera calibration: matrices the files carry themselves (e.g. Olympus `ColorMatrix`), fitting each camera to its own embedded JPEG, then chart shots. Adobe matrices / DCPs are never used. Biggest image-quality gap today |
|
||||
| LR-PROF-CAMERACOLOR | Camera colour calibration (own) | P0 | 🟡 | `crates/raw/src/color.rs`, `crates/raw/src/profile.rs`, `crates/engine/src/camera_preview.rs` | DNG files use the colour matrices and the camera-profile look they carry (`ProfileHueSatMap`, `ProfileLookTable`, `ProfileToneCurve`, read from the file at run time per DNG spec ch. 6 — Lightroom-converted DNGs rendered flat and muted without them, issue #138); Sony ARW, Nikon NEF and Panasonic RW2 get a guarded file-local fit to their own embedded JPEG — chromaticity matrix + hue/saturation/value table + tone and chroma curves, relative WB — or, with a local per-model profile pooled from many photos (`lightcraft-cli calibrate`, ARW and NEF so far, `crates/engine/src/camera_profiles.rs`), the profile's colour and only their own tone/chroma curves (on 29 held-out ILCE-7M4 photos mean ΔE vs the camera JPEG 3.54 → 3.10), kept with smart previews (see docs/camera-preview-colour.md); ARW: on 10 public raw.pixls.us samples from 8 bodies 7 fits were accepted, mean ΔE vs the camera JPEG 17–26 → 3–10; NEF: 13 of 13 decodable samples (D750, D780, D850, D7500, Z 50 and local D7500 shots) accepted, ΔE 13–46 → 3–8 (issue #150); RW2 / RWL: 140 of 174 public samples accepted, from 93 of the 114 bodies with a preview, median ΔE 5.0 with matching lightness (13 of the 14 corpus files with a preview: ΔE 9.6–27 → 2.8–8.9; most rejections are compacts and kit zooms whose camera JPEG is distortion-corrected); this is not measured camera calibration; other raws and rejected fits use a neutral fallback (camera RGB ≈ linear sRGB, `matrix_is_fallback`) with as-shot white balance, so colours are muted and not accurate. Needs our own per-camera calibration: matrices the files carry themselves (e.g. Olympus `ColorMatrix`), fitting each camera to its own embedded JPEG, then chart shots. Adobe matrices / DCPs are never used. Biggest image-quality gap today |
|
||||
| LR-PROF-CREATIVE | Creative profiles (own) | P2 | ✅ | `cmd:develop.profile`, `crates/pipeline/src/profiles.rs` | 16 own looks in Film / Cinematic / Muted / B&W (tone + point-curve fades, colour grading, mixer / B&W mix); scale with `ctl:profile.amount`; sliders untouched |
|
||||
| LR-PROF-LEGACY | Legacy profiles | P2 | ⬜ | | |
|
||||
| LR-PROF-NONRAW | Profiles for non-raw files | P0 | ✅ | `cmd:develop.profile` | same looks apply to JPEG/TIFF |
|
||||
|
|
|
|||
|
|
@ -153,6 +153,12 @@
|
|||
now develop from the raw data instead of the camera's embedded JPEG, so a B&W or other picture style set in the
|
||||
camera no longer gets baked in. Photos already imported as "preview only" switch over on Reload. (Files that
|
||||
Nikon splits into two differently compressed halves still use the preview for now.)
|
||||
- Panasonic and Leica raws (RW2, RWL and the older RAW files, DMC-LX1 to DC-S1R II) develop from the raw data in
|
||||
every format the cameras write. Most bodies opened as preview only before: GH1–GH5, the G, GX, GF, GM, FZ, LX and
|
||||
TZ/ZS series, Leica D-Lux, V-Lux and C-Lux, and the GH6, GH7, G9 II, S5 II and S9 generation. They start from a
|
||||
colour and tone look fitted to the camera's own JPEG, as Nikon and Sony raws do, framed in the aspect ratio set in
|
||||
the camera; mapped-out sensor defects are filled in. `.rwl` and `.raw` files are imported too. Photos already
|
||||
imported as "preview only" switch over on Reload.
|
||||
- Sony ARWs from before about 2017 (RX100, RX100 II–V, RX10, NEX, SLT, ILCE-6000, A7 / A7 II / A7R II and their
|
||||
siblings) no longer open bright green (issue #148): their as-shot white balance and black level are read from the
|
||||
file (Sony stores them only in scrambled maker-note data on these bodies), the few columns of padding at the right
|
||||
|
|
|
|||
|
|
@ -0,0 +1,45 @@
|
|||
{
|
||||
"nodes": {
|
||||
"craft-fonts": {
|
||||
"flake": false,
|
||||
"locked": {
|
||||
"lastModified": 1791295400,
|
||||
"narHash": "sha256-e+6HpOYoAFTh6mNfGBBw2njnBzcE8W8+Jybv1+dKOCA=",
|
||||
"owner": "storytold",
|
||||
"repo": "craft-fonts",
|
||||
"rev": "abb83316d96aa59c1cf64784289e378fe9fa5695",
|
||||
"type": "github"
|
||||
},
|
||||
"original": {
|
||||
"owner": "storytold",
|
||||
"repo": "craft-fonts",
|
||||
"rev": "abb83316d96aa59c1cf64784289e378fe9fa5695",
|
||||
"type": "github"
|
||||
}
|
||||
},
|
||||
"nixpkgs": {
|
||||
"locked": {
|
||||
"lastModified": 1791260994,
|
||||
"narHash": "sha256-Miqqk/ammqnTUxaoCyvtwoPeLbI1ksFyLxi/CazZpWY=",
|
||||
"owner": "NixOS",
|
||||
"repo": "nixpkgs",
|
||||
"rev": "151fa4e8ddfdd8dd25d945ad94ed54a13de9f6e4",
|
||||
"type": "github"
|
||||
},
|
||||
"original": {
|
||||
"owner": "NixOS",
|
||||
"ref": "nixos-unstable",
|
||||
"repo": "nixpkgs",
|
||||
"type": "github"
|
||||
}
|
||||
},
|
||||
"root": {
|
||||
"inputs": {
|
||||
"craft-fonts": "craft-fonts",
|
||||
"nixpkgs": "nixpkgs"
|
||||
}
|
||||
}
|
||||
},
|
||||
"root": "root",
|
||||
"version": 7
|
||||
}
|
||||
|
|
@ -0,0 +1,115 @@
|
|||
{
|
||||
description = "LightCraft — photo library and non-destructive raw developer, in pure Rust";
|
||||
|
||||
inputs = {
|
||||
# LightCraft needs edition 2024 and rustc ≥ 1.90, so the flake tracks unstable. Consumers on a
|
||||
# release channel can retarget it: `inputs.lightcraft.inputs.nixpkgs.follows = "nixpkgs";`
|
||||
nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
|
||||
|
||||
# Optional build input of the repository: the CJK fonts (BIZ UDPGothic, BIZ UDMincho, Noto Sans CJK SC,
|
||||
# Shippori Mincho) that `crates/engine/build.rs` embeds when `CRAFT_FONTS_DIR` is set. Official
|
||||
# releases always build with them; without them everything works but Japanese text has no
|
||||
# glyphs. Pinned to the revision the release workflow uses — bump deliberately
|
||||
# (`nix flake update craft-fonts`); craftrules standards/fonts.md.
|
||||
craft-fonts = {
|
||||
url = "github:storytold/craft-fonts/abb83316d96aa59c1cf64784289e378fe9fa5695";
|
||||
flake = false;
|
||||
};
|
||||
};
|
||||
|
||||
outputs =
|
||||
{
|
||||
self,
|
||||
nixpkgs,
|
||||
craft-fonts,
|
||||
}:
|
||||
let
|
||||
inherit (nixpkgs) lib;
|
||||
|
||||
# No x86_64-darwin: the pinned nixpkgs (26.11, "nixos-unstable") dropped support for it. On
|
||||
# an Intel Mac, retarget nixpkgs at a release that still has it, or install the .dmg from the
|
||||
# GitHub release.
|
||||
systems = [
|
||||
"x86_64-linux"
|
||||
"aarch64-linux"
|
||||
"aarch64-darwin"
|
||||
];
|
||||
|
||||
# The AppStream metadata carries a release date. The flake source's own commit date is
|
||||
# reproducible, unlike "now"; a tarball or dirty tree without one falls back to the epoch.
|
||||
buildDate =
|
||||
let
|
||||
d = self.lastModifiedDate or "";
|
||||
in
|
||||
if builtins.stringLength d >= 8 then
|
||||
"${builtins.substring 0 4 d}-${builtins.substring 4 2 d}-${builtins.substring 6 2 d}"
|
||||
else
|
||||
"1970-01-01";
|
||||
|
||||
packageArgs = {
|
||||
inherit craft-fonts buildDate;
|
||||
};
|
||||
|
||||
forAllSystems = f: lib.genAttrs systems (system: f (import nixpkgs { inherit system; }) system);
|
||||
in
|
||||
{
|
||||
packages = forAllSystems (
|
||||
pkgs: _:
|
||||
let
|
||||
lightcraft = pkgs.callPackage ./nix/package.nix packageArgs;
|
||||
in
|
||||
{
|
||||
inherit lightcraft;
|
||||
default = lightcraft;
|
||||
}
|
||||
);
|
||||
|
||||
# `nixpkgs.overlays = [ inputs.lightcraft.overlays.default ]` → `pkgs.lightcraft` everywhere.
|
||||
overlays.default = final: _prev: {
|
||||
lightcraft = final.callPackage ./nix/package.nix packageArgs;
|
||||
};
|
||||
|
||||
devShells = forAllSystems (
|
||||
pkgs: system:
|
||||
let
|
||||
lightcraft = self.packages.${system}.lightcraft;
|
||||
in
|
||||
{
|
||||
default = pkgs.mkShell (
|
||||
{
|
||||
# The package's own native inputs, so `cargo build` in the shell needs no extra setup.
|
||||
inputsFrom = [ lightcraft ];
|
||||
packages = with pkgs; [
|
||||
cargo
|
||||
rustc
|
||||
clippy
|
||||
rustfmt
|
||||
rust-analyzer
|
||||
];
|
||||
# `cargo run` dlopens the Vulkan loader and libxkbcommon like the packaged binary.
|
||||
LD_LIBRARY_PATH = lib.makeLibraryPath lightcraft.passthru.runtimeLibs;
|
||||
RUST_BACKTRACE = "1";
|
||||
}
|
||||
// lib.optionalAttrs (craft-fonts != null) {
|
||||
# CJK glyphs in `cargo run` too; build.rs reads this as CRAFT_FONTS_DIR.
|
||||
CRAFT_FONTS_DIR = toString craft-fonts;
|
||||
}
|
||||
);
|
||||
}
|
||||
);
|
||||
|
||||
checks = forAllSystems (
|
||||
pkgs: system: {
|
||||
lightcraft = self.packages.${system}.lightcraft;
|
||||
|
||||
# `nix fmt` formats the flake; this keeps it formatted.
|
||||
formatting = pkgs.runCommand "lightcraft-nix-formatting" { nativeBuildInputs = [ pkgs.nixfmt ]; } ''
|
||||
nixfmt --check ${./flake.nix} ${./nix}
|
||||
touch $out
|
||||
'';
|
||||
}
|
||||
);
|
||||
|
||||
formatter = forAllSystems (pkgs: _: pkgs.nixfmt);
|
||||
};
|
||||
}
|
||||
|
|
@ -0,0 +1,206 @@
|
|||
# LightCraft — the desktop app (`lightcraft`) and the CLI/MCP server (`lightcraft-cli`), built from
|
||||
# this repository:
|
||||
#
|
||||
# nix build # → ./result/bin/lightcraft, ./result/bin/lightcraft-cli
|
||||
#
|
||||
# Kept in step with the release workflow (.github/workflows/release.yml) and packaging/env.sh:
|
||||
# * `Cargo.lock` drives the dependencies (`cargoLock`), so there is no vendorHash to bump — only
|
||||
# crates.io, no git sources;
|
||||
# * `CRAFT_FONTS_DIR` embeds the CJK fonts from storytold/craft-fonts (the `craft-fonts`
|
||||
# flake input); without it everything builds and runs, but Japanese and Chinese text have no glyphs;
|
||||
# * the desktop file, hicolor icons and AppStream metadata are the same files the .deb/.rpm ship
|
||||
# (packaging/linux/), so `apt` and NixOS users see one identical LightCraft;
|
||||
# * `doCheck` runs `cargo test --workspace`, what `cargo xtask ci` runs. `nix build` runs it too;
|
||||
# `pkgs.lightcraft.overrideAttrs { doCheck = false; }` skips it for a faster, build-only install.
|
||||
#
|
||||
# Everything in the product is pure Rust (no C/C++ dependencies), so this needs no build system
|
||||
# beyond cargo plus the windowing headers winit's build scripts look for.
|
||||
{
|
||||
lib,
|
||||
stdenv,
|
||||
rustPlatform,
|
||||
pkg-config,
|
||||
makeWrapper,
|
||||
# Windowing/GPU libraries: needed at build time by the crates that link them, at run time by the
|
||||
# ones that open them with dlopen() (see `runtimeLibs`), and by the dev shell (flake.nix).
|
||||
wayland,
|
||||
wayland-scanner,
|
||||
libxkbcommon,
|
||||
libx11,
|
||||
libxcb,
|
||||
libxcursor,
|
||||
libxi,
|
||||
libxrandr,
|
||||
vulkan-loader,
|
||||
# Validates the desktop file and the AppStream metadata we install (as CI does).
|
||||
desktop-file-utils,
|
||||
appstream,
|
||||
# Adds the driver search path (/run/opengl-driver/lib) to the binaries' RPATH so wgpu finds the
|
||||
# installed Vulkan driver on NixOS. Optional: harmless on nixpkgs without it.
|
||||
autoAddDriverRunpath ? null,
|
||||
# storytold/craft-fonts checkout (flake input). `null` builds without CJK glyphs.
|
||||
craft-fonts ? null,
|
||||
# Release date for the AppStream metadata (YYYY-MM-DD); the flake derives it from its own commit.
|
||||
buildDate ? "1970-01-01",
|
||||
}:
|
||||
|
||||
let
|
||||
root = ../.;
|
||||
|
||||
# The version lives in exactly one place: `[workspace.package] version` in the root Cargo.toml.
|
||||
version = (builtins.fromTOML (builtins.readFile (root + "/Cargo.toml"))).workspace.package.version;
|
||||
|
||||
isLinux = stdenv.hostPlatform.isLinux;
|
||||
|
||||
# AppStream id: also the desktop file name, and the icon name in share/icons/hicolor.
|
||||
appId = "ai.storyteller.lightcraft";
|
||||
|
||||
# Libraries the binaries open at run time with dlopen(): nothing links them, so no RPATH points at
|
||||
# them — winit loads libxkbcommon/libxcb, wgpu the Vulkan loader (NixOS patches that loader to
|
||||
# find the drivers in /run/opengl-driver). The wrapper below points LD_LIBRARY_PATH here.
|
||||
runtimeLibs = lib.optionals isLinux [
|
||||
libxkbcommon
|
||||
wayland
|
||||
libx11
|
||||
libxcb
|
||||
libxcursor
|
||||
libxi
|
||||
libxrandr
|
||||
vulkan-loader
|
||||
];
|
||||
|
||||
# The two native binaries. The wasm app (apps/lightcraft-web) is built by `cargo xtask web`, not
|
||||
# here; xtask is tooling.
|
||||
binaries = [
|
||||
"lightcraft"
|
||||
"lightcraft-cli"
|
||||
];
|
||||
in
|
||||
rustPlatform.buildRustPackage {
|
||||
pname = "lightcraft";
|
||||
inherit version;
|
||||
|
||||
src = lib.cleanSourceWith {
|
||||
src = lib.cleanSource root;
|
||||
# The Nix files are not part of the build: editing them must not rebuild any Rust.
|
||||
filter =
|
||||
path: _type:
|
||||
!(builtins.elem (baseNameOf path) [
|
||||
"flake.nix"
|
||||
"flake.lock"
|
||||
"nix"
|
||||
]);
|
||||
};
|
||||
|
||||
cargoLock.lockFile = root + "/Cargo.lock";
|
||||
|
||||
# Both binaries, as the .deb/.rpm install them (`packaging/linux/package.sh`).
|
||||
cargoBuildFlags = [
|
||||
"--locked"
|
||||
"-p"
|
||||
"lightcraft"
|
||||
"-p"
|
||||
"lightcraft-cli"
|
||||
];
|
||||
|
||||
# `cargo xtask ci` runs `cargo test --workspace`. Tests that need what the sandbox cannot have
|
||||
# skip themselves with a note: corpus/ (gitignored, `cargo xtask corpus --download`) and the GPU
|
||||
# equivalence tests (no adapter).
|
||||
cargoTestFlags = [ "--workspace" ];
|
||||
|
||||
# One test thread at a time (RUST_TEST_THREADS=1). `lightcraft-catalog`'s `tests_lock` tests share
|
||||
# a process-global resource: `flock` is inherited across `fork`, so a child process that another
|
||||
# test forked keeps the parent's just-released library lock alive until its `execve` closes the
|
||||
# (CLOEXEC) fd — long enough that the sibling test sees a spurious `LockError::InUse`, on ~30 % of
|
||||
# runs on a many-core machine. Serializing the tests keeps `nix build` deterministic without
|
||||
# touching the repository. The underlying window (any `fork` in a process holding the library
|
||||
# lock — the app spawns `xdg-open`/`open`/`explorer` and `sysctl`) is a product-level question:
|
||||
# `LibraryLock::acquire` could retry on `WouldBlock` for a few hundred ms before reporting InUse.
|
||||
dontUseCargoParallelTests = true;
|
||||
|
||||
strictDeps = true;
|
||||
|
||||
nativeBuildInputs = [
|
||||
pkg-config
|
||||
]
|
||||
++ lib.optionals isLinux [
|
||||
makeWrapper
|
||||
wayland-scanner
|
||||
desktop-file-utils
|
||||
appstream
|
||||
]
|
||||
++ lib.optionals (isLinux && autoAddDriverRunpath != null) [ autoAddDriverRunpath ];
|
||||
|
||||
buildInputs = lib.optionals isLinux [
|
||||
wayland
|
||||
libxkbcommon
|
||||
libx11
|
||||
libxcb
|
||||
libxcursor
|
||||
libxi
|
||||
libxrandr
|
||||
];
|
||||
|
||||
env = lib.optionalAttrs (craft-fonts != null) {
|
||||
CRAFT_FONTS_DIR = toString craft-fonts;
|
||||
# A malformed font input must fail the build rather than silently ship without glyphs.
|
||||
CRAFT_FONTS_REQUIRED = "1";
|
||||
};
|
||||
|
||||
postInstall = ''
|
||||
# Desktop integration: byte-for-byte the files the .deb/.rpm/.AppImage install.
|
||||
install -Dm644 packaging/linux/${appId}.desktop $out/share/applications/${appId}.desktop
|
||||
install -Dm644 packaging/linux/${appId}.mime.xml $out/share/mime/packages/${appId}.xml
|
||||
cp -r assets/app-icon/hicolor $out/share/icons/
|
||||
install -d $out/share/metainfo
|
||||
substitute packaging/linux/${appId}.metainfo.xml.in \
|
||||
$out/share/metainfo/${appId}.metainfo.xml \
|
||||
--subst-var-by VERSION ${version} \
|
||||
--subst-var-by DATE ${buildDate}
|
||||
|
||||
# Licences for everything embedded in the binaries: Inter, the app icon, each craft font.
|
||||
doc=$out/share/doc/lightcraft
|
||||
install -Dm644 -t "$doc" \
|
||||
README.md LICENSE-MIT LICENSE-APACHE NOTICE \
|
||||
assets/ATTRIBUTION.md assets/fonts/OFL-Inter.txt
|
||||
''
|
||||
+ lib.optionalString (craft-fonts != null) ''
|
||||
for lic in ${toString craft-fonts}/fonts/*/OFL.txt; do
|
||||
install -Dm644 "$lic" "$doc/OFL-$(basename "$(dirname "$lic")").txt"
|
||||
done
|
||||
''
|
||||
+ lib.optionalString isLinux ''
|
||||
desktop-file-validate $out/share/applications/${appId}.desktop
|
||||
XDG_CACHE_HOME=$TMPDIR/.cache appstreamcli validate --no-net \
|
||||
$out/share/metainfo/${appId}.metainfo.xml
|
||||
|
||||
# makeWrapper keeps the binaries working when the libs above are only dlopen'd.
|
||||
for bin in ${lib.concatStringsSep " " (map (b: "$out/bin/" + b) binaries)}; do
|
||||
wrapProgram "$bin" --prefix LD_LIBRARY_PATH : "${lib.makeLibraryPath runtimeLibs}"
|
||||
done
|
||||
'';
|
||||
|
||||
passthru = {
|
||||
# Shared with the dev shell so `cargo run` inside `nix develop` finds the same dlopen'd libs.
|
||||
inherit runtimeLibs;
|
||||
};
|
||||
|
||||
meta = {
|
||||
description = "Photo library and non-destructive raw developer";
|
||||
longDescription = ''
|
||||
LightCraft organises a photo library and develops raw files non-destructively: masks,
|
||||
presets, colour grading, local adjustments and batch export, driven by the same command
|
||||
layer as its UI. It ships with lightcraft-cli, a headless renderer, command runner and MCP
|
||||
server for AI agents.
|
||||
'';
|
||||
homepage = "https://getartcraft.com/apps/lightcraft";
|
||||
license = with lib.licenses; [
|
||||
mit
|
||||
asl20
|
||||
];
|
||||
sourceProvenance = with lib.sourceTypes; [ fromSource ];
|
||||
platforms = lib.platforms.linux ++ lib.platforms.darwin;
|
||||
# `nix run` and the desktop entry both start the GUI.
|
||||
mainProgram = "lightcraft";
|
||||
};
|
||||
}
|
||||
|
|
@ -9,4 +9,4 @@ Terminal=false
|
|||
Categories=Graphics;2DGraphics;RasterGraphics;Photography;
|
||||
Keywords=photo;raw;camera;library;develop;
|
||||
StartupWMClass=ai.storyteller.lightcraft
|
||||
MimeType=image/jpeg;image/png;image/tiff;image/webp;image/x-adobe-dng;image/x-sony-arw;image/x-canon-cr2;image/x-canon-cr3;image/x-nikon-nef;image/x-fuji-raf;image/x-olympus-orf;image/x-panasonic-rw2;image/x-pentax-pef;
|
||||
MimeType=image/jpeg;image/png;image/tiff;image/webp;image/x-adobe-dng;image/x-sony-arw;image/x-canon-cr2;image/x-canon-cr3;image/x-nikon-nef;image/x-fuji-raf;image/x-olympus-orf;image/x-panasonic-rw2;image/x-panasonic-rw;image/x-pentax-pef;
|
||||
|
|
|
|||
|
|
@ -9,7 +9,8 @@ use std::process::Command;
|
|||
/// Extensions treated as assets (lower-case).
|
||||
const ASSET_EXT: &[&str] = &[
|
||||
"png", "jpg", "jpeg", "gif", "webp", "avif", "svg", "ico", "icns", "bmp", "ttf", "otf", "woff", "woff2", "wav", "mp3", "ogg", "flac", "mp4",
|
||||
"mov", "tif", "tiff", "dng", "cr2", "cr3", "nef", "arw", "raf", "orf", "rw2", "pef", "heic", "jxl", "psd", "cube", "icc", "icm", "xmp",
|
||||
"mov", "tif", "tiff", "dng", "cr2", "cr3", "nef", "arw", "raf", "orf", "rw2", "rwl", "raw", "pef", "heic", "jxl", "psd", "cube", "icc", "icm",
|
||||
"xmp",
|
||||
];
|
||||
/// Adobe-specific formats we never ship (camera/lens profiles, Lightroom templates).
|
||||
const FORBIDDEN_EXT: &[&str] = &["dcp", "lcp", "lrtemplate", "lrcat", "lrsmcol", "aco", "ase", "abr"];
|
||||
|
|
|
|||
|
|
@ -38,6 +38,7 @@ pub const TABLE: &[(&str, Class)] = &[
|
|||
("raster", Class::Layer(0)),
|
||||
("tiff", Class::Layer(0)),
|
||||
("sysmem", Class::Layer(0)),
|
||||
("fetch", Class::Layer(0)),
|
||||
("raw", Class::Layer(1)),
|
||||
("codecs", Class::Layer(1)),
|
||||
("meta", Class::Layer(1)),
|
||||
|
|
|
|||
|
|
@ -360,11 +360,22 @@ const RAW_SAMPLES: &[(&str, &str)] = &[
|
|||
"raf-fuji-xt20-compressed.raf",
|
||||
"https://raw.pixls.us/getfile.php/1178/nice/Fujifilm%20-%20X-T20%20-%2014bit%2014bit%20compressed%20%283:2%29.RAF",
|
||||
),
|
||||
("raw-panasonic-fz50.raw", "https://raw.pixls.us/getfile.php/2234/nice/Panasonic%20-%20DMC-FZ50%20-%204:3.RAW"),
|
||||
("raw-panasonic-fz8.raw", "https://raw.pixls.us/getfile.php/2282/nice/Panasonic%20-%20DMC-FZ8%20-%204:3.RAW"),
|
||||
("rw2-panasonic-fz1000m2-4x3.rw2", "https://raw.pixls.us/getfile.php/4706/nice/Panasonic%20-%20DC-FZ10002%20-%204:3.RW2"),
|
||||
("rw2-panasonic-g9-b.rw2", "https://raw.pixls.us/getfile.php/2348/nice/Panasonic%20-%20DC-G9%20-%204:3.RW2"),
|
||||
("rw2-panasonic-g9.rw2", "https://raw.pixls.us/getfile.php/2585/nice/Panasonic%20-%20DC-G9%20-%204:3.RW2"),
|
||||
("rw2-panasonic-gh1.rw2", "https://raw.pixls.us/getfile.php/1323/nice/Panasonic%20-%20DMC-GH1%20-%204:3.RW2"),
|
||||
("rw2-panasonic-gh5.rw2", "https://raw.pixls.us/getfile.php/1517/nice/Panasonic%20-%20DC-GH5%20-%204:3.RW2"),
|
||||
("rw2-panasonic-gh5m2.rw2", "https://raw.pixls.us/getfile.php/5082/nice/Panasonic%20-%20DC-GH5M2%20-%204:3.RW2"),
|
||||
("rw2-panasonic-gh5s.rw2", "https://raw.pixls.us/getfile.php/2603/nice/Panasonic%20-%20DC-GH5S%20-%204:3.RW2"),
|
||||
("rw2-panasonic-gh6.rw2", "https://raw.pixls.us/getfile.php/5876/nice/Panasonic%20-%20DC-GH6%20-%204:3.RW2"),
|
||||
("rw2-panasonic-gx80.rw2", "https://raw.pixls.us/getfile.php/1569/nice/Panasonic%20-%20DMC-GX80%20-%204:3.RW2"),
|
||||
("rw2-panasonic-s1.rw2", "https://raw.pixls.us/getfile.php/3038/nice/Panasonic%20-%20DC-S1%20-%203:2.RW2"),
|
||||
("rw2-panasonic-s5-format7.rw2", "https://raw.pixls.us/getfile.php/6339/nice/Panasonic%20-%20DC-S5%20-%203:2.RW2"),
|
||||
("rw2-panasonic-s5m2.rw2", "https://raw.pixls.us/getfile.php/7790/nice/Panasonic%20-%20DC-S5M2%20-%2014bit%20%283:2%29.RW2"),
|
||||
("rw2-panasonic-s9.rw2", "https://raw.pixls.us/getfile.php/7702/nice/Panasonic%20-%20DC-S9%20-%203:2.RW2"),
|
||||
("rwl-leica-dlux7.rwl", "https://raw.pixls.us/getfile.php/4204/nice/Leica%20-%20D-Lux%207%20-%204:3.RWL"),
|
||||
];
|
||||
|
||||
fn cmd_corpus(download: bool) -> Result<(), String> {
|
||||
|
|
|
|||
Loading…
Reference in New Issue