132 lines
8.1 KiB
Markdown
132 lines
8.1 KiB
Markdown
# AI Denoise
|
||
|
||
Edit ▸ Detail ▸ **AI Denoise** is a per-photo switch above **Amount** (0–100).
|
||
Turning it on starts at 50, or preserves an already positive Amount. Turning it off
|
||
preserves Amount and mixes none of the cleaned picture. It affects only the requesting
|
||
photo, independently of Auto Sync, and is undoable. Manual luminance and colour noise
|
||
reduction remain separate controls.
|
||
|
||
If no model is selected, turning it on opens **Settings ▸ AI Denoise** without changing
|
||
the photo. After installation, the pending request enables the original photo even if
|
||
the selection changed. Cancel the pending action in Settings. Cancelled downloads,
|
||
deleted or relinked photos, switching libraries and quitting cancel the request.
|
||
An active slider gesture finishes before the pending action resumes.
|
||
|
||
## Models and terms
|
||
|
||
**No model weights are bundled.** Models live in the per-user `denoise-models` folder
|
||
(override with `LIGHTCRAFT_DENOISE_MODELS`). Settings provides **Install from file…**
|
||
for a user-supplied ONNX file with a `denoise-model.json` beside it. The default build
|
||
offers no model downloads. The separate `rawnind-model` Cargo feature enables the
|
||
pinned RawNIND offer only when a release's model policy permits it.
|
||
Terms must be acknowledged before installation or
|
||
download. Downloads use the shared pure-Rust `lightcraft-fetch` client, with resume,
|
||
progress, cancellation, size limits and SHA-256 verification.
|
||
|
||
RawNIND's optional ONNX weights are **GPL-3.0**, as recorded in the pinned model catalog.
|
||
They are downloaded separately after consent and read by our own Rust executor.
|
||
Whether releases should offer this download remains a maintainer policy decision;
|
||
permissively licensed or independently trained weights remain a gap. This is not a
|
||
claim that optional downloading settles licensing questions. No GPL decoder or neural
|
||
runtime code is included. Model metadata is in `crates/denoise-core/src/known.rs`.
|
||
|
||
Installation checks the manifest, model hash and size, then runs a synthetic self-test:
|
||
finite output of the promised shape, repeatability, the declared scale and reduced
|
||
noise. Unsupported operators, malformed files and a failed self-test return errors.
|
||
The UI performs copying, extraction, hashing, parsing and CPU/GPU self-tests on one
|
||
background installation worker. `denoise.models.downloads` and `denoise.pump` report
|
||
`installing`, `installed` or `failed`; an existing model is preserved on a failed replacement.
|
||
Command clients can request the same asynchronous path with `background: true`.
|
||
The synchronous command remains available for scripts that require the result immediately.
|
||
|
||
The current contract is `bayerToRgb`: normalized, black-subtracted, non-white-balanced
|
||
RGGB planes `[1, 4, tile, tile]` produce camera RGB `[1, 3, 2*tile, 2*tile]`.
|
||
RawNIND uses a 512-cell tile, 64-cell overlap and a mean-matched nominal gain of 1e6.
|
||
See `DenoiserManifest` for the validated JSON schema.
|
||
|
||
## Photos, previews and exports
|
||
|
||
Original RAW files stay untouched. A cleaned picture is cached under
|
||
`<library>/denoise/<key>.lcdn`, keyed by source content, model and algorithm version.
|
||
The compressed half-float cache has checked headers, bounded windows and checksums.
|
||
It can be cleared or recreated; it is not a new catalog photo or a duplicate DNG.
|
||
|
||
Plain and cleaned camera RGB go through the same highlight reconstruction, white
|
||
balance, colour matrix, DNG profile tables, camera hue/saturation look, crop and
|
||
orientation. Amount blends the two before the remaining develop pipeline, so edits
|
||
and exports use the same data. At zero Amount or with Detail disabled, the ordinary
|
||
render path remains in use.
|
||
|
||
The queue prioritizes the open and selected photos, pauses after input, and adjusts
|
||
its worker count while the user works. Progress, cancellation and retry are available
|
||
in Settings and through commands. The default cache limit is 20 GB; unused products
|
||
are evicted oldest first. A failed file is not continuously retried.
|
||
|
||
An export with a nonzero effective Amount creates a missing product synchronously.
|
||
It returns a clear error if it cannot produce the denoised result; it never silently
|
||
exports the plain image instead. Unsupported sensor layouts retain manual noise
|
||
reduction and report their limitation.
|
||
|
||
## Pure-Rust CPU and GPU execution
|
||
|
||
The CPU executor reads a bounded subset of ONNX into checked network data. Convolutions
|
||
use bounded im2col blocks and `faer-core 0.17.1` with its pure-Rust GEMM backend,
|
||
without BLAS, C dependencies or an inner thread pool. That backend includes
|
||
architecture-specific assembly in its dependencies. Tiles share the outer worker
|
||
pool. A scalar implementation checks the optimized operators on synthetic networks.
|
||
Tract is not a workspace dependency or a product feature.
|
||
CPU inference and the GPU model module are feature-gated; ordinary engine/web builds
|
||
do not compile the faer/GEMM backend. The engine depends on the inference crate only through `denoise = ["dep:lightcraft-denoise", …]`;
|
||
lightweight `lightcraft-denoise-core` metadata and cache formats remain available without inference. Pictures are capped at 100 megapixels; at most
|
||
four tile calls run together, chosen within the process-wide working-memory budget.
|
||
One idle CPU workspace is retained, and cache costs reserve room for the blended picture.
|
||
|
||
The GPU executor uses our WGSL kernels through wgpu. It checks a real model's output
|
||
against the CPU and falls back on errors, unsupported networks or a slower device.
|
||
Setup has a time limit and a crash marker. Automatic selection estimates throughput
|
||
from a check tile; this estimate can choose imperfectly on unmeasured hardware.
|
||
Settings offers Automatic, Graphics card and Processor. GPU tests cover supported
|
||
synthetic networks against the scalar executor; a real-model test compares CPU/GPU.
|
||
|
||
### Historical performance, measured 2026-10-07
|
||
|
||
These are the contributor's measurements **before the current upstream integration**,
|
||
on a Ryzen 9 7945HX and RTX 4090 Laptop GPU (DX12), release build, model SHA-256
|
||
`da27509dab6a2915da67e988acd86cf71f9d5bbc8d1aa0ed32933578a887b901`.
|
||
The 24.34 MP Sony a7 III photo is 35 tiles. Numbers exclude model loading, source
|
||
decode, GPU setup, cache encoding and export, but include packing and blending.
|
||
|
||
| Work | Pure-Rust CPU | Prior Tract reference | GPU |
|
||
|---|---:|---:|---:|
|
||
| One warm 512-cell tile, one caller | 1148 ms | 1463 ms | 15.1 ms |
|
||
| Whole photo, 16 CPU / 4 GPU feeding workers | 5.185 s | 9.534 s | 0.582 s |
|
||
|
||
The largest relative differences were 1.68e-6 CPU/reference and 1.57e-6 CPU/GPU for
|
||
the check tile, below the required 1e-3 tolerance. Reference comparisons were made
|
||
on the older contributor branch; that reference runtime is intentionally excluded
|
||
from the current workspace. These numbers are not a fresh benchmark of this rebase.
|
||
|
||
Fresh checks can be run using an already installed model and CC0 raw samples:
|
||
|
||
```text
|
||
LC_DENOISE_MODEL=<model.onnx> cargo test --release -p lightcraft-gpu --lib real_model_on_the_gpu_matches_cpu -- --ignored --nocapture
|
||
LC_DENOISE_MODEL=<model.onnx> LC_DENOISE_RAW=<corpus/raw> cargo test --release -p lightcraft-engine --features rawnind-model --test denoise_cpu -- --ignored --nocapture
|
||
LC_DENOISE_MODEL=<model.onnx> LC_DENOISE_RAW=<corpus/raw> cargo test --release -p lightcraft-engine --features rawnind-model --test denoise_real -- --ignored --nocapture
|
||
```
|
||
|
||
## Commands and limits
|
||
|
||
UI, CLI, control channel and MCP dispatch `denoise.toggle`, `denoise.settings`,
|
||
`denoise.queue`, `.cancel`, `.status`, `.pump`, `.clear`, and `denoise.models.*`
|
||
(`list`, `install`, `download`, `downloads`, `downloadCancel`, `test`, `remove`,
|
||
`select`). The develop control is `enhance.denoise`.
|
||
|
||
Native apps enable inference through their `denoise` feature. A build without it and
|
||
the web build report inference unavailable. Manifest/cache primitives still compile
|
||
for WASM. Settings and models remain opt-in; nothing is downloaded on launch.
|
||
|
||
This implementation supports **Bayer RAW only**. X-Trans, Foveon, demosaiced DNG and
|
||
non-RAW images remain unsupported by this model contract. No measured quality parity
|
||
with Lightroom is claimed. Apple/Metal and wider hardware verification, model policy,
|
||
a linear-RGB contract and a side-by-side fidelity suite remain future work.
|