LR-PROF-CAMERACOLOR: fit the camera look away from edges when a lens-corrected JPEG misaligns (#232)
Sony A7R II camera JPEGs are distortion-corrected by default while the raw is not. On the public CC0 ILCE-7RM2 samples the JPEG is up to 15 px of 1440 off, and the misaligned window frames alone pushed the held-out RMS to 0.108-0.115 (gate 0.10): the photo opened in the grey neutral fallback, 13 L* too dark. The same shot with distortion correction off fits at 0.059. When the fit on all pixels fails the gates, try it once more, with the same gates, on pixels whose 3x3 neighbourhood in either proxy spans a luminance ratio of at most 3. Fits that already pass are unchanged (29 other corpus ARW/NEF/RW2 files render pixel-identical). ILCE-7RM2 ΔE76 vs the camera JPEG (registered) 16.6-16.9 -> 3.7-3.8; corpus GX80 19.1 -> 7.9. RENDER_CACHE_VERSION 11 -> 12.
This commit is contained in:
radialmonster
parent
629e39380e
commit
15feb6e1fc
|
|
@ -102,7 +102,7 @@ pub(crate) fn profile_pairs(raw: &RawImage, bytes: &[u8]) -> Option<Vec<([f64; 3
|
|||
let transform = lightcraft_raw::color::camera_transform(raw, lightcraft_raw::color::as_shot_white_xy(raw));
|
||||
let (sensor, reference) = proxies(raw, bytes, &transform, PROFILE_PROXY)?;
|
||||
let to_camera = transform.matrix.inverse()?;
|
||||
let (pairs, _) = collect_pairs(&sensor, &reference)?;
|
||||
let (pairs, _) = collect_pairs(&sensor, &reference, None)?;
|
||||
Some(pairs.into_iter().map(|(x, y)| (to_camera.apply(x), y)).collect())
|
||||
}
|
||||
|
||||
|
|
@ -150,10 +150,31 @@ fn fit_pairs(sensor: &Rgb32f, reference: &Rgb32f) -> Option<CameraLook> {
|
|||
fit_pairs_with(sensor, reference, None)
|
||||
}
|
||||
|
||||
/// Largest luminance ratio within the 3×3 neighbourhood of pixel `i` (1 on flat areas, large on edges).
|
||||
fn local_contrast(image: &Rgb32f, i: usize) -> f32 {
|
||||
let (w, h) = (image.width, image.height);
|
||||
if w == 0 {
|
||||
return f32::INFINITY;
|
||||
}
|
||||
let (x, y) = (i % w, i / w);
|
||||
let (mut lo, mut hi) = (f32::INFINITY, 0f32);
|
||||
for row in y.saturating_sub(1)..(y + 2).min(h) {
|
||||
for col in x.saturating_sub(1)..(x + 2).min(w) {
|
||||
let l = image.data.get(row * w + col).map_or(0.0, |p| luminance_2020(*p));
|
||||
lo = lo.min(l);
|
||||
hi = hi.max(l);
|
||||
}
|
||||
}
|
||||
if hi.is_finite() { hi / lo.max(1e-4) } else { f32::INFINITY }
|
||||
}
|
||||
|
||||
/// Training pairs (sensor → JPEG, unclipped midtones) and the wider set including highlights
|
||||
/// (for the chroma curve); `None` when too few, or the photo has too little colour.
|
||||
/// (for the chroma curve); `None` when too few, or the photo has too little colour. With
|
||||
/// `edge_limit`, pixels whose 3×3 neighbourhood in either proxy spans a larger luminance ratio
|
||||
/// are left out: there a small geometric mismatch between the raw and its JPEG pairs unrelated
|
||||
/// colours (see [`EDGE_CONTRAST`]).
|
||||
type Pairs = Vec<([f64; 3], [f64; 3])>;
|
||||
fn collect_pairs(sensor: &Rgb32f, reference: &Rgb32f) -> Option<(Pairs, Pairs)> {
|
||||
fn collect_pairs(sensor: &Rgb32f, reference: &Rgb32f, edge_limit: Option<f32>) -> Option<(Pairs, Pairs)> {
|
||||
if (sensor.width, sensor.height) != (reference.width, reference.height) || sensor.data.len() != reference.data.len() {
|
||||
return None;
|
||||
}
|
||||
|
|
@ -161,7 +182,10 @@ fn collect_pairs(sensor: &Rgb32f, reference: &Rgb32f) -> Option<(Pairs, Pairs)>
|
|||
// Highlights too (camera JPEGs bleach colours toward white there), for the chroma curve only.
|
||||
let mut bright = Vec::new();
|
||||
let mut colour = 0;
|
||||
for (input, output) in sensor.data.iter().zip(&reference.data) {
|
||||
for (i, (input, output)) in sensor.data.iter().zip(&reference.data).enumerate() {
|
||||
if edge_limit.is_some_and(|limit| local_contrast(sensor, i) > limit || local_contrast(reference, i) > limit) {
|
||||
continue;
|
||||
}
|
||||
let y = luminance_2020(*output);
|
||||
if !input.iter().all(|v| v.is_finite() && *v > 0.001 && *v < 1.5) || !output.iter().all(|v| v.is_finite() && *v >= 0.0) {
|
||||
continue;
|
||||
|
|
@ -215,11 +239,25 @@ fn fit_matrix(pairs: &[([f64; 3], [f64; 3])]) -> Option<Mat3> {
|
|||
matrix.0.iter().flatten().all(|v| v.is_finite() && v.abs() < 8.0).then_some(matrix)
|
||||
}
|
||||
|
||||
/// Neighbourhood luminance ratio above which a pixel counts as an edge for the second attempt
|
||||
/// of [`fit_pairs_with`]. Camera JPEGs are often lens-corrected (Sony "Distortion Comp.: Auto",
|
||||
/// compacts and kit zooms): on a 51 mm ILCE-7RM2 shot of a glass façade the JPEG is up to 15 px
|
||||
/// of 1440 (about one proxy pixel) off the raw, and the fit failed the gate on the mismatched
|
||||
/// window frames alone (held-out RMS 0.111; 0.059 for the same shot with distortion correction
|
||||
/// off). Away from edges: 0.049. Issue #232.
|
||||
const EDGE_CONTRAST: f32 = 3.0;
|
||||
|
||||
/// The photo's look: its own matrix (with and without a hue/saturation table), or the given
|
||||
/// colour model (a camera profile's, in the sensor proxy's space), each completed with a tone
|
||||
/// and chroma curve fitted to this photo.
|
||||
/// and chroma curve fitted to this photo. When the fit on all pixels fails the acceptance gates,
|
||||
/// it is tried once more on the pixels away from edges (same gates): there colour pairs stay
|
||||
/// valid when the camera JPEG's geometry differs slightly from the raw's.
|
||||
fn fit_pairs_with(sensor: &Rgb32f, reference: &Rgb32f, colour: Option<(Mat3, Option<HsvTable>)>) -> Option<CameraLook> {
|
||||
let (pairs, bright) = collect_pairs(sensor, reference)?;
|
||||
fit_pairs_on(sensor, reference, colour.clone(), None).or_else(|| fit_pairs_on(sensor, reference, colour, Some(EDGE_CONTRAST)))
|
||||
}
|
||||
|
||||
fn fit_pairs_on(sensor: &Rgb32f, reference: &Rgb32f, colour: Option<(Mat3, Option<HsvTable>)>, edge_limit: Option<f32>) -> Option<CameraLook> {
|
||||
let (pairs, bright) = collect_pairs(sensor, reference, edge_limit)?;
|
||||
let candidates = match colour {
|
||||
Some(given) => vec![given],
|
||||
None => {
|
||||
|
|
@ -270,9 +308,10 @@ fn fit_pairs_with(sensor: &Rgb32f, reference: &Rgb32f, colour: Option<(Mat3, Opt
|
|||
.sum();
|
||||
if lightcraft_pipeline::profiling() {
|
||||
eprintln!(
|
||||
"[profile] camera look holdout RMS {:.5} -> {:.5} ({samples} channels)",
|
||||
"[profile] camera look holdout RMS {:.5} -> {:.5} ({samples} channels{})",
|
||||
(before / samples as f64).sqrt(),
|
||||
(after / samples as f64).sqrt()
|
||||
(after / samples as f64).sqrt(),
|
||||
if edge_limit.is_some() { ", away from edges" } else { "" }
|
||||
);
|
||||
}
|
||||
if samples == 0 || after >= before * MIN_IMPROVEMENT || after / samples as f64 > MAX_HOLDOUT_RMS.powi(2) {
|
||||
|
|
@ -740,7 +779,7 @@ mod tests {
|
|||
let mut pool = Vec::new();
|
||||
for seed in 0..4 {
|
||||
let (sensor, reference) = photo(seed, 2.0 + seed as f32 * 0.5);
|
||||
pool.extend(collect_pairs(&sensor, &reference).unwrap().0);
|
||||
pool.extend(collect_pairs(&sensor, &reference, None).unwrap().0);
|
||||
}
|
||||
let (matrix, _) = fit_profile(&pool).unwrap();
|
||||
// the camera's colours (luminance-normalised: the tone curve sets brightness)
|
||||
|
|
@ -818,6 +857,68 @@ mod tests {
|
|||
assert!(info.relative_wb && info.as_shot_temp == 6500.0 && info.as_shot_tint == 0.0);
|
||||
}
|
||||
|
||||
/// Issue #232 (skipped without the corpus): the public ILCE-7RM2 sample's camera JPEG is
|
||||
/// distortion-corrected, and on its glass façade the fit on all pixels misses the gate
|
||||
/// (held-out RMS 0.113): it used to open grey with the neutral fallback.
|
||||
#[test]
|
||||
fn corpus_arw_with_a_lens_corrected_preview_gets_a_camera_look() {
|
||||
let path = std::env::var_os("LIGHTCRAFT_CORPUS")
|
||||
.map(std::path::PathBuf::from)
|
||||
.unwrap_or_else(|| std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("../../corpus"))
|
||||
.join("raw/arw-sony-a7rm2-12bit-uncompressed.arw");
|
||||
let Ok(bytes) = std::fs::read(&path) else {
|
||||
eprintln!("skip: {} absent", path.display());
|
||||
return;
|
||||
};
|
||||
let (_, info) = crate::files::load_bytes(&bytes, 400).unwrap();
|
||||
assert!(info.camera_tone.is_some(), "no camera look fitted");
|
||||
}
|
||||
|
||||
/// Issue #232: an ILCE-7RM2 camera JPEG is lens-corrected ("Distortion Comp.: Auto"), so on a
|
||||
/// façade of window frames the raw and its JPEG are a pixel or two apart. The colours agree
|
||||
/// everywhere away from the edges, and the look must still be found there.
|
||||
#[test]
|
||||
fn fits_a_slightly_misaligned_camera_jpeg_away_from_edges() {
|
||||
let known = Mat3([[1.8, -0.4, -0.1], [-0.2, 1.5, -0.1], [-0.05, -0.3, 1.7]]);
|
||||
let (w, h, block, shift) = (96, 64, 8, 2);
|
||||
// Patches of 8×8 pixels, alternately dark and bright (frames and panes), varied in colour.
|
||||
let scene = |x: usize, y: usize| -> [f32; 3] {
|
||||
let (bx, by) = (x / block, y / block);
|
||||
let ev = if (bx + by) % 2 == 0 { 0.04 + (bx % 3) as f32 * 0.01 } else { 0.3 + (by % 4) as f32 * 0.08 };
|
||||
[ev * (0.7 + ((bx * 7 + by * 3) % 11) as f32 * 0.05), ev, ev * (0.7 + ((bx * 5 + by) % 13) as f32 * 0.04)]
|
||||
};
|
||||
let camera = |p: [f32; 3]| {
|
||||
let q = known.apply_f32(p);
|
||||
let y = luminance_2020(q);
|
||||
q.map(|v| v * (1.0 - (-2.5 * y).exp()) / y)
|
||||
};
|
||||
let mut sensor = Rgb32f::new(w, h);
|
||||
let mut reference = sensor.clone();
|
||||
for y in 0..h {
|
||||
for x in 0..w {
|
||||
sensor.data[y * w + x] = scene(x, y);
|
||||
// The JPEG shows the scene `shift` pixels to the right of where the raw has it.
|
||||
reference.data[y * w + x] = camera(scene(x.saturating_sub(shift), y));
|
||||
}
|
||||
}
|
||||
let fit = fit_pairs(&sensor, &reference).expect("look fitted away from the misaligned edges");
|
||||
// The look itself is right: on the aligned scene it reproduces the camera.
|
||||
let tone = ToneMap::camera(&fit.tone, 0.0, 0.0, 0.0);
|
||||
let (mut error, mut n) = (0.0, 0);
|
||||
for y in 0..h {
|
||||
for x in 0..w {
|
||||
let p = displayed(fit.matrix.apply(scene(x, y).map(f64::from)), &tone);
|
||||
let t = camera(scene(x, y));
|
||||
error += (0..3).map(|c| (p[c] - f64::from(t[c])).powi(2)).sum::<f64>();
|
||||
n += 3;
|
||||
}
|
||||
}
|
||||
assert!((error / n as f64).sqrt() < 0.03, "{}", (error / n as f64).sqrt());
|
||||
// Unrelated colours stay rejected away from edges too (covered below); a flat JPEG as well.
|
||||
reference.data.fill([0.2; 3]);
|
||||
assert!(fit_pairs(&sensor, &reference).is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_monochrome_invalid_and_unrelated_previews() {
|
||||
let mut sensor = Rgb32f::new(32, 32);
|
||||
|
|
|
|||
|
|
@ -62,7 +62,7 @@ impl SettingsHashes {
|
|||
}
|
||||
|
||||
/// Bump when the pipeline's output changes, to invalidate cached thumbnails.
|
||||
pub const RENDER_CACHE_VERSION: u64 = 11;
|
||||
pub const RENDER_CACHE_VERSION: u64 = 12;
|
||||
|
||||
/// Thumbnails render at one of these long edges (so window/cell size changes reuse the cache).
|
||||
pub const THUMB_SIZES: [usize; 4] = [128, 256, 384, 512];
|
||||
|
|
|
|||
|
|
@ -14,7 +14,7 @@ Matrix-only and matrix + table looks are each completed (tone curve, then chroma
|
|||
|
||||
The matrix is applied to scene-linear sensor pixels. The curve is carried in `SourceInfo` and applied in the pipeline finish stage, after exposure/WB/local edits. It is never baked into decoded luminance or clipped to JPEG precision. CPU and GPU use the same tone LUT. Contrast, whites and blacks adjust the starting look through the display-domain controls. JPEG neutral rendering and the existing DNG colour model are unchanged.
|
||||
|
||||
One third of the pixels are held out from both fits. The combined model must reduce linear-display squared error by at least 30% and have per-channel RMS error ≤0.10 (0.055 rejected half of the public raw.pixls.us samples tried — ILCE-6400, -6700, -7M4 — whose fits were still 1.5–3× closer to the camera JPEG than the fallback and visibly better). These are provisional acceptance gates, not a colour-accuracy certification. Monochrome, singular, unrelated, aspect-mismatched and poor-fitting references retain the documented fallback. Both fits use the same fixed proxy at every decode resolution.
|
||||
One third of the pixels are held out from both fits. The combined model must reduce linear-display squared error by at least 30% and have per-channel RMS error ≤0.10 (0.055 rejected half of the public raw.pixls.us samples tried — ILCE-6400, -6700, -7M4 — whose fits were still 1.5–3× closer to the camera JPEG than the fallback and visibly better). When the fit on all pixels fails these gates, it is tried once more, with the same gates, on the pixels away from edges (where the 3×3 neighbourhood in either proxy spans a luminance ratio above 3). Many camera JPEGs are lens-corrected while the raw is not (Sony "Distortion Comp.: Auto", compacts and kit zooms), and on a detailed scene the misaligned edges alone fail the gate although the colours agree everywhere else (issue #232). The public ILCE-7RM2 samples (raw.pixls.us, one façade shot in seven compression/correction variants, 24–70 mm at 51 mm) are up to 15 px of 1440 off their JPEG; the six with distortion correction on missed the gate at held-out RMS 0.108–0.115 and opened in the grey neutral fallback, 13 L* too dark (ΔE76 16.6–16.9 against the JPEG after registering its geometry). Away from edges they pass at 0.049–0.052 and land at ΔE 3.7–3.8 with lightness within 0.6 L*, like the variant shot with distortion correction off (0.059 on all pixels, ΔE 3.9). The corpus GX80 kit-zoom sample (up to 63 px of 1440 off) passes too: 0.107 → 0.081, ΔE 19.1 → 7.9. Fits accepted on all pixels are unchanged. These are provisional acceptance gates, not a colour-accuracy certification. Monochrome, singular, unrelated, aspect-mismatched and poor-fitting references retain the documented fallback. Both fits use the same fixed proxy at every decode resolution.
|
||||
|
||||
## Source metadata and white balance
|
||||
|
||||
|
|
|
|||
|
|
@ -276,7 +276,7 @@ under the UI command that wraps it (a no-op error); the UI command now wins.
|
|||
| LR-PROF-ADOBE | Standard raw looks (own equivalents) | P0 | ✅ | `crates/engine/src/presets.rs` (`PROFILES`), `crates/pipeline/src/profiles.rs` | six own looks: Color, Neutral, Vivid, Landscape, Portrait, Monochrome |
|
||||
| LR-PROF-ADAPTIVE | Adaptive profiles | P2 | ⬜ | | |
|
||||
| LR-PROF-CAMERA | Camera-matching looks | P2 | ⬜ | | |
|
||||
| LR-PROF-CAMERACOLOR | Camera colour calibration (own) | P0 | 🟡 | `crates/raw/src/color.rs`, `crates/raw/src/profile.rs`, `crates/engine/src/camera_preview.rs` | DNG files use the colour matrices and the camera-profile look they carry (`ProfileHueSatMap`, `ProfileLookTable`, `ProfileToneCurve`, read from the file at run time per DNG spec ch. 6 — Lightroom-converted DNGs rendered flat and muted without them, issue #138); Sony ARW, Nikon NEF and Panasonic RW2 get a guarded file-local fit to their own embedded JPEG — chromaticity matrix + hue/saturation/value table + tone and chroma curves, relative WB — or, with a local per-model profile pooled from many photos (`lightcraft-cli calibrate`, ARW and NEF so far, `crates/engine/src/camera_profiles.rs`), the profile's colour and only their own tone/chroma curves (on 29 held-out ILCE-7M4 photos mean ΔE vs the camera JPEG 3.54 → 3.10), kept with smart previews (see docs/camera-preview-colour.md); ARW: on 10 public raw.pixls.us samples from 8 bodies 7 fits were accepted, mean ΔE vs the camera JPEG 17–26 → 3–10; NEF: 13 of 13 decodable samples (D750, D780, D850, D7500, Z 50 and local D7500 shots) accepted, ΔE 13–46 → 3–8 (issue #150); RW2 / RWL: 140 of 174 public samples accepted, from 93 of the 114 bodies with a preview, median ΔE 5.0 with matching lightness (13 of the 14 corpus files with a preview: ΔE 9.6–27 → 2.8–8.9; most rejections are compacts and kit zooms whose camera JPEG is distortion-corrected); this is not measured camera calibration; other raws and rejected fits use a neutral fallback (camera RGB ≈ linear sRGB, `matrix_is_fallback`) with as-shot white balance, so colours are muted and not accurate. Needs our own per-camera calibration: matrices the files carry themselves (e.g. Olympus `ColorMatrix`), fitting each camera to its own embedded JPEG, then chart shots. Adobe matrices / DCPs are never used. Biggest image-quality gap today |
|
||||
| LR-PROF-CAMERACOLOR | Camera colour calibration (own) | P0 | 🟡 | `crates/raw/src/color.rs`, `crates/raw/src/profile.rs`, `crates/engine/src/camera_preview.rs` | DNG files use the colour matrices and the camera-profile look they carry (`ProfileHueSatMap`, `ProfileLookTable`, `ProfileToneCurve`, read from the file at run time per DNG spec ch. 6 — Lightroom-converted DNGs rendered flat and muted without them, issue #138); Sony ARW, Nikon NEF and Panasonic RW2 get a guarded file-local fit to their own embedded JPEG — chromaticity matrix + hue/saturation/value table + tone and chroma curves, relative WB — or, with a local per-model profile pooled from many photos (`lightcraft-cli calibrate`, ARW and NEF so far, `crates/engine/src/camera_profiles.rs`), the profile's colour and only their own tone/chroma curves (on 29 held-out ILCE-7M4 photos mean ΔE vs the camera JPEG 3.54 → 3.10), kept with smart previews (see docs/camera-preview-colour.md); ARW: on 10 public raw.pixls.us samples from 8 bodies 7 fits were accepted, mean ΔE vs the camera JPEG 17–26 → 3–10; NEF: 13 of 13 decodable samples (D750, D780, D850, D7500, Z 50 and local D7500 shots) accepted, ΔE 13–46 → 3–8 (issue #150); RW2 / RWL: 140 of 174 public samples accepted, from 93 of the 114 bodies with a preview, median ΔE 5.0 with matching lightness (13 of the 14 corpus files with a preview: ΔE 9.6–27 → 2.8–8.9; most rejections are compacts and kit zooms whose camera JPEG is distortion-corrected); a fit that fails the gates on all pixels is retried away from edges, where a lens-corrected camera JPEG still lines up (issue #232: the six distortion-corrected public ILCE-7RM2 samples ΔE 16.6–16.9 → 3.7–3.8, the corpus GX80 19.1 → 7.9); this is not measured camera calibration; other raws and rejected fits use a neutral fallback (camera RGB ≈ linear sRGB, `matrix_is_fallback`) with as-shot white balance, so colours are muted and not accurate. Needs our own per-camera calibration: matrices the files carry themselves (e.g. Olympus `ColorMatrix`), fitting each camera to its own embedded JPEG, then chart shots. Adobe matrices / DCPs are never used. Biggest image-quality gap today |
|
||||
| LR-PROF-CREATIVE | Creative profiles (own) | P2 | ✅ | `cmd:develop.profile`, `crates/pipeline/src/profiles.rs` | 16 own looks in Film / Cinematic / Muted / B&W (tone + point-curve fades, colour grading, mixer / B&W mix); scale with `ctl:profile.amount`; sliders untouched |
|
||||
| LR-PROF-LEGACY | Legacy profiles | P2 | ⬜ | | |
|
||||
| LR-PROF-NONRAW | Profiles for non-raw files | P0 | ✅ | `cmd:develop.profile` | same looks apply to JPEG/TIFF |
|
||||
|
|
|
|||
|
|
@ -3,6 +3,8 @@
|
|||
## October 2026
|
||||
|
||||
### RAW decoding
|
||||
- Sony A7R II (and other) raws whose camera JPEG is lens-corrected no longer open grey and too dark (issue #232): the
|
||||
starting look is fitted to the camera JPEG away from edges when the misaligned edges spoil the fit on all pixels.
|
||||
- Sony ILCE-7M4 downsized lossless ARWs now decode subsampled YCbCr tiles into linear RGB,
|
||||
preserving RAW editing & full-resolution export instead of using embedded JPEG previews.
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue