lightcraft/crates
Brandon Thomas ae1cbbe87b Issue #37: Catalog: stream the snapshot into the temp file instead of building two strings
Journal::snapshot serialised the catalog into a String and then copied it into a second
`format!`-ed String before writing (2x the snapshot size in memory, ~620 MB at 85k photos).
It now streams `{"format":…,"seq":N,"catalog":` + serde_json::to_writer + `}\n` through a
1 MiB BufWriter into the store (new `Store::write_atomic_with`; FsStore writes the temp file
directly, then flush + sync_all + rename + dir fsync as before; other stores buffer, as before).

Durability and format are unchanged: same temp + fsync + atomic rename; a failed write leaves
the old snapshot (and now removes the temp file); the bytes are identical to the old format
(test `streamed_snapshot_keeps_the_on_disk_format`, which also loads a legacy snapshot), so
no version bump. Also tested: `failed_streamed_write_keeps_the_old_file`.

persist_bench, 85k photos (310 MB snapshot), compaction at the 2,000-record threshold,
before/after binaries alternated in the same session (load avg ~130-170), building+compacting
only (PERSIST_BENCH_NO_LOAD):
  compaction median: before 3.5 s / 4.6 s, after 1.9 s / 2.5 s
  peak RSS:          before 926 MB / 1263 MB, after 308 MB / 366 MB
  10k photos peak RSS: 149-235 MB -> 61-68 MB (times too noisy to compare)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 15:43:13 -04:00
..
catalog Issue #37: Catalog: stream the snapshot into the temp file instead of building two strings 2026-10-04 15:43:13 -04:00
codecs Merge pull request #54 from storytold/pr/never-crash-codecs-catalog 2026-10-04 14:23:05 -04:00
color Never crash: color, geom, raster, develop, scenes, preview, pipeline 2026-10-04 12:34:53 -04:00
develop Never crash: color, geom, raster, develop, scenes, preview, pipeline 2026-10-04 12:34:53 -04:00
engine Issue #37: Catalog: instrument persistence (append, snapshot stages) and whole-frame update time 2026-10-04 15:43:13 -04:00
geom Never crash: color, geom, raster, develop, scenes, preview, pipeline 2026-10-04 12:34:53 -04:00
gpu Never crash: gpu and merge free of panicking calls 2026-10-04 12:34:53 -04:00
mcp Merge pull request #57 from storytold/pr/never-crash-engine-mcp 2026-10-04 14:23:58 -04:00
merge Never crash: gpu and merge free of panicking calls 2026-10-04 12:34:53 -04:00
meta Merge main into feat/gpx-tracklog 2026-10-04 15:31:45 -04:00
pipeline Merge pull request #56 from storytold/pr/never-crash-core 2026-10-04 14:23:22 -04:00
preview Merge pull request #56 from storytold/pr/never-crash-core 2026-10-04 14:23:22 -04:00
raster Merge pull request #56 from storytold/pr/never-crash-core 2026-10-04 14:23:22 -04:00
raw Merge pull request #53 from storytold/pr/never-crash-raw-tiff 2026-10-04 14:22:47 -04:00
scenes Never crash: color, geom, raster, develop, scenes, preview, pipeline 2026-10-04 12:34:53 -04:00
tiff Merge pull request #53 from storytold/pr/never-crash-raw-tiff 2026-10-04 14:22:47 -04:00
ui-egui Issue #37: Catalog: instrument persistence (append, snapshot stages) and whole-frame update time 2026-10-04 15:43:13 -04:00