shifathosn
/
pdfcraft
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Reduce large-PDF memory growth (#307) (#419) 

* M14: bound large-PDF memory use (#307)

Inspect large documents lazily, share parser/resource metadata, bound full-save
and raster retention, and cache signature candidate discovery safely.

Preserve upstream rendering deduplication, pixel alignment, signature previews,
and organizer insertion behavior, with regression coverage for their interaction.

Validated with 1,336 workspace tests, 193 parser tests, strict workspace Clippy,
30 WASM crate checks, 1x/2x GUI protocols, and same-PDF workflow comparisons.

* cos: drop the writer cache-policy probe and its test-only hooks in production paths

* cos: the temporary reader keeps parsed objects until a count or owned-bytes budget

Since #552 a stream read from the file is a view into the input, so clearing the whole parse
cache after every read saved little memory and cost most of the full-save slowdown (every
object parsed twice, indirect /Length objects re-parsed per stream). Clear only past
OBJECTS entries or STREAM_BYTES of data of the cache's own (decrypted streams, strings).

* cos: the writers refuse a document opened with a stream limit (read-only inspection)

* cos: a full save fails rather than writing null when an object reads differently the second time

* render: parsing is not timed by the watchdog; timing tests with wide margins

A slow first parse (a huge or damaged file) made the whole pool fail every page for its
lifetime. Parsing, shared or private, is now its own untimed stage, as on main before the
shared parser; rendering is timed from when parsing ends. The init-timeout tests become one
deterministic test (a 2 s parse against a 10 ms limit, shared and private). Watchdog tests
use limits and injected delays ~10x apart instead of 25-50 ms against real renders. Also:
saturating retired-worker count, main's stop-flag test runs on private parsers (the first
shared timeout is a private retry), set_queue documents that unlisted results are dropped,
and retire() documents the permanent per-worker-parser fallback.

* render: page numbers in the lazy page tree use checked conversion

* render: the lazy inspector is compared with lopdf on named destinations and object streams too

* ui: on-screen pages always render; pages turned back to stay cached; stable thumbnail tags

- Visible pages and tiles are always admitted and uploaded (the viewport bounds them); the
  byte and count limits decide only how much prefetch fits. A HiDPI two-up view or a 5K
  viewport left pages on 'Rendering...' for good.
- Pages no longer demanded stay cached, nearest the current page first, while the 128 MiB
  allowance and the 24-page count have room: paging back no longer re-renders.
- Thumbnail density follows the demand count rounded up to a power of two, so rows scrolling
  in and out of an organize grid don't change every thumbnail's tag and re-render it.
- A suspended (background) tab drains its pool's finished results instead of holding them.
- Organize drag-and-drop ends on the primary button only.
- Tests: the stale-result test uses valid pixels and checks nothing was uploaded; the
  organizer wheel test runs 40 steps; regressions for the first three fixes.

* vendor: mark the hayro-syntax lock and SegmentList patches and list them as (8) and (9); clippy 1.99 as_chunks in a raster test

* render: clippy 1.99 — contains over iter().any() in a raster test

---------

Co-authored-by: Brandon Thomas <echelon@gmail.com>
This commit is contained in:
skylar taylor-barrick 2026-10-09 12:38:01 -06:00 committed by GitHub
parent 75b31fb04d
commit dbd0748177
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
29 changed files with 4357 additions and 474 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

24
ROADMAP.md
View File

@ -42,7 +42,7 @@ Read this before choosing work. The feature table above counts what exists; this
|---|---|---|
| Feature count | 51.0% shipped (P0 88%, P1 53%, P2 8%, P3 2%) | A typical viewer, annotator, form-filler or page organizer is mostly covered |
| Effort | ≈ 30–35% | The remaining work is the hardest: M2 15%, M7 17%, M11 18%, M12 23%, M14 0% |
| Foundations | Weakest | Rendering is still the bootstrap `hayro`; the inspector is `lopdf`; 18 vendored patches carried |
| Foundations | Weakest | Rendering is still the bootstrap `hayro`; the local #307 candidate reads inspector data lazily through COS with `lopdf` compatibility logic; vendored patches are still carried |
| Robustness | Early beta | Each 15-minute fuzz run found new out-of-memory crashes or hangs until 2026-10-05 |
| Acrobat fidelity | Unmeasured | No systematic side-by-side comparison with Acrobat, visual or behavioural |
| Product readiness | 0.2.1 | No code signing, performance budgets or keyboard-only operation yet; localization has its catalog system (English, partial Japanese) but few strings go through it |
@ -106,8 +106,7 @@ Hours are for a single agent (low–high). "Done" is the estimated fraction of t
| M11 | Optimize, preflight, PDF/A/X/UA, print production | 200–350 | 18% | 165–290 | Done: new `optimize` crate: Reduce File Size and the PDF Optimizer (images measured where drawn, bicubic downsampling, JPEG/ZIP recompression only when smaller, discard objects and user data, Flate clean-up, resource merging, object streams). Missing: fonts and transparency panels, space audit, preflight, PDF/A/X/UA |
| M12 | Accessibility, compare, measure, search, XFA | 200–380 | 23% | 155–295 | Done: new `a11y` crate with the Accessibility Checker (all 32 rules, report, Fix/Skip/Explain, options dialog and results panel, agent tools). Done since: 2D distance, perimeter and area measurements with persistent viewport calibration, snapping, live information and CSV export. Missing: autotag, Tags/Order/Content panels, Reading Order tool, alt-text workflow, compare, geospatial/3D measurement, search index, XFA |
| M13 | Automation (MCP, Action Wizard, CLI) + AI providers | 60–120 | 45% | 33–66 | Done: MCP resources (document info, text, page images); headless tool table (123 tools incl. signing, optimizing, initial view, links, stamps, data exchange, comment review, forms authoring and scripts, redaction, sanitize, print, add content), opt-in MCP server over stdio, CLI `run`/`tools` (closed stdout pipes exit cleanly), UI control channel with drag. Missing: Action Wizard, AI providers |
| M14 | 1.0 polish: performance, localization, installers | 120–250 | 5% | 115–240 | Done: PhotoCraft's translation system (`i18n/`: TSV catalogs, `tl!`, command-id and plural entries, system-language detection, strict catalog tests); every dialog, panel and notice goes through `tl!`; Japanese, Traditional Chinese, Simplified Chinese, Russian, German, Spanish and French catalogs have ≈1,800 entries each; Simplified Chinese, French and German coverage is checked against the UI source; Czech and Brazilian Portuguese for menus. French locales (`fr`, `fr_FR`, `fr_CA`) follow the French catalog. Missing: more catalogs filled in, installers, performance budgets |
| M14 | 1.0 polish: performance, localization, installers | 120–250 | 5% | 115–240 | Done: PhotoCraft's translation system (`i18n/`: TSV catalogs, `tl!`, command-id and plural entries, system-language detection, strict catalog tests); every dialog, panel and notice goes through `tl!`; Japanese, Traditional Chinese, Simplified Chinese, Russian, Spanish, French and Ukrainian catalogs have ≈1,800–2,000 entries each; Simplified Chinese, French and Ukrainian coverage is checked against the UI source; Czech and Brazilian Portuguese for menus. French locales (`fr`, `fr_FR`, `fr_CA`) follow the French catalog; Ukrainian locales (`uk`, `uk_UA`) follow the Ukrainian one. Missing: more catalogs filled in, installers, performance budgets |
| M14 | 1.0 polish: performance, localization, installers | 120–250 | 5% | 115–240 | Done: PhotoCraft's translation system (`i18n/`: TSV catalogs, `tl!`, command-id and plural entries, system-language detection, strict catalog tests); every dialog, panel and notice goes through `tl!`; Japanese, Traditional Chinese, Simplified Chinese, Russian, Bulgarian, German, Spanish, French, Hungarian and Ukrainian catalogs have ≈1,800–2,000 entries each; Simplified Chinese, French, German, Hungarian and Ukrainian coverage is checked against the UI source; Czech and Brazilian Portuguese for menus. French locales (`fr`, `fr_FR`, `fr_CA`) follow the French catalog; Ukrainian locales (`uk`, `uk_UA`) follow the Ukrainian one. #307 (bounded large-PDF memory): share resource indexes and native parser metadata, inspect structure lazily, bound scan/full-save caches and raster storage, and reuse signature discovery safely. Three Linux trials retain full-save workflow RSS at 1.06 GiB (previously 3.69 GiB); rebased engine navigation uses 821 MiB (previously 997 MiB). With the RSS guard removed, the unmodified baseline was kernel OOM-killed in a 15 GiB container; rebased optimized inspection completed at 497.7 MiB in 1.78 s. Strict Clippy, 1,336 workspace tests and 30 WASM crate checks pass. The larger-host follow-up is closed. File-backed input, global memory budgets and Windows validation remain pending. Missing: more catalogs filled in, installers, performance budgets |
| | **Total (original plan sizing)** | **2,085–3,840** | **≈ 35%** | **≈ 1,350–2,500 at the planned rate; ≈ 600–1,100 at the measured rate** | |
**Overall progress: about 30–35% of the effort (51.0% of features shipped).** The viewer and the core are far ahead of the editing features, because the viewer was built first so progress could be seen; rendering itself is still borrowed from `hayro`. See [Honest assessment](#honest-assessment-2026-10-05).
@ -135,8 +134,6 @@ M0 → M1 → M2 → M3 → M4 must happen in order. After M4, M5–M12 can run
- **2026-10-09 (M14, #577):** The desktop GPU setup enables the selected adapter's supported 2D texture extent instead of egui-wgpu's fixed 8192-pixel extent. This removes the artificial surface ceiling behind a mixed-DPI Windows startup crash when opening Outlook PDF attachments. Synthetic regressions cover the observed 8600 × 3423 surface, downlevel adapters and preservation of other device limits. Hardware resolution limits still apply; milestone and overall effort estimates are unchanged.
- **2026-10-09 (M5, Fill & Sign flatten on save):** Preferences and `doc_save` can bake Fill & Sign text, marks and signatures into the page when saving. The option defaults off. Other comments and form fields stay, a second flatten does not reuse an XObject name already on the page, and a document that cannot be changed is left unwritten. Overall estimate unchanged (about 30–35%).
- **2026-10-09 (M5, Fill & Sign flatten on save):** Preferences and `doc_save` can bake Fill & Sign text, marks and signatures into the page when saving. The option defaults off. Other comments and form fields stay, a second flatten does not reuse an XObject name already on the page, and a document that cannot be changed is left unwritten. Overall estimate unchanged (about 30–35%).
- **2026-10-09 (M5, Fill & Sign flatten on save):** Preferences and `doc_save` can bake Fill & Sign text, marks and signatures into the page when saving. The option defaults off. Other comments and form fields stay, a second flatten does not reuse an XObject name already on the page, and a document that cannot be changed is left unwritten. Overall estimate unchanged (about 30–35%).
- **2026-10-09 (M5.7, #434 partial):** Natural-size image stamps placed at a click or through `stamp_custom` remain upright on rotated pages, with a displayed-space-sized rectangle and a counterrotated standard appearance matrix. Move, resize, restyle and undo/redo preserve that orientation. Native and external Poppler save/reopen rendering cover all four quarter-turns; UI tests verify the displayed bounds. Explicit rectangle placement, image signatures, PDF-page stamps and image-file orientation metadata remain outside this repair. #434 stays open; milestone table and effort estimates remain unchanged.
- **2026-10-09 (M1, raw-stream decoding limits):** COS checks unfiltered decoded data against the caller and per-stream limits before copying it, including empty filter arrays and streams whose Crypt filter was applied on load. Strict decoding uses the same raw-data guard. Tiny synthetic regressions cover exceeded, exact and zero-byte limits while retaining the filtered decoder behavior. Milestone and overall effort estimates unchanged (about 30–35%).
@ -145,11 +142,7 @@ M0 → M1 → M2 → M3 → M4 must happen in order. After M4, M5–M12 can run
- **2026-10-09 (M4, page grid zoom):** The Organize page grid zooms from 50% to 300%: toolbar buttons with a percentage that resets, trackpad pinch or Ctrl/⌘ + wheel about the page under the pointer, and ⌘+ / ⌘− / ⌘0. Pages drawn larger than their thumbnails are rendered at the size drawn while in view, so text stays sharp in documents of any length. Not yet: a slider, remembering the size, a size control for the Pages panel. Overall estimate unchanged (about 30–35%).
- **2026-10-09 (M4, page grid zoom):** The Organize page grid zooms from 50% to 300%: toolbar buttons with a percentage that resets, trackpad pinch or Ctrl/⌘ + wheel about the page under the pointer, and ⌘+ / ⌘− / ⌘0. Pages drawn larger than their thumbnails are rendered at the size drawn while in view, so text stays sharp in documents of any length. Not yet: a slider, remembering the size, a size control for the Pages panel. Overall estimate unchanged (about 30–35%).
- **2026-10-09 (M4, page grid zoom):** The Organize page grid zooms from 50% to 300%: toolbar buttons with a percentage that resets, trackpad pinch or Ctrl/⌘ + wheel about the page under the pointer, and ⌘+ / ⌘− / ⌘0. Pages drawn larger than their thumbnails are rendered at the size drawn while in view, so text stays sharp in documents of any length. Not yet: a slider, remembering the size, a size control for the Pages panel. Overall estimate unchanged (about 30–35%).
- **2026-10-09 (M4, page grid zoom):** The Organize page grid zooms from 50% to 300%: toolbar buttons with a percentage that resets, trackpad pinch or Ctrl/⌘ + wheel about the page under the pointer, and ⌘+ / ⌘− / ⌘0. Pages drawn larger than their thumbnails are rendered at the size drawn while in view, so text stays sharp in documents of any length. Not yet: a slider, remembering the size, a size control for the Pages panel. Overall estimate unchanged (about 30–35%).
- **2026-10-08 (M14, #307, upstream integration):** Rebased the local candidate onto main `a6bc63a`, preserving render-request deduplication, pixel-aligned rasters, signature editing and organizer insertion controls. Regressions cover inactive signature-preview cleanup, insertion gaps under virtualization, and canceled/reintroduced render requests. All 1,336 workspace tests pass (6 ignored), 193 parser tests pass (2 missing-fixture tests filtered), strict workspace/all-target Clippy passes without allowances, and 30 WASM crate checks pass. All 1,010 navigation frames, 12 workflow artifacts and eight 100-page external PDF checks match; 1×/2× desktop protocols pass again. Inspection is 497.7 MiB / 1.78 s, full-save 1.06 GiB / 29.64 s, and fresh desktop navigation/close/reopen 877 MiB. The desktop result is higher than the pre-rebase 832 MiB, while remaining 21.9% below the earlier bounded-writer stage. Windows, cold-start latency and global memory budgets remain open; no milestone percentage increase.
- **2026-10-08 (M5.7, signature resize proportions):** Image signature corner handles preserve the embedded image's original aspect ratio and anchor the opposite corner; edge midpoint handles stretch width or height independently. The live preview and committed edit share the constrained geometry, including page/view rotation and reopened PDFs whose rectangles were stretched previously. Synthetic UI tests cover all corners, edge resizing, undo and redo. Milestone and effort estimates unchanged.
- **2026-10-08 (M5.7, live signature adjustment):** Embedded image signatures resize and move with every pointer frame. Movement hides the frame and handles, which return on release; Escape cancels without committing. A cached background preserves the page beneath the image, and only the completed gesture enters undo history. The read-only engine preview and `comment_image_preview` tool share bounded image decoding and render layers. Synthetic pixel tests cover live geometry, alpha, coloured page content, rotation, cancellation and encrypted save/reopen. Milestone and effort estimates unchanged.
@ -157,6 +150,17 @@ M0 → M1 → M2 → M3 → M4 must happen in order. After M4, M5–M12 can run
- **2026-10-08 (M5.7, signature placement):** Image signatures and initials preview at the pointer at their actual page size, including zoom and document/view rotation. A successful placement switches to Select with the new annotation selected; stamp resize handles use the existing engine edit, automation tool and undo history while preserving the appearance. Synthetic rendered UI and automation tests verify positioning, one placement per selection, resizing, undo/redo and save/reopen. Milestone and overall effort estimates unchanged.
- **2026-10-08 (M5.7, image signatures):** Fill & Sign imports local PNG/JPEG signatures and initials through the Image tab, with previews, local persistence and undoable placement. The shared engine validates bounded images and preserves transparency and orientation; `fill_sign_add` accepts an image path. Synthetic UI and automation tests cover cancellation, invalid input, settings, undo/redo and save/reopen rendering. Overall effort estimate unchanged (about 30–35%).
- **2026-10-08 (M14, #307, confirmed baseline OOM):** Removed the benchmark RSS guard and ran the frozen original and optimized CLI binaries in separate containers with a verified 15 GiB kernel memory limit, no swap and four CPUs. The unmodified baseline was kernel OOM-killed after 20.06 s with 14.97 GiB observed RSS; Docker recorded `OOMKilled=true`, exit 137 and an `oom` event. The optimized build completed the same 9,156-page inspection at 497.5 MiB in 1.75 s with unchanged metadata and no warnings. The original RSS remains a lower bound and elapsed time is time to failure. The kernel limit still constrains this experiment; no remote run was needed. No product changes or milestone percentage increase.
- **2026-10-08 (M14, #307, original-baseline cutoff):** Repeated original unmodified CLI inspection with a 14 GiB RSS guard, up from 10 GiB, on the unchanged 9,156-page PDF. It reached 14,682,080 KiB and was stopped after 48.21 s before completion; no cgroup OOM kill occurred. The optimized build completed the same inspection at 495.7 MiB in 1.70 s. The original measurement is a lower bound, not a completed peak or timing. This guarded run failed before completion without an OS OOM kill. The user closed the larger-host follow-up; the prepared benchmark is retained as evidence. The later local guard-free experiment is recorded above. No product changes or milestone percentage increase.
- **2026-10-08 (M14, #307, shared parsing and raster budgets):** Native render workers share a lazily initialized parser after cold-load/repair concurrency and watchdog recovery fixes, while retaining private render caches. Pages, organizer and Print request current visible thumbnails; retained active-view textures are limited to 24/128/128 MiB for thumbnails/pages/tiles and completed raster storage to 32 MiB/32 results, allowing one oversized result alone. Signature discovery uses a bounded source/edit-aware cache; the writer borrows unchanged objects and reuses xref rows. Engine navigation drops from 997 to 807 MiB with all 1,010 frame hashes unchanged. Three-trial full/incremental-save workflow times improve to 29.92/16.35 s at unchanged 1.06/1.08 GiB peaks; full serialization remains slower than before bounded saving, and the single mixed workflow takes 67.99 s versus 65.52 s. All 1,158 workspace tests pass (6 ignored in the regular run), 193 vendor tests pass (2 missing-fixture tests filtered), eight outputs pass 100-page external checks, and 29 WASM crate checks pass. Existing strict workspace Clippy warnings remain; UI lint validation explicitly allows the existing collapsible-match category. Fresh desktop navigation/close/reopen falls from 1,123 to 832 MiB; 1x/2x GUI runs pass visible-demand and texture-budget checks on original 10,000-page fixtures and the real file. Output-capacity and LRU experiments did not justify production changes. These are local Linux results and specific storage budgets, not global RAM limits or Windows validation. No parity or milestone percentage increase.
- **2026-10-08 (M14, #307, bounded saving):** Full rewriting now uses private temporary caches and packs objects in batches limited by count and serialized bytes, preserving unknown objects, reference identity and encryption behavior without populating undo snapshots. Native render workers initialize only on a live request, avoiding unused parser copies during refresh/rebase. Three same-file Linux trials reduce rotate/full-save/close/reopen peak RSS from 3.69 to 1.06 GiB (35.53 to 32.51 s); incremental workflow RSS falls from 1.56 to 1.08 GiB. All 1,114 workspace tests pass (4 ignored); eight saved outputs pass external text/raster checks on 100 sampled pages each, and 1,010 navigation frames match. First use of each worker now pays parser initialization; steady-state viewing remains about 1 GiB. Single oversized objects, whole source/output buffers, and Windows behavior remain outside a global budget claim. No parity or milestone percentage increase.
- **2026-10-08 (M14, #307, research/comparison):** Added a local external-reference suite outside the repository, using the same 9,156-page PDF with MuPDF, PDFium, Poppler and qpdf. Three Linux trials put the candidate's single-renderer 102-frame peak at 392 MiB, versus 348/347 MiB for MuPDF/PDFium with whole-file bytes (164/163 MiB with file input). The rotate/full-save/close/reopen workflow peaks at 3.69 GiB during serialization; incremental saving peaks at 1.56 GiB during rebasing. Different UI features, input/cache policies and writer compression settings prevent treating these as app rankings. Next targets are bounded object-stream packing, parsed-object lifetimes, snapshot overlap and file-backed input. No further runtime optimization or milestone percentage increase in this research step.
- **2026-10-08 (M14, #307, local candidate):** Validated the 192.9 MB, 9,156-page VMware document in Linux. The original CLI exceeded a 10 GiB measurement cutoff before finishing inspection; the candidate inspects in about 1.8 s at 496 MiB and a 1,000-page navigation/close/reopen probe peaks at 990 MiB. It shares validated large resource indexes within each parser, reads panel structure on demand through COS with the previous inspector as a compatibility fallback, and scans standalone timestamps with temporary caches instead of retaining every object in the editor. Inspection output and 104 common sampled page renders match the handoff candidate. This is a different PDF revision from the original Windows issue; Windows GUI private memory and responsiveness remain unverified. No parity or milestone percentage increase.
- **2026-10-08 (M9, #179):** Windows Current User Personal certificate store identities join file-based digital IDs, signing through CNG without exporting keys. RSA-2048 and ECDSA P-256 round trips use temporary certificates with cleanup guards; enumeration, missing identities, automation errors and the password-free UI have regression coverage. Smart cards and PKCS #11 remain planned; overall effort estimate unchanged.

5
crates/cos/src/bytes.rs
View File

@ -23,6 +23,11 @@ impl Bytes {
self.range.len() < self.buf.len() && !Arc::ptr_eq(&self.buf, owner)
}
/// Whether these bytes are (a view into) `buf`, holding no memory of their own.
pub(crate) fn shares(&self, buf: &Arc<Vec<u8>>) -> bool {
Arc::ptr_eq(&self.buf, buf)
}
/// The same bytes, in a buffer of their own.
pub(crate) fn detached(&self) -> Self {
Self::from(self.to_vec())

342
crates/cos/src/document.rs
View File

@ -6,7 +6,7 @@
//! the file (full save).
use std::collections::{BTreeMap, HashMap, HashSet};
use std::sync::{Arc, Mutex};
use std::sync::{Arc, Mutex, Weak};
use crate::CosError;
use crate::object::{Dict, ObjRef, Object};
@ -84,15 +84,31 @@ enum Slot {
Freed(u16),
}
/// An immutable input/decode context shared by document clones. Holding this identity
/// retains neither the input bytes nor the parsed document. Weak ownership prevents an
/// expired identity from matching a later allocation at the same address.
#[derive(Clone, Debug)]
pub struct SourceIdentity(Weak<()>);
impl PartialEq for SourceIdentity {
fn eq(&self, other: &Self) -> bool {
self.0.ptr_eq(&other.0)
}
}
impl Eq for SourceIdentity {}
#[derive(Clone)]
pub struct Document {
data: Arc<Vec<u8>>,
source_identity: Arc<()>,
entries: Arc<BTreeMap<u32, XrefEntry>>,
trailer: Dict,
revisions: Arc<Vec<Revision>>,
repair_log: Arc<Vec<String>>,
cache: Arc<Mutex<HashMap<u32, Arc<Object>>>>,
objstms: Arc<Mutex<HashMap<u32, Arc<ObjStm>>>>,
stream_limit: usize,
overlay: BTreeMap<u32, Slot>,
next_num: u32,
/// Position of the `%PDF-` header (some files have junk before it).
@ -111,6 +127,80 @@ pub struct Document {
out_encrypt_num: Option<u32>,
}
/// A temporary reader for whole-document work. Its caches never populate the editor or
/// undo snapshots. Returned objects survive cache eviction; a stream read from the file is a
/// view into the input bytes (it keeps the whole input alive while held, not a copy).
pub(crate) struct ObjectReader {
pub(crate) document: Document,
decoded_count: usize,
}
impl ObjectReader {
const OBJECTS: usize = 128;
const STREAM_BYTES: usize = 8 * 1024 * 1024;
pub(crate) fn get(&mut self, reference: ObjRef) -> Arc<Object> {
self.try_get(reference).unwrap_or_else(|_| Arc::new(Object::Null))
}
pub(crate) fn try_get(&mut self, reference: ObjRef) -> Result<Arc<Object>, CosError> {
let object = self.document.try_get(reference.num);
if let Ok(mut streams) = self.document.objstms.lock() {
// Only insertion changes this private cache between reads. Most neighboring
// objects reuse a decoded stream: do not sum the whole store for every object.
if streams.len() != self.decoded_count {
let bytes = streams.values().fold(0usize, |bytes, stream| {
bytes
.saturating_add(stream.data.capacity())
.saturating_add(stream.index.capacity().saturating_mul(std::mem::size_of::<(u32, usize)>()))
});
if bytes > Self::STREAM_BYTES || streams.len() >= Self::OBJECTS {
// Retain one oversized active stream so its next object does not require
// decoding it again. That stream still obeys the document's decode cap.
let current = match self.document.xref_entry(reference.num) {
Some(XrefEntry::InStream { stream, .. }) => Some(stream),
_ => None,
};
streams.retain(|num, _| Some(*num) == current);
}
self.decoded_count = streams.len();
}
}
// Parsed objects stay cached (an indirect /Length or a dictionary read again costs a
// lookup, not a parse) until the cache passes its object count or holds more than
// STREAM_BYTES of data of its own. Streams read from the file are views into the
// input bytes, which the document keeps anyway, so they don't count; decrypted
// streams and strings do. Checked after failed loads too: resolving a /Length may
// have cached large objects.
if let Ok(mut cache) = self.document.cache.lock() {
let data = &self.document.data;
if cache.len() >= Self::OBJECTS || cache.values().fold(0usize, |n, o| n.saturating_add(owned_bytes(o, data, 0))) > Self::STREAM_BYTES {
cache.clear();
}
}
object
}
}
/// Bytes `object` holds of its own: string and stream data that is not a view into `data`
/// (the input file), nested a few levels deep (deeper data is rare and still bounded by the
/// reader's object count).
fn owned_bytes(object: &Object, data: &Arc<Vec<u8>>, depth: usize) -> usize {
if depth > 4 {
return 0;
}
match object {
Object::String(s) => s.bytes.len(),
Object::Stream(s) => {
let own = if s.raw.shares(data) { 0 } else { s.raw.len() };
s.dict.iter().fold(own, |n, (_, v)| n.saturating_add(owned_bytes(v, data, depth + 1)))
}
Object::Array(a) => a.iter().fold(0usize, |n, v| n.saturating_add(owned_bytes(v, data, depth + 1))),
Object::Dict(d) => d.iter().fold(0usize, |n, (_, v)| n.saturating_add(owned_bytes(v, data, depth + 1))),
_ => 0,
}
}
impl std::fmt::Debug for Document {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("Document").field("bytes", &self.data.len()).field("objects", &self.entries.len()).field("edits", &self.overlay.len()).finish()
@ -123,12 +213,14 @@ impl Document {
pub fn new_empty() -> Self {
let mut doc = Document {
data: Arc::new(Vec::new()),
source_identity: Arc::new(()),
entries: Arc::new(BTreeMap::new()),
trailer: Dict::new(),
revisions: Arc::new(Vec::new()),
repair_log: Arc::new(Vec::new()),
cache: Arc::default(),
objstms: Arc::default(),
stream_limit: crate::object::MAX_DECODED,
overlay: BTreeMap::new(),
next_num: 1,
header_offset: 0,
@ -162,6 +254,14 @@ impl Document {
/// Parse a document, authenticating encrypted ones with `password` (owner or user; `None`
/// tries the empty password). Fails with `NeedsPassword` / `WrongPassword` as appropriate.
pub fn open_with_password(data: Arc<Vec<u8>>, password: Option<&str>) -> Result<Self, CosError> {
Self::open_with_stream_limit(data, password, crate::object::MAX_DECODED)
}
/// Open with a tighter decompression bound for object and cross-reference streams.
/// Inspection can use this without changing the editing reader's default limit. Such a
/// document is read-only: objects in a larger object stream read as missing, so the
/// writers refuse to save it ([`CosError::ReadOnlyLimit`]).
pub fn open_with_stream_limit(data: Arc<Vec<u8>>, password: Option<&str>, stream_limit: usize) -> Result<Self, CosError> {
// Viewers accept files whose header is missing or damaged as long as the body looks
// like PDF; so do we (a note goes to the repair log).
let header = find(&data, b"%PDF-", 0, 1024);
@ -176,12 +276,14 @@ impl Document {
};
let mut doc = Document {
data,
source_identity: Arc::new(()),
entries: Arc::new(BTreeMap::new()),
trailer: Dict::new(),
revisions: Arc::new(Vec::new()),
repair_log: Arc::new(Vec::new()),
cache: Arc::default(),
objstms: Arc::default(),
stream_limit: stream_limit.min(crate::object::MAX_DECODED),
overlay: BTreeMap::new(),
next_num: 1,
header_offset,
@ -237,6 +339,8 @@ impl Document {
}
fn authenticate(&mut self, enc: &Object, password: Option<&str>) -> Result<(), CosError> {
// Authentication can change how the same input is decoded.
self.source_identity = Arc::new(());
let (num, dict) = match enc {
Object::Ref(r) => (Some(r.num), self.get(*r).as_dict().cloned()),
Object::Dict(d) => (None, Some(d.clone())),
@ -388,6 +492,12 @@ impl Document {
&self.data
}
/// Identity of the original bytes, xref and authenticated decode context. Edits and
/// clones preserve it; reopening (including after any save) produces a new identity.
pub fn source_identity(&self) -> SourceIdentity {
SourceIdentity(Arc::downgrade(&self.source_identity))
}
pub fn version(&self) -> &str {
&self.version
}
@ -493,6 +603,59 @@ impl Document {
Ok(obj)
}
/// Visit every current object without filling this document's editing cache.
///
/// Full-document searches (for example, standalone timestamps) need to inspect objects
/// that may never be used again. Keep their parse caches private and release them in
/// batches. Returned objects own their data and remain valid after the next batch.
pub fn scan_objects(&self) -> impl Iterator<Item = (ObjRef, Arc<Object>)> + use<> {
let numbers = self.object_numbers();
let mut reader = self.object_reader();
numbers.into_iter().map(move |num| {
let reference = ObjRef::new(num, reader.document.generation(num));
(reference, reader.get(reference))
})
}
/// Like [`Self::scan_objects`], preserving read errors for callers that must distinguish
/// incomplete discovery from an actual null object. Caches remain private and bounded.
pub fn scan_objects_checked(&self) -> impl Iterator<Item = (ObjRef, Result<Arc<Object>, CosError>)> + use<> {
self.scan_objects_subset(self.object_numbers())
}
/// Inspect selected current objects with the same bounded cache as a whole-file scan.
/// Missing/freed numbers yield `Null`; malformed objects return their read error.
pub fn scan_objects_subset(&self, numbers: Vec<u32>) -> impl Iterator<Item = (ObjRef, Result<Arc<Object>, CosError>)> + use<> {
let mut reader = self.object_reader();
numbers.into_iter().map(move |num| {
let reference = ObjRef::new(num, reader.document.generation(num));
(reference, reader.try_get(reference))
})
}
/// Inspect selected objects in the immutable source, ignoring all unsaved edits. This
/// allows a consumer to check an edit's dependencies without retaining an old document.
pub fn scan_original_objects_subset(&self, numbers: Vec<u32>) -> impl Iterator<Item = (ObjRef, Result<Arc<Object>, CosError>)> + use<> {
let mut reader = self.object_reader();
reader.document.overlay.clear();
numbers.into_iter().map(move |num| {
let reference = ObjRef::new(num, reader.document.generation(num));
(reference, reader.try_get(reference))
})
}
/// Opened with a tighter stream limit than editing uses ([`Document::open_with_stream_limit`]).
pub(crate) fn stream_limited(&self) -> bool {
self.stream_limit < crate::object::MAX_DECODED
}
pub(crate) fn object_reader(&self) -> ObjectReader {
let mut document = self.clone();
document.cache = Arc::default();
document.objstms = Arc::default();
ObjectReader { document, decoded_count: 0 }
}
fn load(&self, num: u32, depth: u32) -> Result<Object, CosError> {
if depth > 16 {
return Err(CosError::Syntax { offset: 0, detail: "reference cycle while loading".into() });
@ -541,7 +704,7 @@ impl Document {
let Object::Stream(s) = &*self.try_get(num)? else { return Err(CosError::MissingObject(num)) };
let n = s.dict.int(b"N").unwrap_or(0).clamp(0, 1_000_000) as usize;
let first = s.dict.int(b"First").unwrap_or(0).max(0) as usize;
let data = s.decoded()?;
let data = s.decoded_within(self.stream_limit)?;
let mut lx = Lexer::new(&data, 0);
let mut index = Vec::with_capacity(n);
for _ in 0..n {
@ -645,16 +808,27 @@ impl Document {
/// All object numbers known (file and overlay), excluding freed ones.
pub fn object_numbers(&self) -> Vec<u32> {
let mut set: HashSet<u32> = self.entries.iter().filter(|(_, e)| !matches!(e, XrefEntry::Free { .. })).map(|(n, _)| *n).collect();
for (n, s) in &self.overlay {
match s {
Slot::Set(..) => set.insert(*n),
Slot::Freed(_) => set.remove(n),
};
// Both maps are ordered. Overlay slots replace source slots, including frees;
// merge them directly without a second set and a sort of every object number.
let mut numbers = Vec::with_capacity(self.entries.len().max(self.overlay.len()));
let mut source = self.entries.iter().peekable();
for (&num, slot) in &self.overlay {
while source.peek().is_some_and(|(n, _)| **n < num) {
if let Some((&n, entry)) = source.next()
&& !matches!(entry, XrefEntry::Free { .. })
{
numbers.push(n);
}
}
if source.peek().is_some_and(|(n, _)| **n == num) {
source.next();
}
if matches!(slot, Slot::Set(..)) {
numbers.push(num);
}
}
let mut v: Vec<u32> = set.into_iter().collect();
v.sort_unstable();
v
numbers.extend(source.filter_map(|(&num, entry)| (!matches!(entry, XrefEntry::Free { .. })).then_some(num)));
numbers
}
/// Rebase this document on bytes written by the writer (after a save): edits become part of
@ -807,7 +981,7 @@ impl Document {
// One row per object, and a document has at most 8,388,607 indirect objects (the classic
// implementation limit): decoding more than that (about 200 MB at the widest rows) is a
// decompression bomb, not cross-reference data. The section is then reconstructed.
let raw = s.decoded_within(MAX_XREF_OBJECTS.saturating_mul(row))?;
let raw = s.decoded_within(MAX_XREF_OBJECTS.saturating_mul(row).min(self.stream_limit))?;
let field = |r: &[u8], from: usize, len: usize, default: u64| -> u64 {
if len == 0 {
return default;
@ -1056,6 +1230,150 @@ mod tests {
assert!(doc.get(ObjRef::new(1, 0)).as_dict().unwrap().get(b"Lang").is_none());
}
#[test]
fn object_numbers_merge_source_and_overlay_in_order() {
let mut doc = Document::new_empty();
doc.overlay.clear();
doc.entries = Arc::new(BTreeMap::from([
(0, XrefEntry::Free { next_generation: 65535 }),
(2, XrefEntry::InFile { offset: 10, generation: 7 }),
(4, XrefEntry::Free { next_generation: 9 }),
(6, XrefEntry::InStream { stream: 80, index: 1 }),
(8, XrefEntry::InFile { offset: 20, generation: 5 }),
(10, XrefEntry::InFile { offset: 30, generation: 3 }),
(u32::MAX, XrefEntry::InFile { offset: 40, generation: 1 }),
]));
doc.set(ObjRef::new(1, 2), Object::Int(1)); // before the first live source slot
doc.set(ObjRef::new(2, 8), Object::Int(2)); // override, without duplication
doc.set(ObjRef::new(4, 9), Object::Int(4)); // reuse a source free slot
doc.free(ObjRef::new(6, 0)); // remove a compressed source object
doc.free(ObjRef::new(7, 2)); // free a number absent from the source
doc.free(ObjRef::new(10, 3));
doc.set(ObjRef::new(11, 4), Object::Int(11));
assert_eq!(doc.object_numbers(), vec![1, 2, 4, 8, 11, u32::MAX]);
assert_eq!(doc.generation(2), 8);
assert_eq!(doc.generation(4), 9);
assert_eq!(doc.generation(6), 1);
assert_eq!(doc.generation(8), 5);
assert_eq!(doc.generation(10), 4);
doc.overlay.clear();
assert_eq!(doc.object_numbers(), vec![2, 6, 8, 10, u32::MAX]);
doc.entries = Arc::default();
assert!(doc.object_numbers().is_empty());
doc.set(ObjRef::new(5, 7), Object::Int(5));
assert_eq!(doc.object_numbers(), vec![5]);
}
#[test]
fn full_object_scans_preserve_edits_without_retaining_the_scan() {
let mut bodies = vec!["<< /Type /Catalog /Pages 2 0 R >>".to_owned(), "<< /Type /Pages /Kids [] /Count 0 >>".to_owned()];
bodies.extend((0..400).map(|i| format!("<< /Value {i} >>")));
let refs: Vec<_> = bodies.iter().map(String::as_str).collect();
let mut doc = Document::open(Arc::new(build(&refs, "/Root 1 0 R"))).unwrap();
doc.set(ObjRef::new(3, 0), Object::Int(900));
doc.free(ObjRef::new(4, 0));
let added = doc.add(Object::Int(901));
let cached = doc.cache.lock().unwrap().len();
let held: Vec<_> = doc.scan_objects().collect();
assert_eq!(held.len(), 402);
assert_eq!(doc.cache.lock().unwrap().len(), cached, "scanning must not populate the editor cache");
assert!(doc.objstms.lock().unwrap().is_empty());
assert!(!held.iter().any(|(r, _)| r.num == 4));
assert_eq!(held.iter().find(|(r, _)| r.num == 3).unwrap().1.as_int(), Some(900));
assert_eq!(held.iter().find(|(r, _)| *r == added).unwrap().1.as_int(), Some(901));
assert_eq!(held.iter().find(|(r, _)| r.num == 400).unwrap().1.as_dict().unwrap().int(b"Value"), Some(397));
// Previously returned objects survive cache eviction and dropping the source.
drop(doc);
assert_eq!(held.iter().find(|(r, _)| r.num == 128).unwrap().1.as_dict().unwrap().int(b"Value"), Some(125));
}
#[test]
fn temporary_reader_keeps_file_views_and_evicts_data_of_its_own() {
// A stream read from the file is a view into the input: caching it costs no copy, so
// even a large one stays cached (its /Length object too) instead of being re-parsed.
let body = "x".repeat(ObjectReader::STREAM_BYTES + 1);
let stream = format!("<< /Length 4 0 R >>\nstream\n{body}\nendstream");
let length = body.len().to_string();
let bytes = build(&["<< /Type /Catalog /Pages 2 0 R >>", "<< /Type /Pages /Kids [] /Count 0 >>", &stream, &length], "/Root 1 0 R");
let doc = Document::open(Arc::new(bytes)).unwrap();
let mut reader = doc.object_reader();
let read = reader.get(ObjRef::new(3, 0));
let Object::Stream(s) = &*read else { panic!("stream") };
assert!(s.raw.shares(&reader.document.data), "no copy of the file's bytes");
assert!(reader.document.cache.lock().unwrap().contains_key(&3), "a view costs nothing to keep");
// Data of its own (decrypted streams, strings) counts: past the budget the cache is
// cleared after the next read, including a failed one.
let owned = Object::Stream(crate::Stream::from_raw(Dict::new(), vec![b'y'; ObjectReader::STREAM_BYTES + 1]));
reader.document.cache.lock().unwrap().insert(3, Arc::new(owned));
assert_eq!(*reader.get(ObjRef::new(99, 0)), Object::Null);
assert!(reader.document.cache.lock().unwrap().is_empty(), "owned data over budget is evicted");
}
#[test]
fn temporary_reader_and_full_save_do_not_fill_snapshot_caches() {
let mut source = Document::new_empty();
let objects: Vec<_> = (0..260)
.map(|i| {
let mut value = vec![b'x'; 48 * 1024];
value[..4].copy_from_slice(&(i as u32).to_le_bytes());
Object::Ref(source.add(Object::String(crate::PdfString::literal(value))))
})
.collect();
source.update_dict(source.root().unwrap(), |d| d.set(b"Extension".to_vec(), Object::Array(objects))).unwrap();
let bytes = Arc::new(crate::write_full(&source, &crate::SaveOptions::default()).unwrap());
let doc = Document::open(bytes).unwrap();
let root = doc.get(doc.root().unwrap());
let references = root.as_dict().unwrap().get(b"Extension").unwrap().as_array().unwrap();
let snapshot = doc.clone();
let cached = doc.cache.lock().unwrap().len();
let streams = doc.objstms.lock().unwrap().len();
let mut reader = doc.object_reader();
let mut held = None;
for (i, reference) in references.iter().enumerate() {
let Object::Ref(reference) = reference else { panic!("indirect object") };
let object = reader.get(*reference);
assert_eq!(&object.as_string().unwrap().bytes[..4], &(i as u32).to_le_bytes());
if i == 0 {
held = Some(object);
}
assert!(reader.document.cache.lock().unwrap().len() < ObjectReader::OBJECTS);
let decoded = reader.document.objstms.lock().unwrap();
assert!(decoded.len() < ObjectReader::OBJECTS);
let bytes: usize = decoded.values().map(|s| s.data.capacity() + s.index.capacity() * std::mem::size_of::<(u32, usize)>()).sum();
assert!(bytes <= ObjectReader::STREAM_BYTES || decoded.len() == 1, "only one oversized stream may remain");
}
drop(reader);
assert_eq!(&held.unwrap().as_string().unwrap().bytes[..4], &0u32.to_le_bytes(), "owned results survive evictions");
for object_streams in [false, true] {
let saved = crate::write_full(&doc, &crate::SaveOptions { object_streams, ..Default::default() }).unwrap();
assert!(Document::open(Arc::new(saved)).unwrap().repair_log().is_empty());
assert_eq!(doc.cache.lock().unwrap().len(), cached, "full saves must not fill the editor cache");
assert_eq!(doc.objstms.lock().unwrap().len(), streams);
assert!(Arc::ptr_eq(&doc.cache, &snapshot.cache), "do not detach or clear caller snapshots");
}
}
#[test]
fn a_stream_limited_document_cannot_be_saved() {
let bytes = Arc::new(crate::write_full(&Document::new_empty(), &crate::SaveOptions::default()).unwrap());
let doc = Document::open_with_stream_limit(bytes.clone(), None, 1 << 20).unwrap();
for saved in [crate::write_full(&doc, &crate::SaveOptions::default()), crate::write_incremental(&doc, &crate::SaveOptions::default())] {
assert!(matches!(saved, Err(CosError::ReadOnlyLimit)), "{saved:?}");
}
let doc = Document::open(bytes).unwrap();
assert!(crate::write_full(&doc, &crate::SaveOptions::default()).is_ok());
}
#[test]
fn inspection_stream_limit_applies_before_loading_compressed_objects() {
let mut doc = Document::new_empty();
let large = doc.add(Object::String(crate::PdfString::literal(vec![b'x'; 4096])));
doc.update_dict(doc.root().unwrap(), |catalog| catalog.set(b"Extension".to_vec(), Object::Ref(large))).unwrap();
let bytes = Arc::new(crate::write_full(&doc, &crate::SaveOptions::default()).unwrap());
assert!(Document::open(bytes.clone()).is_ok());
assert!(Document::open_with_stream_limit(bytes, None, 512).is_err());
}
/// A /Prev chain that loops back on itself stops at the loop (and says so).
#[test]
fn looping_revision_chains_stop() {

4
crates/cos/src/lib.rs
View File

@ -20,7 +20,7 @@ mod security;
mod writer;
pub use bytes::Bytes;
pub use document::{Document, Revision, XrefEntry};
pub use document::{Document, Revision, SourceIdentity, XrefEntry};
pub use object::{Dict, MAX_DECODED, Name, ObjRef, Object, PdfString, Stream};
pub use parser::{Lexer, parse_indirect};
pub use pdfcraft_crypt::{Algorithm, Auth, Method as CryptMethod, NewEncryption, Permissions, SecurityHandler};
@ -46,4 +46,6 @@ pub enum CosError {
Security(String),
#[error("internal lock poisoned")]
Poisoned,
#[error("this document was opened read-only with a decode limit and cannot be saved")]
ReadOnlyLimit,
}

235
crates/cos/src/writer.rs
View File

@ -4,13 +4,14 @@
//! as the file's last one (table or stream) and a trailer with `/Prev`. The original bytes are
//! never modified, which keeps digital signatures over earlier revisions valid.
//! * `write_full` writes only objects reachable from the trailer, renumbered densely, with a
//! classic cross-reference table — the "Save As" / garbage-collecting path.
//! cross-reference table or stream — the "Save As" / garbage-collecting path.
use std::borrow::Cow;
use std::collections::{BTreeMap, HashMap, VecDeque};
use std::io::Write;
use crate::CosError;
use crate::document::Document;
use crate::document::{Document, ObjectReader};
use crate::object::{Dict, ObjRef, Object, PdfString, Stream};
/// Serialize one object. Deterministic: same object → same bytes.
@ -122,10 +123,10 @@ fn write_stream(s: &Stream, out: &mut Vec<u8>) {
/// The object as written: encrypted with its (output) number when the document is encrypted,
/// except the `/Encrypt` dictionary itself (`source_num` is its number in the source document).
fn prepared(doc: &Document, source_num: u32, num: u32, generation: u16, o: &Object) -> Object {
fn prepared<'a>(doc: &Document, source_num: u32, num: u32, generation: u16, o: &'a Object) -> Cow<'a, Object> {
match doc.output_security() {
(Some(h), encrypt_num) if Some(source_num) != encrypt_num => crate::security::transform(h, o, num, generation, false),
_ => o.clone(),
(Some(h), encrypt_num) if Some(source_num) != encrypt_num => Cow::Owned(crate::security::transform(h, o, num, generation, false)),
_ => Cow::Borrowed(o),
}
}
@ -155,6 +156,9 @@ impl Default for SaveOptions {
/// Objects per object stream (Acrobat and qpdf use 100–200).
const OBJSTM_SIZE: usize = 100;
/// Keep a batch's serialized body small as well as limiting its object count. One object
/// may exceed this target; it is emitted in its own object stream without retaining a batch.
const OBJSTM_BYTES: usize = 8 * 1024 * 1024;
/// Where an object lives, for the cross-reference section.
#[derive(Clone, Copy, Debug, PartialEq)]
@ -173,6 +177,9 @@ enum Row {
/// incrementally — other readers would follow the broken chain — so it is rewritten in full,
/// which also repairs the file (what users expect after "the file was damaged and repaired").
pub fn write_incremental(doc: &Document, opts: &SaveOptions) -> Result<Vec<u8>, CosError> {
if doc.stream_limited() {
return Err(CosError::ReadOnlyLimit);
}
// Reconstructed files have no chain to append to; added or removed encryption must
// rewrite every object; redaction must not leave the old revision behind. All need a full save.
if doc.revisions().is_empty() || doc.encryption_changed() || doc.full_save_required() {
@ -188,7 +195,7 @@ pub fn write_incremental(doc: &Document, opts: &SaveOptions) -> Result<Vec<u8>,
let base = (original.len() + usize::from(newline)) as u64;
let mut out = Vec::new();
let mut offsets: BTreeMap<u32, (u64, u16, bool)> = BTreeMap::new(); // num → (offset, gen, in use)
for (num, generation, obj) in doc.overlay_entries().collect::<Vec<_>>() {
for (num, generation, obj) in doc.overlay_entries() {
match obj {
Some(o) => {
offsets.insert(num, (base + out.len() as u64, generation, true));
@ -229,14 +236,20 @@ pub fn write_incremental(doc: &Document, opts: &SaveOptions) -> Result<Vec<u8>,
Ok(file)
}
/// Write only reachable objects, renumbered from 1, with a classic cross-reference table.
/// Write only reachable objects, renumbered from 1, with a table or cross-reference stream.
pub fn write_full(doc: &Document, opts: &SaveOptions) -> Result<Vec<u8>, CosError> {
let mut doc = doc.clone();
stamp_mod_date(&mut doc, opts);
let trailer_in = doc.trailer().clone();
if doc.stream_limited() {
return Err(CosError::ReadOnlyLimit);
}
// A full traversal must not fill the editor's shared caches (which undo snapshots also
// own). Parsed objects and decoded object streams are temporary working data here.
let mut reader = doc.object_reader();
stamp_mod_date(&mut reader.document, opts);
let trailer_in = reader.document.trailer().clone();
// Breadth-first walk from the trailer, assigning new numbers in visit order.
let mut map: HashMap<ObjRef, u32> = HashMap::new();
let mut order: Vec<ObjRef> = Vec::new();
// Record the type while traversing so classifying packed objects needs no second parse.
let mut order: Vec<(ObjRef, bool)> = Vec::new();
let mut queue: VecDeque<ObjRef> = VecDeque::new();
let visit_refs = |o: &Object, queue: &mut VecDeque<ObjRef>| collect_refs(o, &mut |r| queue.push_back(r));
for key in [&b"Root"[..], b"Info", b"Encrypt"] {
@ -248,15 +261,15 @@ pub fn write_full(doc: &Document, opts: &SaveOptions) -> Result<Vec<u8>, CosErro
if map.contains_key(&r) {
continue;
}
let o = doc.get(r);
let o = reader.get(r);
if matches!(*o, Object::Null) {
continue; // dangling reference: written as null by renumbering below
}
map.insert(r, (order.len() + 1) as u32);
order.push(r);
map.insert(r, output_number(order.len())?);
order.push((r, matches!(*o, Object::Stream(_))));
visit_refs(&o, &mut queue);
}
let version = doc.version().chars().take(3).collect::<String>();
let version = reader.document.version().chars().take(3).collect::<String>();
let version = if opts.object_streams && version.as_str() < "1.5" { "1.5".to_string() } else { version };
let mut out = Vec::new();
let _ = writeln!(out, "%PDF-{version}");
@ -268,46 +281,45 @@ pub fn write_full(doc: &Document, opts: &SaveOptions) -> Result<Vec<u8>, CosErro
let mut pending: Vec<ObjRef> = trailer_in.reference(b"Encrypt").into_iter().collect();
while let Some(r) = pending.pop() {
if encryption_objects.insert(r) {
collect_refs(&doc.get(r), &mut |x| pending.push(x));
collect_refs(&reader.get(r), &mut |x| pending.push(x));
}
}
let mut rows: BTreeMap<u32, Row> = BTreeMap::new();
let mut packed: Vec<(u32, Object)> = Vec::new(); // objects bound for object streams
for (i, r) in order.iter().enumerate() {
let num = (i + 1) as u32;
let o = renumber(&doc.get(*r), &map);
let in_stream = opts.object_streams && !matches!(o, Object::Stream(_)) && !encryption_objects.contains(r);
// Keep only references, not renamed copies of the entire object graph. Standalone
// objects still precede object streams, preserving the established output order.
let mut packed: Vec<(u32, ObjRef)> = Vec::new();
for (i, (r, is_stream)) in order.iter().enumerate() {
let num = output_number(i)?;
let in_stream = opts.object_streams && !is_stream && !encryption_objects.contains(r);
if in_stream {
packed.push((num, o));
packed.push((num, *r));
} else {
let o = renumber(&*reread(&mut reader, *r, *is_stream)?, &map);
rows.insert(num, Row::InFile(out.len() as u64, 0));
write_indirect(num, 0, &prepared(&doc, r.num, num, 0, &o), &mut out);
write_indirect(num, 0, &prepared(&reader.document, r.num, num, 0, &o), &mut out);
}
}
// Object streams take the numbers after the last object. Their contents are not encrypted
// individually: the whole stream is (§7.5.7, §7.6.2).
let mut next = order.len() as u32 + 1;
for chunk in packed.chunks(OBJSTM_SIZE) {
let stm_num = next;
next += 1;
let mut header = Vec::new();
let mut body = Vec::new();
for (index, (num, o)) in chunk.iter().enumerate() {
let _ = write!(header, "{} {} ", num, body.len());
serialize(o, &mut body);
body.push(b'\n');
rows.insert(*num, Row::InStream(stm_num, index as u32));
let mut next = output_number(order.len())?;
let mut batch = ObjectStreamBatch::default();
for (num, reference) in packed {
let mut encoded = Vec::new();
serialize(&renumber(&*reread(&mut reader, reference, false)?, &map), &mut encoded);
if batch.count > 0 && batch.body.len().saturating_add(encoded.len()).saturating_add(1) > OBJSTM_BYTES {
batch.flush(&reader.document, &mut next, &mut rows, &mut out)?;
}
let _ = write!(batch.header, "{} {} ", num, batch.body.len());
batch.body.extend_from_slice(&encoded);
batch.body.push(b'\n');
drop(encoded);
rows.insert(num, Row::InStream(next, batch.count as u32));
batch.count += 1;
if batch.count == OBJSTM_SIZE || batch.body.len() >= OBJSTM_BYTES {
batch.flush(&reader.document, &mut next, &mut rows, &mut out)?;
}
let first = header.len();
header.extend_from_slice(&body);
let mut d = Dict::new();
d.set(b"Type".to_vec(), Object::name("ObjStm"));
d.set(b"N".to_vec(), Object::Int(chunk.len() as i64));
d.set(b"First".to_vec(), Object::Int(first as i64));
let stm = Object::Stream(Stream::flate(d, &header));
rows.insert(stm_num, Row::InFile(out.len() as u64, 0));
write_indirect(stm_num, 0, &prepared(&doc, u32::MAX, stm_num, 0, &stm), &mut out);
}
batch.flush(&reader.document, &mut next, &mut rows, &mut out)?;
let mut trailer = Dict::new();
for key in [&b"Root"[..], b"Info", b"Encrypt"] {
if let Some(r) = trailer_in.reference(key).and_then(|r| map.get(&r)) {
@ -320,8 +332,8 @@ pub fn write_full(doc: &Document, opts: &SaveOptions) -> Result<Vec<u8>, CosErro
if let Some(id) = trailer_in.get(b"ID") {
trailer.set(b"ID".to_vec(), id.clone());
}
if doc.output_security().0.is_none() {
ensure_id(&mut trailer, opts, doc.bytes());
if reader.document.output_security().0.is_none() {
ensure_id(&mut trailer, opts, reader.document.bytes());
}
if opts.object_streams {
write_xref_stream(&mut out, 0, &mut trailer, rows, next);
@ -341,6 +353,52 @@ pub fn write_full(doc: &Document, opts: &SaveOptions) -> Result<Vec<u8>, CosErro
Ok(out)
}
/// Object `r` read again for writing. The traversal read it once and only kept its number, so
/// a second read that fails, finds nothing or finds a different kind of object must fail the
/// save rather than write `null` in its place.
fn reread(reader: &mut ObjectReader, r: ObjRef, is_stream: bool) -> Result<std::sync::Arc<Object>, CosError> {
let o = reader.try_get(r)?;
if matches!(*o, Object::Null) || matches!(*o, Object::Stream(_)) != is_stream {
return Err(CosError::Syntax { offset: 0, detail: format!("object {} read differently while saving; nothing was written", r.num) });
}
Ok(o)
}
#[derive(Default)]
struct ObjectStreamBatch {
header: Vec<u8>,
body: Vec<u8>,
count: usize,
}
impl ObjectStreamBatch {
fn flush(&mut self, doc: &Document, next: &mut u32, rows: &mut BTreeMap<u32, Row>, out: &mut Vec<u8>) -> Result<(), CosError> {
if self.count == 0 {
return Ok(());
}
let following = next.checked_add(1).ok_or_else(|| CosError::Syntax { offset: 0, detail: "too many objects to save".into() })?;
// Release capacity after each batch too: an unusually large object must not leave
// a large scratch buffer alive for the rest of the save.
let Self { mut header, body, count } = std::mem::take(self);
let first = header.len();
header.extend_from_slice(&body);
drop(body);
let mut d = Dict::new();
d.set(b"Type".to_vec(), Object::name("ObjStm"));
d.set(b"N".to_vec(), Object::Int(count as i64));
d.set(b"First".to_vec(), Object::Int(first as i64));
let stm = Object::Stream(Stream::flate(d, &header));
rows.insert(*next, Row::InFile(out.len() as u64, 0));
write_indirect(*next, 0, &prepared(doc, u32::MAX, *next, 0, &stm), out);
*next = following;
Ok(())
}
}
fn output_number(index: usize) -> Result<u32, CosError> {
u32::try_from(index).ok().and_then(|n| n.checked_add(1)).ok_or_else(|| CosError::Syntax { offset: 0, detail: "too many objects to save".into() })
}
fn collect_refs(o: &Object, f: &mut impl FnMut(ObjRef)) {
match o {
Object::Ref(r) => f(*r),
@ -401,27 +459,33 @@ fn write_xref_stream(out: &mut Vec<u8>, base: u64, trailer: &mut Dict, mut rows:
let (w2, w3) = rows.values().map(fields).fold((1, 1), |(a, b), (_, x, y)| (width(x).max(a), width(y).max(b)));
let cols = 1 + w2 + w3;
let mut index = Vec::new();
let nums: Vec<u32> = rows.keys().copied().collect();
let mut i = 0;
while i < nums.len() {
let mut j = i;
while j + 1 < nums.len() && nums[j + 1] == nums[j] + 1 {
j += 1;
let mut numbers = rows.keys().copied().peekable();
while let Some(start) = numbers.next() {
let mut last = start;
let mut count = 1i64;
while numbers.peek().copied().is_some_and(|next| last.checked_add(1) == Some(next)) {
if let Some(next) = numbers.next() {
last = next;
count += 1;
}
}
index.push(Object::Int(nums[i] as i64));
index.push(Object::Int((j - i + 1) as i64));
i = j + 1;
index.push(Object::Int(i64::from(start)));
index.push(Object::Int(count));
}
let mut data = Vec::with_capacity(rows.len() * (cols + 1));
let mut prev = vec![0u8; cols];
let mut data = Vec::with_capacity(rows.len().saturating_mul(cols + 1));
// Type is one byte, and each remaining field is at most one u64. Reuse the
// predictor row on the stack instead of allocating a Vec for every xref entry.
let mut prev = [0u8; 17];
for r in rows.values() {
let (t, a, b) = fields(r);
let mut row = vec![t as u8];
row.extend_from_slice(&a.to_be_bytes()[8 - w2..]);
row.extend_from_slice(&b.to_be_bytes()[8 - w3..]);
let a = a.to_be_bytes();
let b = b.to_be_bytes();
let fields = std::iter::once(t as u8).chain(a.into_iter().skip(8 - w2)).chain(b.into_iter().skip(8 - w3));
data.push(2); // PNG "Up"
data.extend(row.iter().zip(&prev).map(|(c, p)| c.wrapping_sub(*p)));
prev = row;
for (value, previous) in fields.zip(&mut prev) {
data.push(value.wrapping_sub(*previous));
*previous = value;
}
}
let mut d = trailer.clone();
d.set(b"Type".to_vec(), Object::name("XRef"));
@ -503,6 +567,55 @@ mod tests {
Lexer::new(&out, 0).object().expect("parses")
}
#[test]
fn a_second_read_that_differs_fails_the_save_instead_of_writing_null() {
let mut doc = Document::new_empty();
let dict = doc.add(Object::Dict(Dict::new()));
let mut reader = doc.object_reader();
assert!(reread(&mut reader, dict, false).is_ok());
assert!(reread(&mut reader, dict, true).is_err(), "a dictionary where a stream was traversed");
assert!(reread(&mut reader, ObjRef::new(9999, 0), false).is_err(), "nothing where an object was traversed");
}
#[test]
fn xref_predictor_rows_preserve_variable_width_fields_and_sparse_runs() {
for offset in [0, 255, 256, 65535, 65536, u32::MAX as u64, u32::MAX as u64 + 1, u64::MAX] {
for index in [0, 255, 256, u32::MAX] {
for incremental in [false, true] {
let mut trailer = Dict::new();
if incremental {
trailer.set(b"Prev".to_vec(), Object::Int(17));
}
let rows = BTreeMap::from([(1, Row::InFile(offset, 32768)), (2, Row::Free(65535)), (4, Row::InStream(65536, index))]);
let mut bytes = Vec::new();
write_xref_stream(&mut bytes, 0, &mut trailer, rows, 5);
let (_, object) = crate::parser::parse_indirect(&bytes, 0, &|_| None).unwrap();
let Object::Stream(stream) = object else { panic!("xref stream") };
let widths: Vec<_> = stream.dict.get(b"W").unwrap().as_array().unwrap().iter().map(|o| o.as_int().unwrap() as usize).collect();
assert_eq!(widths[0], 1);
assert!((3..=8).contains(&widths[1]));
assert!((2..=4).contains(&widths[2]));
let runs: Vec<_> = stream.dict.get(b"Index").unwrap().as_array().unwrap().iter().map(|o| o.as_int().unwrap()).collect();
assert_eq!(runs, if incremental { vec![1, 2, 4, 2] } else { vec![0, 3, 4, 2] });
let mut expected = vec![(1, offset, 32768), (0, 0, 65535), (2, 65536, u64::from(index)), (1, 0, 0)];
if !incremental {
expected.insert(0, (0, 0, 65535));
}
let decoded = stream.decoded().unwrap();
let columns: usize = widths.iter().sum();
assert_eq!(decoded.len(), columns * expected.len());
for (row, (kind, first, second)) in decoded.chunks_exact(columns).zip(expected) {
let values: Vec<_> = [0..1, 1..1 + widths[1], 1 + widths[1]..columns]
.into_iter()
.map(|range| row[range].iter().fold(0u64, |value, byte| value * 256 + u64::from(*byte)))
.collect();
assert_eq!(values, vec![kind, first, second]);
}
}
}
}
}
#[test]
fn reals_keep_full_precision_without_exponents() {
for r in [0.1, -0.000_000_123_456_789, 1e-12, 123_456.789_012_345, 1.0 / 3.0, -2.5e7 + 0.25] {

236
crates/cos/tests/bounded_writer.rs Normal file
View File

@ -0,0 +1,236 @@
//! Full rewrites must preserve the graph when temporary parse/packing caches are evicted.
//! Fixtures are generated here; no third-party PDF assets are used.
use std::collections::{BTreeMap, HashSet};
use std::sync::Arc;
use pdfcraft_cos::{Algorithm, Dict, Document, NewEncryption, ObjRef, Object, PdfString, SaveOptions, XrefEntry, write_full};
const NODES: usize = 260;
const EDITED: usize = 129;
fn node(i: usize) -> ObjRef {
ObjRef::new(301 + i as u32, (i % 5 + 1) as u16)
}
fn stream_body(i: usize) -> Vec<u8> {
format!("opaque stream {i}: preserve these bytes\n").into_bytes()
}
/// Nonzero generations, sparse numbers, shared references, a cycle and interleaved streams.
/// A raw classic xref avoids depending on the writer under test to construct the input.
fn fixture() -> Vec<u8> {
let mut objects = vec![
(ObjRef::new(10, 2), format!("<< /Type /Catalog /Pages 20 0 R /Extension {} /Shared 42 3 R /Freed 43 8 R /Missing 9000 0 R >>", node(0))),
(ObjRef::new(20, 0), "<< /Type /Pages /Kids [] /Count 0 >>".into()),
(ObjRef::new(42, 3), "<< /Token (shared extension value) >>".into()),
(ObjRef::new(43, 8), "<< /Token (deleted extension value) >>".into()),
(ObjRef::new(70, 4), "<< /Title (original title) >>".into()),
];
for i in 0..NODES {
let fields = format!("/Value {i} /Next {} /Shared 42 3 R /Opaque << /Nested [42 3 R (retained data)] >>", node((i + 1) % NODES));
let body = if i % 19 == 0 {
let raw = stream_body(i);
format!("<< {fields} /Length {} >>\nstream\n{}endstream", raw.len(), String::from_utf8(raw).unwrap())
} else {
format!("<< {fields} >>")
};
objects.push((node(i), body));
}
let mut bytes = b"%PDF-1.7\n".to_vec();
let mut offsets = BTreeMap::new();
for (reference, body) in objects {
offsets.insert(reference.num, (bytes.len(), reference.generation));
bytes.extend_from_slice(format!("{} {} obj\n{body}\nendobj\n", reference.num, reference.generation).as_bytes());
}
let size = offsets.last_key_value().unwrap().0 + 1;
let xref = bytes.len();
bytes.extend_from_slice(format!("xref\n0 {size}\n").as_bytes());
for n in 0..size {
match offsets.get(&n) {
Some((offset, generation)) => bytes.extend_from_slice(format!("{offset:010} {generation:05} n \n").as_bytes()),
None => bytes.extend_from_slice(b"0000000000 65535 f \n"),
}
}
bytes.extend_from_slice(format!("trailer\n<< /Size {size} /Root 10 2 R /Info 70 4 R >>\nstartxref\n{xref}\n%%EOF\n").as_bytes());
bytes
}
fn edited_document() -> Document {
let mut doc = Document::open(Arc::new(fixture())).unwrap();
assert!(doc.repair_log().is_empty());
assert_eq!(doc.root(), Some(ObjRef::new(10, 2)));
assert_eq!(doc.generation(node(EDITED).num), node(EDITED).generation);
doc.update_dict(node(EDITED), |d| d.set(b"Value".to_vec(), Object::Int(8129))).unwrap();
doc.free(ObjRef::new(43, 8));
let added = doc.add(Object::String(PdfString::text("added extension value")));
doc.update_dict(doc.root().unwrap(), |d| d.set(b"Added".to_vec(), Object::Ref(added))).unwrap();
doc
}
fn check_graph(doc: &Document) {
assert!(doc.repair_log().is_empty(), "{:?}", doc.repair_log());
let catalog = doc.get(doc.root().unwrap());
let catalog = catalog.as_dict().unwrap();
let shared = catalog.reference(b"Shared").unwrap();
let shared_object = doc.get(shared);
assert_eq!(shared_object.as_dict().unwrap().get(b"Token").unwrap().as_string().unwrap().to_text(), "shared extension value");
assert_eq!(catalog.get(b"Freed"), Some(&Object::Null));
assert_eq!(catalog.get(b"Missing"), Some(&Object::Null));
assert_eq!(doc.resolve(catalog.get(b"Added").unwrap()).as_string().unwrap().to_text(), "added extension value");
let first = catalog.reference(b"Extension").unwrap();
let mut current = first;
let mut visited = HashSet::new();
for i in 0..NODES {
assert!(visited.insert(current), "cycle closed before node {i}");
let object = doc.get(current);
let dict = object.as_dict().unwrap();
assert_eq!(dict.int(b"Value"), Some(if i == EDITED { 8129 } else { i as i64 }));
assert_eq!(dict.reference(b"Shared"), Some(shared));
let opaque = dict.get(b"Opaque").unwrap().as_dict().unwrap().get(b"Nested").unwrap().as_array().unwrap();
assert_eq!(opaque.first(), Some(&Object::Ref(shared)));
assert_eq!(opaque.get(1).unwrap().as_string().unwrap().to_text(), "retained data");
if i % 19 == 0 {
let Object::Stream(stream) = object.as_ref() else { panic!("node {i} lost its stream") };
assert_eq!(stream.decoded().unwrap(), stream_body(i));
} else {
assert!(matches!(object.as_ref(), Object::Dict(_)));
}
current = dict.reference(b"Next").unwrap();
}
assert_eq!(current, first, "the extension cycle is preserved");
assert!(doc.object_numbers().into_iter().all(|n| doc.generation(n) == 0));
}
#[test]
fn full_rewrites_preserve_graph_across_multiple_object_streams_and_evictions() {
for object_streams in [false, true] {
let doc = edited_document();
let opts = SaveOptions { object_streams, mod_date: Some("D:20261008000000Z".into()), ..SaveOptions::default() };
let saved = write_full(&doc, &opts).unwrap();
assert_eq!(saved, write_full(&doc, &opts).unwrap(), "unchanged inputs produce identical bytes");
let back = Document::open(Arc::new(saved)).unwrap();
check_graph(&back);
let info = back.resolve(back.trailer().get(b"Info").unwrap());
let info = info.as_dict().unwrap();
assert_eq!(info.get(b"Title").unwrap().as_string().unwrap().to_text(), "original title");
assert_eq!(info.get(b"ModDate").unwrap().as_string().unwrap().to_text(), "D:20261008000000Z");
let containers: HashSet<_> = back
.object_numbers()
.into_iter()
.filter_map(|num| match back.xref_entry(num) {
Some(XrefEntry::InStream { stream, .. }) => Some(stream),
_ => None,
})
.collect();
if object_streams {
assert!(containers.len() >= 3, "fixture must span multiple packed-object chunks");
} else {
assert!(containers.is_empty());
}
// Read an already packed input as well: eviction must not lose decoded ObjStm data.
check_graph(&Document::open(Arc::new(write_full(&back, &opts).unwrap())).unwrap());
// Writing must not apply the date stamp or consume the caller's edits.
let original_info = doc.resolve(doc.trailer().get(b"Info").unwrap());
assert!(!original_info.as_dict().unwrap().contains(b"ModDate"));
assert!(doc.is_modified());
assert_eq!(doc.get(node(EDITED)).as_dict().unwrap().int(b"Value"), Some(8129));
}
}
fn protect(doc: &mut Document, algorithm: Algorithm) {
doc.set_encryption(&NewEncryption {
algorithm,
user_password: "user",
owner_password: "owner",
permissions: -4,
encrypt_metadata: true,
seed: [23; 32],
})
.unwrap();
}
#[test]
fn encrypted_full_rewrites_preserve_data_across_evictions() {
for algorithm in [Algorithm::Rc4_40, Algorithm::Rc4_128, Algorithm::Aes128, Algorithm::Aes256] {
let mut doc = edited_document();
protect(&mut doc, algorithm);
let encrypted = Arc::new(write_full(&doc, &SaveOptions::default()).unwrap());
let cold = Document::open_with_password(encrypted, Some("user")).unwrap();
// The second rewrite must decrypt lazily from the input, then encrypt the new streams.
let rewritten = Arc::new(write_full(&cold, &SaveOptions::default()).unwrap());
assert!(Document::open_with_password(rewritten.clone(), Some("wrong")).is_err());
let back = Document::open_with_password(rewritten, Some("owner")).unwrap();
assert!(back.security().is_some());
check_graph(&back);
}
}
#[test]
fn encryption_dependencies_stay_outside_object_streams_across_evictions() {
let mut doc = edited_document();
protect(&mut doc, Algorithm::Aes256);
let encrypt = doc.trailer().reference(b"Encrypt").unwrap();
let filters = doc.get(encrypt).as_dict().unwrap().get(b"CF").cloned().unwrap();
let filters = doc.add(filters);
let mut next = filters;
for i in (0..140).rev() {
let mut dict = Dict::new();
dict.set(b"Value".to_vec(), Object::Int(i));
dict.set(b"Next".to_vec(), Object::Ref(next));
next = doc.add(dict);
}
doc.update_dict(encrypt, |dict| {
dict.set(b"CF".to_vec(), Object::Ref(filters));
dict.set(b"PrivateChain".to_vec(), Object::Ref(next));
})
.unwrap();
let saved = Arc::new(write_full(&doc, &SaveOptions::default()).unwrap());
let cold = Document::open_with_password(saved, Some("user")).unwrap();
let rewritten = Arc::new(write_full(&cold, &SaveOptions::default()).unwrap());
let back = Document::open_with_password(rewritten, Some("user")).unwrap();
check_graph(&back);
let encrypt = back.trailer().reference(b"Encrypt").unwrap();
assert!(matches!(back.xref_entry(encrypt.num), Some(XrefEntry::InFile { .. })));
let encrypt = back.get(encrypt);
let encrypt = encrypt.as_dict().unwrap();
let filters = encrypt.reference(b"CF").unwrap();
assert!(matches!(back.xref_entry(filters.num), Some(XrefEntry::InFile { .. })));
let mut current = encrypt.reference(b"PrivateChain").unwrap();
for i in 0..140 {
assert!(matches!(back.xref_entry(current.num), Some(XrefEntry::InFile { .. })));
let object = back.get(current);
let dict = object.as_dict().unwrap();
assert_eq!(dict.int(b"Value"), Some(i));
current = dict.reference(b"Next").unwrap();
}
assert_eq!(current, filters);
}
#[test]
fn large_packed_strings_round_trip_across_byte_limited_batches() {
let mut doc = Document::new_empty();
let mut references = Vec::new();
// More than 8 MiB in fewer than 100 objects: a byte cap may flush before the count cap.
for i in 0..70u8 {
let mut bytes = vec![b'a' + i % 26; 192 * 1024];
bytes[0] = i;
references.push(Object::Ref(doc.add(Object::String(PdfString::literal(bytes)))));
}
doc.update_dict(doc.root().unwrap(), |dict| dict.set(b"Extension".to_vec(), Object::Array(references))).unwrap();
let source = Arc::new(write_full(&doc, &SaveOptions { object_streams: false, ..SaveOptions::default() }).unwrap());
drop(doc);
let cold = Document::open(source).unwrap();
let back = Document::open(Arc::new(write_full(&cold, &SaveOptions::default()).unwrap())).unwrap();
assert!(back.repair_log().is_empty());
let catalog = back.get(back.root().unwrap());
let values = catalog.as_dict().unwrap().get(b"Extension").unwrap().as_array().unwrap();
assert_eq!(values.len(), 70);
for (i, reference) in values.iter().enumerate() {
let value = back.resolve(reference);
let bytes = &value.as_string().unwrap().bytes;
assert_eq!(bytes.len(), 192 * 1024);
assert_eq!(bytes[0], i as u8);
assert!(bytes[1..].iter().all(|&b| b == b'a' + i as u8 % 26));
}
}

65
crates/render/src/inspect.rs
View File

@ -1,8 +1,9 @@
//! Document inspection: everything panels need that is not pixels.
//!
//! Page geometry comes from hayro (which resolves inheritance and rotation); the rest comes from
//! lopdf (bootstrap, replaced by `pdfcraft-model` in M2). Inspection is *tolerant*: when lopdf
//! cannot load a file that hayro can render, panels are simply empty and `warnings` says why.
//! the lazy COS reader through a read-only lopdf object adapter. The original lopdf loader
//! remains the compatibility fallback. When neither can inspect a renderable file, panels
//! are empty and `warnings` says why.
use std::collections::{BTreeMap, HashMap, HashSet};
use std::sync::Arc;
@ -16,6 +17,7 @@ use lopdf::{Dictionary, Document, LoadOptions, Object, ObjectId};
use pdfcraft_cos::page_labels::{MAX_LABEL_BYTES, MAX_LABEL_TOTAL_BYTES, MAX_LABEL_TREE_DEPTH, MAX_LABEL_TREE_WORK, MAX_PREFIX_BYTES, alpha, roman};
use crate::OpenError;
use crate::structure::{LazyStructure, Structure};
/// lopdf decodes object and cross-reference streams while it loads, with no limit unless one is
/// set: a few hundred bytes of nested FlateDecode then inflate to gigabytes. Real object and
@ -372,7 +374,28 @@ pub fn inspect(bytes: Arc<Vec<u8>>, password: Option<&str>) -> Result<DocInfo, O
Err(p) => return Err(OpenError::Invalid(format!("the page tree could not be read: {}", crate::raster::panic_message(&p)))),
}
let options = load_options(password);
// Geometry no longer needs the renderer parser. Do not overlap its allocations with
// structural inspection, including the compatibility fallback for damaged files.
drop(pdf);
inspect_structure(&mut info, |tmp| {
// The lazy reader first; lopdf's repair/tolerance path when it can't expose the same
// page tree or an object a panel needs. No partial result is published.
let lazy = catch_unwind(AssertUnwindSafe(|| {
let doc = LazyStructure::new(bytes.clone(), password).ok()?;
let mut t = DocInfo { pages: tmp.pages.clone(), ..Default::default() };
let inspector = Inspector::new(&doc);
if inspector.page_index.len() != t.pages.len() {
return None;
}
inspector.fill(&mut t);
(!doc.failed()).then_some(t)
}))
.ok()
.flatten();
if let Some(t) = lazy {
*tmp = t;
return Ok(());
}
let doc = Document::load_mem_with_options(&bytes, options).map_err(|e| e.to_string())?;
Inspector::new(&doc).fill(tmp);
Ok(())
@ -404,7 +427,7 @@ fn inspect_structure(info: &mut DocInfo, fill: impl FnOnce(&mut DocInfo) -> Resu
}
struct Inspector<'a> {
doc: &'a Document,
doc: &'a dyn Structure,
page_index: HashMap<ObjectId, usize>,
/// Named destinations (`/Names /Dests` tree), keyed by raw string bytes, built once.
/// Looking each name up by walking the tree was O(links × names): minutes on manuals.
@ -412,7 +435,7 @@ struct Inspector<'a> {
}
impl<'a> Inspector<'a> {
fn new(doc: &'a Document) -> Self {
fn new(doc: &'a dyn Structure) -> Self {
let page_index = doc.get_pages().into_iter().map(|(n, id)| (id, n as usize - 1)).collect();
let mut me = Self { doc, page_index, named: HashMap::new() };
let mut named = HashMap::new();
@ -427,8 +450,8 @@ impl<'a> Inspector<'a> {
fn fill(&self, info: &mut DocInfo) {
// lopdf drops /Encrypt from the trailer once it has decrypted the file.
info.encrypted = self.doc.is_encrypted() || self.doc.was_encrypted() || self.doc.trailer.get(b"Encrypt").is_ok();
if let Some(d) = self.doc.trailer.get(b"Info").ok().and_then(|o| self.dict(o)) {
info.encrypted = self.doc.encrypted();
if let Some(d) = self.doc.trailer().get(b"Info").ok().and_then(|o| self.dict(o)) {
info.title = self.text(d, b"Title");
info.author = self.text(d, b"Author");
info.subject = self.text(d, b"Subject");
@ -1493,6 +1516,36 @@ trailer << /Root 1 0 R >>
assert_eq!(info.layer_groups, [vec![(5, 0), (6, 0)], vec![(6, 0), (7, 0)]]);
}
#[test]
fn lazy_structure_matches_the_compatibility_inspector() {
// Each fixture as lopdf writes it, and with its objects in object streams.
let packed = |bytes: &[u8]| {
let doc = pdfcraft_cos::Document::open(Arc::new(bytes.to_vec())).unwrap();
pdfcraft_cos::write_full(&doc, &pdfcraft_cos::SaveOptions { object_streams: true, ..Default::default() }).unwrap()
};
let fixtures = [ATTACHMENTS, LAYERS, DESTS].into_iter().flat_map(|bytes| {
let mut original = Document::load_mem(bytes).unwrap();
let mut plain = Vec::new();
original.save_to(&mut plain).unwrap();
let stm = packed(&plain);
[plain, stm]
});
for bytes in fixtures {
let lazy = LazyStructure::new(Arc::new(bytes.clone()), None).unwrap();
let eager = Document::load_mem(&bytes).unwrap();
let mut a = inspect(Arc::new(bytes), None).unwrap();
let mut b = DocInfo { pages: a.pages.clone(), ..Default::default() };
let mut c = DocInfo { pages: a.pages.clone(), ..Default::default() };
Inspector::new(&lazy).fill(&mut b);
Inspector::new(&eager).fill(&mut c);
assert!(!lazy.failed());
assert_eq!(format!("{b:?}"), format!("{c:?}"));
a.file_size = 0;
a.pdf_version.clear();
assert_eq!(format!("{a:?}"), format!("{c:?}"));
}
}
#[test]
fn links_read_set_layer_actions() {
let info = inspect(Arc::new(LAYERS.to_vec()), None).expect("opens");

7
crates/render/src/lib.rs
View File

@ -1,9 +1,9 @@
//! pdfcraft-render — page rasterization and document inspection.
//!
//! **Bootstrap status (see ADR-0004):** rasterization goes straight through the `hayro` crate
//! (hayro-interpret + vello_cpu), and inspection (outline, annotations, fields, layers,
//! attachments, metadata) uses `lopdf`. Both are replaced by our own `cos`/`model` crates and the
//! DisplayList device design in M1–M2. The public API here is what the engine and UI rely on, so
//! (hayro-interpret + vello_cpu). Inspection uses the lazy `cos` reader with a read-only lopdf
//! object adapter and retains `lopdf` loading for compatibility repairs. The renderer and that
//! adapter are replaced by `model` and the DisplayList design in M1–M2. The public API is what the engine and UI rely on, so
//! the swap stays internal to this crate.
#![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable)]
@ -11,6 +11,7 @@
mod inspect;
mod pixels;
mod raster;
mod structure;
pub mod text;
pub use inspect::{

12
crates/render/src/pixels.rs
View File

@ -24,6 +24,18 @@ impl Pixels {
pub fn into_words(self) -> Vec<u32> {
self.0
}
/// Bytes the buffer holds allocated (its capacity), for memory budgets.
pub(crate) fn byte_capacity(&self) -> usize {
self.0.capacity().saturating_mul(4)
}
}
/// Pixels from words, one per pixel, its bytes in R, G, B, A memory order.
impl From<Vec<u32>> for Pixels {
fn from(words: Vec<u32>) -> Self {
Self(words)
}
}
impl Deref for Pixels {

921
crates/render/src/raster.rs
View File

File diff suppressed because it is too large Load Diff

499
crates/render/src/raster/work_queue.rs Normal file
View File

@ -0,0 +1,499 @@
//! Pending work and bounded completed results. Pixels already being rendered are outside
//! this storage budget; at most one oversized completed result is admitted by itself.
use std::collections::{HashMap, HashSet, VecDeque};
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{Arc, Condvar, Mutex};
#[cfg(any(test, not(target_arch = "wasm32")))]
use std::time::Duration;
use super::{RenderRequest, RenderedPage, RequestKind, Tile, lock};
const RESULT_BYTES: usize = 32 * 1024 * 1024;
const RESULT_COUNT: usize = 32;
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
struct Key {
page: usize,
kind: RequestKind,
tile: Option<Tile>,
scale: u32,
tag: u64,
}
impl From<RenderRequest> for Key {
fn from(r: RenderRequest) -> Self {
Self { page: r.page, kind: r.kind, tile: r.tile, scale: r.scale.to_bits(), tag: r.tag }
}
}
struct Completed {
page: RenderedPage,
bytes: usize,
valid: Option<Arc<AtomicBool>>,
}
fn valid(token: &Option<Arc<AtomicBool>>) -> bool {
token.as_ref().is_none_or(|v| v.load(Ordering::Acquire))
}
#[cfg(any(test, not(target_arch = "wasm32")))]
fn result_bytes(page: &RenderedPage) -> usize {
let mut bytes = std::mem::size_of::<RenderedPage>().saturating_add(page.rgba.byte_capacity());
if let Some(error) = &page.error {
bytes = bytes.saturating_add(error.capacity());
}
if let Some(text) = &page.text {
bytes = bytes
.saturating_add(std::mem::size_of_val(&**text))
.saturating_add(text.glyphs.capacity().saturating_mul(std::mem::size_of::<crate::TextGlyph>()))
.saturating_add(text.line_of.capacity().saturating_mul(std::mem::size_of::<u32>()))
.saturating_add(text.space_before.capacity());
for glyph in &text.glyphs {
bytes = bytes.saturating_add(glyph.text.capacity());
}
}
bytes
}
#[derive(Default)]
struct State {
/// Most urgent last, so a worker can pop in constant time.
pending: Vec<RenderRequest>,
/// Current consumer priority: zero is the most urgent visible request.
desired: HashMap<Key, usize>,
active: HashSet<Key>,
results: VecDeque<Completed>,
bytes: usize,
closed: bool,
}
impl State {
fn retry(&mut self, req: RenderRequest) {
let key = Key::from(req);
self.active.remove(&key);
if !self.closed
&& let Some(&priority) = self.desired.get(&key)
&& !self.pending.iter().any(|r| Key::from(*r) == key)
{
// Preserve reverse demand order: the pop end belongs to visible work, even
// when an older background completion needs retrying after parser retirement.
let index = self.pending.partition_point(|r| self.desired.get(&Key::from(*r)).copied().unwrap_or(usize::MAX) > priority);
self.pending.insert(index, req);
}
}
fn take(&mut self) -> Option<RenderedPage> {
while !self.results.is_empty() {
// Completion order can differ from demand order, especially after a page jump
// while search results are buffered. Look up current priority at delivery so
// arrivals after replace() also put visible pixels ahead of background work.
let index = self
.results
.iter()
.enumerate()
.min_by_key(|(_, r)| self.desired.get(&Key::from(r.page.request)).copied().unwrap_or(usize::MAX))
.map(|(index, _)| index)?;
let done = self.results.remove(index)?;
self.bytes = self.bytes.saturating_sub(done.bytes);
let key = Key::from(done.page.request);
if !self.desired.contains_key(&key) {
continue;
}
if !valid(&done.valid) {
self.retry(done.page.request);
continue;
}
return Some(done.page);
}
None
}
}
pub(super) struct WorkQueue {
state: Mutex<State>,
changed: Condvar,
max_bytes: usize,
max_count: usize,
}
impl Default for WorkQueue {
fn default() -> Self {
Self { state: Mutex::default(), changed: Condvar::new(), max_bytes: RESULT_BYTES, max_count: RESULT_COUNT }
}
}
impl WorkQueue {
pub(super) fn replace(&self, requests: Vec<RenderRequest>) {
let mut state = lock(&self.state);
if state.closed {
return;
}
let mut desired = HashSet::with_capacity(requests.len());
let requests: Vec<_> = requests.into_iter().filter(|r| desired.insert(Key::from(*r))).collect();
state.results.retain(|r| desired.contains(&Key::from(r.page.request)) && valid(&r.valid));
state.bytes = state.results.iter().fold(0usize, |n, r| n.saturating_add(r.bytes));
let completed: HashSet<_> = state.results.iter().map(|r| Key::from(r.page.request)).collect();
state.desired = requests.iter().enumerate().map(|(priority, req)| (Key::from(*req), priority)).collect();
state.pending =
requests.into_iter().rev().filter(|r| !state.active.contains(&Key::from(*r)) && !completed.contains(&Key::from(*r))).collect();
drop(state);
self.changed.notify_all();
}
pub(super) fn pop(&self) -> Option<RenderRequest> {
let mut state = lock(&self.state);
if state.closed || state.results.len() >= self.max_count || state.bytes >= self.max_bytes {
return None;
}
let req = state.pending.pop()?;
state.active.insert(Key::from(req));
Some(req)
}
/// Skip an obsolete request before entering the next expensive stage. Work already
/// executing cannot be interrupted, but cancelled parsing need not start a raster.
#[cfg(any(test, not(target_arch = "wasm32")))]
pub(super) fn cancel_obsolete(&self, req: RenderRequest) -> bool {
let mut state = lock(&self.state);
let key = Key::from(req);
if state.closed || !state.desired.contains_key(&key) {
state.active.remove(&key);
true
} else {
false
}
}
/// Inline rendering has no completed buffer. Remove its in-flight identity on return.
pub(super) fn finish_inline(&self, req: RenderRequest) {
lock(&self.state).active.remove(&Key::from(req));
}
#[cfg(test)]
pub(super) fn send(&self, page: RenderedPage) -> Result<(), ()> {
self.send_valid(page, None)
}
/// A failed shared parser invalidates its token. Such results are retried by a worker
/// using the pool's fallback parser, including when failure occurs after enqueueing.
#[cfg(any(test, not(target_arch = "wasm32")))]
pub(super) fn send_valid(&self, page: RenderedPage, token: Option<Arc<AtomicBool>>) -> Result<(), ()> {
let key = Key::from(page.request);
let bytes = result_bytes(&page);
let mut state = lock(&self.state);
loop {
if state.closed {
state.active.remove(&key);
return Err(());
}
if !state.desired.contains_key(&key) {
state.active.remove(&key);
return Ok(());
}
if !valid(&token) {
state.retry(page.request);
self.changed.notify_all();
return Ok(());
}
if state.results.len() < self.max_count && (state.results.is_empty() || state.bytes.saturating_add(bytes) <= self.max_bytes) {
state.active.remove(&key);
state.bytes = state.bytes.saturating_add(bytes);
state.results.push_back(Completed { page, bytes, valid: token });
drop(state);
self.changed.notify_all();
return Ok(());
}
// The parser validity token can change independently of this queue. Bound the
// wait so failed generations are released even while the consumer is paused.
state = self.changed.wait_timeout(state, Duration::from_millis(25)).unwrap_or_else(|p| p.into_inner()).0;
}
}
/// A shared-parser timeout may belong to another page holding a cold-load gate.
/// Retry still-current demand after the pool switches permanently to private parsers.
#[cfg(any(test, not(target_arch = "wasm32")))]
pub(super) fn retry(&self, request: RenderRequest) {
lock(&self.state).retry(request);
self.changed.notify_all();
}
/// Watchdog errors contain no page payload. They must never block the UI thread behind
/// a full result buffer; evict/retry one completed request if necessary.
#[cfg(any(test, not(target_arch = "wasm32")))]
pub(super) fn abandon(&self, page: RenderedPage) {
let mut state = lock(&self.state);
let req = page.request;
state.active.remove(&Key::from(req));
if state.closed || !state.desired.contains_key(&Key::from(req)) {
return;
}
let bytes = result_bytes(&page);
while !state.results.is_empty() && (state.results.len() >= self.max_count || state.bytes.saturating_add(bytes) > self.max_bytes) {
if let Some(old) = state.results.pop_back() {
state.bytes = state.bytes.saturating_sub(old.bytes);
state.retry(old.page.request);
}
}
state.bytes = state.bytes.saturating_add(bytes);
state.results.push_front(Completed { page, bytes, valid: None });
drop(state);
self.changed.notify_all();
}
fn close(&self) {
let mut state = lock(&self.state);
state.closed = true;
state.pending.clear();
state.desired.clear();
state.results.clear();
state.bytes = 0;
drop(state);
self.changed.notify_all();
}
}
pub(super) struct ResultReceiver(pub(super) Arc<WorkQueue>);
impl ResultReceiver {
pub(super) fn try_recv(&self) -> Result<RenderedPage, std::sync::mpsc::TryRecvError> {
let mut state = lock(&self.0.state);
let result = state.take();
let closed = state.closed;
drop(state);
self.0.changed.notify_all();
result.ok_or(if closed { std::sync::mpsc::TryRecvError::Disconnected } else { std::sync::mpsc::TryRecvError::Empty })
}
#[cfg(test)]
pub(super) fn recv_timeout(&self, timeout: Duration) -> Result<RenderedPage, std::sync::mpsc::RecvTimeoutError> {
let start = std::time::Instant::now();
loop {
if let Ok(page) = self.try_recv() {
return Ok(page);
}
if start.elapsed() >= timeout {
return Err(std::sync::mpsc::RecvTimeoutError::Timeout);
}
std::thread::sleep(Duration::from_millis(2));
}
}
}
impl Drop for ResultReceiver {
fn drop(&mut self) {
self.0.close();
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::Pixels;
fn req(page: usize) -> RenderRequest {
RenderRequest { page, scale: 1.0, ..Default::default() }
}
fn page(req: RenderRequest, bytes: usize) -> RenderedPage {
RenderedPage { request: req, width: 1, height: 1, rgba: Pixels::from(vec![0u32; bytes / 4]), error: None, text: None, millis: 0 }
}
#[test]
fn replacing_demands_deduplicates_active_and_completed_then_allows_eviction_retry() {
let q = Arc::new(WorkQueue::default());
let recv = ResultReceiver(q.clone());
q.replace(vec![req(0), req(0), req(1)]);
assert_eq!(q.pop(), Some(req(0)));
q.replace(vec![req(0), req(1)]);
assert_eq!(q.pop(), Some(req(1)));
q.send(page(req(0), 4)).unwrap();
q.replace(vec![req(0), req(1)]);
assert!(q.pop().is_none());
assert_eq!(recv.try_recv().unwrap().request, req(0));
q.replace(vec![req(1)]);
q.send(page(req(1), 4)).unwrap();
q.replace(vec![req(0)]);
assert!(recv.try_recv().is_err(), "obsolete completion released before delivery");
assert_eq!(q.pop(), Some(req(0)), "evicted UI textures can request the same key again");
}
#[test]
fn request_identity_preserves_distinct_work_and_deduplicates_nan_scales() {
// Keep #400's bitwise scale identity: NaN must match itself while rendering and
// awaiting delivery, without conflating different pages, kinds, tiles or tags.
let q = Arc::new(WorkQueue::default());
let recv = ResultReceiver(q.clone());
let base = RenderRequest { scale: f32::from_bits(0x7fc0_0001), ..req(0) };
let requests = [
base,
RenderRequest { page: 1, ..base },
RenderRequest { kind: RequestKind::Text, ..base },
RenderRequest { tile: Some(Tile { x: 0, y: 0, w: 1, h: 1 }), ..base },
RenderRequest { scale: f32::from_bits(0x7fc0_0002), ..base },
RenderRequest { tag: 1, ..base },
];
let repeated = || requests.into_iter().flat_map(|r| [r, r]).collect();
q.replace(repeated());
for request in requests {
assert_eq!(q.pop().map(Key::from), Some(Key::from(request)));
}
q.replace(repeated());
assert!(q.pop().is_none(), "active requests are not rendered twice");
for request in requests {
q.send(page(request, 4)).unwrap();
}
q.replace(repeated());
assert!(q.pop().is_none(), "completed requests remain deduplicated until delivery");
for request in requests {
assert_eq!(Key::from(recv.try_recv().unwrap().request), Key::from(request));
}
assert!(recv.try_recv().is_err());
q.replace(vec![base]);
assert_eq!(q.pop().map(Key::from), Some(Key::from(base)), "delivery releases even a NaN request for a later retry");
}
#[test]
fn returning_to_active_work_reuses_it_but_cancellation_allows_a_fresh_request() {
let q = Arc::new(WorkQueue::default());
let recv = ResultReceiver(q.clone());
q.replace(vec![req(0)]);
assert_eq!(q.pop(), Some(req(0)));
q.replace(Vec::new());
q.replace(vec![req(0)]);
assert!(q.pop().is_none(), "returning to a page reuses its in-flight render");
q.send(page(req(0), 4)).unwrap();
assert_eq!(recv.try_recv().unwrap().request, req(0));
assert!(recv.try_recv().is_err());
q.replace(vec![req(0)]);
assert_eq!(q.pop(), Some(req(0)));
q.replace(Vec::new());
assert!(q.cancel_obsolete(req(0)), "a worker can cancel before rasterization");
q.replace(vec![req(0)]);
assert_eq!(q.pop(), Some(req(0)), "a cancelled render must not suppress later demand");
}
#[test]
fn delivery_uses_current_priority_for_buffered_and_later_completions() {
let q = Arc::new(WorkQueue::default());
let recv = ResultReceiver(q.clone());
let text = RenderRequest { kind: RequestKind::Text, ..req(0) };
let thumbnail = RenderRequest { tag: 1 << 63, ..req(1) };
let visible = req(2);
q.replace(vec![text, thumbnail, visible]);
for _ in 0..3 {
assert!(q.pop().is_some());
}
q.send(page(text, 4)).unwrap();
q.send(page(thumbnail, 4)).unwrap();
// A newly visible main page takes priority even though the other work is still
// desired, already completed, and was requested first in an earlier frame.
q.replace(vec![visible, thumbnail, text]);
q.send(page(visible, 4)).unwrap();
assert_eq!(recv.try_recv().unwrap().request, visible);
assert_eq!(recv.try_recv().unwrap().request, thumbnail);
assert_eq!(recv.try_recv().unwrap().request, text);
assert!(recv.try_recv().is_err());
assert_eq!(lock(&q.state).bytes, 0);
assert!(q.pop().is_none(), "priority changes must not duplicate completed work");
}
#[test]
fn invalid_background_completion_retries_after_pending_visible_work() {
let q = Arc::new(WorkQueue::default());
let recv = ResultReceiver(q.clone());
let token = Arc::new(AtomicBool::new(true));
let text = RenderRequest { kind: RequestKind::Text, ..req(0) };
let visible = req(1);
q.replace(vec![text]);
assert_eq!(q.pop(), Some(text));
q.send_valid(page(text, 4), Some(token.clone())).unwrap();
q.replace(vec![visible, text]);
token.store(false, Ordering::Release);
assert!(recv.try_recv().is_err());
assert_eq!(q.pop(), Some(visible));
assert_eq!(q.pop(), Some(text));
assert!(q.pop().is_none());
}
#[test]
fn watchdog_eviction_retries_background_work_after_visible_work() {
let q = Arc::new(WorkQueue { max_count: 1, ..Default::default() });
let recv = ResultReceiver(q.clone());
let text = RenderRequest { kind: RequestKind::Text, ..req(0) };
let stuck = req(1);
let visible = req(2);
q.replace(vec![text, stuck]);
assert_eq!(q.pop(), Some(text));
assert_eq!(q.pop(), Some(stuck));
q.send(page(text, 4)).unwrap();
q.replace(vec![visible, text, stuck]);
let mut error = page(stuck, 0);
error.error = Some("watchdog".into());
q.abandon(error);
assert_eq!(recv.try_recv().unwrap().request, stuck);
assert_eq!(q.pop(), Some(visible));
assert_eq!(q.pop(), Some(text));
assert!(q.pop().is_none());
}
#[test]
fn stalled_consumer_bounds_storage_and_drop_unblocks_publishers() {
let q = Arc::new(WorkQueue { max_bytes: 1024, max_count: 2, ..Default::default() });
let recv = ResultReceiver(q.clone());
q.replace(vec![req(0), req(1), req(2)]);
assert!(q.pop().is_some());
assert!(q.pop().is_some());
q.send(page(req(0), 700)).unwrap();
let blocked = q.clone();
let (tx, rx) = std::sync::mpsc::channel();
let worker = std::thread::spawn(move || {
tx.send(blocked.send(page(req(1), 700))).unwrap();
});
assert!(rx.recv_timeout(Duration::from_millis(30)).is_err());
assert!(lock(&q.state).bytes <= 1024);
drop(recv);
assert!(rx.recv_timeout(Duration::from_secs(2)).unwrap().is_err());
worker.join().unwrap();
}
#[test]
fn oversized_completion_is_admitted_alone_and_cancellation_releases_waiters() {
let q = Arc::new(WorkQueue { max_bytes: 512, max_count: 2, ..Default::default() });
let recv = ResultReceiver(q.clone());
q.replace(vec![req(0), req(1)]);
q.pop();
q.pop();
q.send(page(req(0), 1024)).unwrap();
let producer = q.clone();
let (tx, rx) = std::sync::mpsc::channel();
let worker = std::thread::spawn(move || {
tx.send(producer.send(page(req(1), 1024))).unwrap();
});
assert!(rx.recv_timeout(Duration::from_millis(30)).is_err());
assert_eq!(lock(&q.state).results.len(), 1);
q.replace(Vec::new());
assert!(rx.recv_timeout(Duration::from_secs(2)).unwrap().is_ok());
worker.join().unwrap();
assert!(recv.try_recv().is_err());
assert_eq!(lock(&q.state).bytes, 0);
}
#[test]
fn invalid_parser_generations_are_retried_before_or_after_publication() {
let q = Arc::new(WorkQueue::default());
let recv = ResultReceiver(q.clone());
let token = Arc::new(AtomicBool::new(true));
q.replace(vec![req(0)]);
q.pop();
q.send_valid(page(req(0), 4), Some(token.clone())).unwrap();
token.store(false, Ordering::Release);
assert!(recv.try_recv().is_err());
assert_eq!(q.pop(), Some(req(0)));
q.send_valid(page(req(0), 4), Some(token)).unwrap();
assert_eq!(q.pop(), Some(req(0)), "invalid pre-publication result retries too");
q.send(page(req(0), 4)).unwrap();
assert!(recv.try_recv().unwrap().error.is_none());
}
}

239
crates/render/src/structure.rs Normal file
View File

@ -0,0 +1,239 @@
//! Read-only structure inspection backed by the lazy COS reader. Keep the existing
//! inspector's lopdf object vocabulary while loading only the objects its queries use.
use std::collections::{BTreeMap, HashSet};
use std::sync::{
Arc, OnceLock,
atomic::{AtomicBool, Ordering},
};
use lopdf::{Dictionary, Error, Object, ObjectId, Result};
use pdfcraft_cos::{Document, Object as CosObject};
pub(crate) trait Structure {
fn trailer(&self) -> &Dictionary;
fn get_object(&self, id: ObjectId) -> Result<&Object>;
fn encrypted(&self) -> bool;
fn dereference<'a>(&'a self, mut object: &'a Object) -> Result<(Option<ObjectId>, &'a Object)> {
let mut id = None;
for _ in 0..128 {
match object {
Object::Reference(reference) => {
id = Some(*reference);
object = self.get_object(*reference)?;
}
_ => return Ok((id, object)),
}
}
Err(Error::ReferenceLimit)
}
fn get_dictionary(&self, id: ObjectId) -> Result<&Dictionary> {
self.get_object(id).and_then(|o| self.dereference(o)).and_then(|(_, o)| o.as_dict())
}
fn catalog(&self) -> Result<&Dictionary> {
self.trailer().get(b"Root").and_then(Object::as_reference).and_then(|id| self.get_dictionary(id))
}
fn get_pages(&self) -> BTreeMap<u32, ObjectId> {
let mut pages = BTreeMap::new();
let Ok(root) = self.catalog().and_then(|c| c.get(b"Pages")).and_then(Object::as_reference) else { return pages };
let mut pending = vec![(root, 0usize)];
let mut seen = HashSet::new();
while let Some((id, depth)) = pending.pop() {
if depth > 256 || !seen.insert(id) {
continue;
}
let Ok(dict) = self.get_dictionary(id) else { continue };
match dict.get_type() {
Ok(b"Page") => {
// Page numbers are 1-based u32s; a tree past u32::MAX pages stops there.
let Some(number) = u32::try_from(pages.len()).ok().and_then(|n| n.checked_add(1)) else { return pages };
pages.insert(number, id);
}
Ok(b"Pages") => {
if let Ok(kids) = dict.get(b"Kids").and_then(|o| self.dereference(o)).and_then(|(_, o)| o.as_array()) {
pending.extend(kids.iter().rev().filter_map(|k| k.as_reference().ok()).map(|id| (id, depth + 1)));
}
}
_ => {}
}
}
pages
}
// Match lopdf's font-resource precedence: a page's direct resources, then its
// indirect resources and ancestors. Stop cycles and excessive inheritance depth.
fn get_page_fonts(&self, page: ObjectId) -> Result<BTreeMap<Vec<u8>, &Dictionary>> {
let mut resources = Vec::new();
let mut current = Some(page);
let mut seen = HashSet::new();
while let Some(id) = current {
if !seen.insert(id) {
return Err(Error::ReferenceCycle(id));
}
if seen.len() > 100 {
return Err(Error::RecursionLimit);
}
let dict = self.get_dictionary(id)?;
if let Ok(resource) = dict.get(b"Resources") {
match resource {
Object::Dictionary(dict) if id == page => resources.push(dict),
Object::Reference(id) => {
if let Ok(dict) = self.get_dictionary(*id) {
resources.push(dict);
}
}
_ => {}
}
}
current = dict.get(b"Parent").and_then(Object::as_reference).ok();
}
let mut fonts = BTreeMap::new();
for resources in resources {
if let Ok(dict) = resources.get(b"Font").and_then(|o| self.dereference(o)).and_then(|(_, o)| o.as_dict()) {
for (name, font) in dict.iter() {
if let Ok(font) = self.dereference(font).and_then(|(_, o)| o.as_dict()) {
fonts.entry(name.clone()).or_insert(font);
}
}
}
}
Ok(fonts)
}
}
impl Structure for lopdf::Document {
fn trailer(&self) -> &Dictionary {
&self.trailer
}
fn get_object(&self, id: ObjectId) -> Result<&Object> {
self.get_object(id)
}
fn encrypted(&self) -> bool {
self.is_encrypted() || self.was_encrypted() || self.trailer.has(b"Encrypt")
}
fn get_pages(&self) -> BTreeMap<u32, ObjectId> {
self.get_pages()
}
fn get_page_fonts(&self, page: ObjectId) -> Result<BTreeMap<Vec<u8>, &Dictionary>> {
self.get_page_fonts(page)
}
}
type Slot = (u32, OnceLock<Option<Box<Object>>>);
pub(crate) struct LazyStructure {
source: Document,
trailer: Dictionary,
// Sorted slots keep returned references stable, without allocating by the largest
// input object number (which can be sparse or hostile). Only queried objects fill slots.
objects: Vec<Slot>,
failed: AtomicBool,
}
impl LazyStructure {
pub(crate) fn new(bytes: Arc<Vec<u8>>, password: Option<&str>) -> std::result::Result<Self, String> {
let source = Document::open_with_stream_limit(bytes, password, 256 << 20).map_err(|e| e.to_string())?;
if !source.repair_log().is_empty() {
return Err("the structure requires compatibility repair".into());
}
let trailer = convert_dict(source.trailer(), 0).map_err(|e| e.to_string())?;
let objects = source.object_numbers().into_iter().map(|n| (n, OnceLock::new())).collect();
Ok(Self { source, trailer, objects, failed: AtomicBool::new(false) })
}
pub(crate) fn failed(&self) -> bool {
self.failed.load(Ordering::Relaxed)
}
}
impl Structure for LazyStructure {
fn trailer(&self) -> &Dictionary {
&self.trailer
}
fn encrypted(&self) -> bool {
self.source.security().is_some() || self.trailer.has(b"Encrypt")
}
fn get_object(&self, id: ObjectId) -> Result<&Object> {
let missing = || Error::ObjectNotFound(id);
let slot = self.objects.binary_search_by_key(&id.0, |(n, _)| *n).ok().and_then(|i| self.objects.get(i)).ok_or_else(missing)?;
if self.source.generation(id.0) != id.1 {
return Err(missing());
}
slot.1
.get_or_init(|| {
let result = self.source.try_get(id.0).ok().and_then(|o| convert(&o, 0).ok()).map(Box::new);
if result.is_none() {
self.failed.store(true, Ordering::Relaxed);
}
result
})
.as_deref()
.ok_or_else(missing)
}
}
fn convert_dict(dict: &pdfcraft_cos::Dict, depth: usize) -> Result<Dictionary> {
if depth > 100 {
return Err(Error::RecursionLimit);
}
dict.iter().map(|(key, value)| Ok((key.clone(), convert(value, depth + 1)?))).collect::<Result<Vec<_>>>().map(|items| items.into_iter().collect())
}
fn convert(object: &CosObject, depth: usize) -> Result<Object> {
if depth > 100 {
return Err(Error::RecursionLimit);
}
Ok(match object {
CosObject::Null => Object::Null,
CosObject::Bool(value) => Object::Boolean(*value),
CosObject::Int(value) => Object::Integer(*value),
CosObject::Real(value) => Object::Real(*value as f32),
CosObject::Name(value) => Object::Name(value.clone()),
CosObject::String(value) => {
Object::String(value.bytes.clone(), if value.hex { lopdf::StringFormat::Hexadecimal } else { lopdf::StringFormat::Literal })
}
CosObject::Ref(value) => Object::Reference((value.num, value.generation)),
CosObject::Array(value) => Object::Array(value.iter().map(|o| convert(o, depth + 1)).collect::<Result<_>>()?),
CosObject::Dict(value) => Object::Dictionary(convert_dict(value, depth + 1)?),
CosObject::Stream(value) => Object::Stream(lopdf::Stream::new(convert_dict(&value.dict, depth + 1)?, value.raw.to_vec())),
})
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn inspection_loads_only_requested_objects_in_a_compressed_document() {
let mut source = lopdf::Document::with_version("1.7");
let catalog = source.add_object(lopdf::dictionary! { "Type" => "Catalog", "Pages" => (2, 0) });
source.add_object(lopdf::dictionary! { "Type" => "Pages", "Kids" => vec![Object::Reference((3, 0))], "Count" => 1 });
source
.add_object(lopdf::dictionary! { "Type" => "Page", "Parent" => (2, 0), "MediaBox" => vec![0.into(), 0.into(), 100.into(), 100.into()] });
source.trailer.set("Root", catalog);
let unused: Vec<_> = (0..300).map(|i| Object::Reference(source.add_object(lopdf::dictionary! { "Unused" => i }))).collect();
// Keep these objects reachable when the full writer removes orphan objects,
// through a catalog extension that the panel inspector has no reason to read.
source.get_dictionary_mut(catalog).unwrap().set("Uninspected", unused);
let mut bytes = Vec::new();
source.save_to(&mut bytes).unwrap();
let cos = Document::open(Arc::new(bytes)).unwrap();
let compressed = pdfcraft_cos::write_full(&cos, &pdfcraft_cos::SaveOptions::default()).unwrap();
let expected = lopdf::Document::load_mem(&compressed).unwrap().get_pages();
let view = LazyStructure::new(Arc::new(compressed), None).unwrap();
assert_eq!(view.get_pages().len(), 1);
assert_eq!(view.get_pages(), expected);
assert!(!view.failed());
assert!(
view.objects.iter().filter(|(_, slot)| slot.get().is_some()).count() <= 3,
"unused objects must not be materialized to inspect a page tree"
);
let unused = view.catalog().unwrap().get(b"Uninspected").unwrap().as_array().unwrap();
let last = unused.last().unwrap().as_reference().unwrap();
assert!(matches!(view.get_dictionary(last).unwrap().get(b"Unused"), Ok(Object::Integer(299))));
}
}

118
crates/sign/src/pdf.rs
View File

@ -23,6 +23,10 @@ use crate::keys::DigestAlg;
use crate::pkcs12::DigitalId;
use crate::x509::{Certificate, build_chain};
#[cfg(test)]
#[path = "pdf_discovery_tests.rs"]
mod discovery_tests;
/// Digests of signed byte ranges (and parsed signed revisions), kept across revalidations of
/// one document: incremental edits only append to the file, so the signed bytes and their
/// digest don't change, and rehashing a large file on every edit was the expensive part of
@ -34,6 +38,50 @@ pub struct DigestCache {
map: std::sync::Mutex<HashMap<RangeKey, Vec<u8>>>,
/// Signed revisions opened for change classification, by length and fingerprint.
revisions: std::sync::Mutex<HashMap<(usize, [u8; 32]), Document>>,
/// Candidate identities only: validation still uses current bytes and current trust.
discovery: std::sync::Mutex<Option<TimestampCandidates>>,
#[cfg(test)]
discovery_scans: std::sync::atomic::AtomicUsize,
}
/// Keep only the most recently visited source/overlay. In particular, this must not retain
/// a Document, its bytes, or parsed signature dictionaries across edits and saves.
#[derive(Debug)]
struct TimestampCandidates {
source: pdfcraft_cos::SourceIdentity,
modified: Vec<u32>,
candidates: Vec<ObjRef>,
}
const DISCOVERY_CACHE_LIMIT: usize = 4096;
fn is_document_timestamp(object: &Object) -> bool {
object.as_dict().is_some_and(|d| d.name(b"Type") == Some(b"Sig") && d.name(b"SubFilter") == Some(b"ETSI.RFC3161") && d.contains(b"ByteRange"))
}
/// Only dictionary/null edits have a local effect on candidate membership. Changes to an
/// object stream or a scalar used as an indirect /Length can affect unedited objects too.
/// Checking both the original and current objects also covers freeing those dependencies.
/// Unsupported or unreadable edits deliberately fall back to the complete bounded scan.
fn changed_timestamp_candidates(doc: &Document, numbers: &[u32]) -> Option<Vec<ObjRef>> {
for (reference, object) in doc.scan_original_objects_subset(numbers.to_vec()) {
match object.ok()?.as_ref() {
Object::Dict(_) => {}
Object::Null if matches!(doc.xref_entry(reference.num), None | Some(pdfcraft_cos::XrefEntry::Free { .. })) => {}
_ => return None,
}
}
let mut candidates = Vec::new();
for (reference, object) in doc.scan_objects_subset(numbers.to_vec()) {
let object = object.ok()?;
if !matches!(object.as_ref(), Object::Dict(_) | Object::Null) {
return None;
}
if is_document_timestamp(&object) {
candidates.push(reference);
}
}
Some(candidates)
}
/// (end of first range, start of second, end of second, algorithm, fingerprint).
@ -53,6 +101,61 @@ fn fingerprint(covered: &[u8]) -> [u8; 32] {
}
impl DigestCache {
fn timestamp_candidates(&self, doc: &Document) -> Vec<ObjRef> {
let mut slot = self.discovery.lock().ok();
let previous = slot.as_mut().and_then(|slot| slot.take());
let source = doc.source_identity();
let modified = doc.modified_objects();
// Reconstructed/unknown input and output-security changes take the original path.
let eligible = slot.is_some()
&& doc.repair_log().is_empty()
&& !doc.revisions().is_empty()
&& !doc.encryption_changed()
&& modified.len() <= DISCOVERY_CACHE_LIMIT;
if eligible && let Some(previous) = previous.filter(|previous| previous.source == source) {
// Undo/redo can remove overlays as well as add them. Revisit the union so an
// original candidate replaced or freed in the last snapshot can reappear.
let mut changed = previous.modified;
changed.extend_from_slice(&modified);
changed.sort_unstable();
changed.dedup();
if changed.len() <= DISCOVERY_CACHE_LIMIT
&& let Some(mut candidates) = changed_timestamp_candidates(doc, &changed)
{
candidates.extend(previous.candidates.into_iter().filter(|r| changed.binary_search(&r.num).is_err()));
candidates.sort_unstable_by_key(|r| r.num);
if candidates.len() <= DISCOVERY_CACHE_LIMIT
&& let Some(slot) = slot.as_mut()
{
**slot = Some(TimestampCandidates { source, modified, candidates: candidates.clone() });
}
return candidates;
}
}
#[cfg(test)]
self.discovery_scans.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
let mut complete = true;
let candidates: Vec<_> = doc
.scan_objects_checked()
.filter_map(|(reference, object)| match object {
Ok(object) => is_document_timestamp(&object).then_some(reference),
Err(_) => {
complete = false;
None
}
})
.collect();
if eligible
&& complete
&& candidates.len() <= DISCOVERY_CACHE_LIMIT
&& changed_timestamp_candidates(doc, &modified).is_some()
&& let Some(slot) = slot.as_mut()
{
**slot = Some(TimestampCandidates { source, modified, candidates: candidates.clone() });
}
candidates
}
/// The signed revision `bytes[..end]`, parsed (and kept).
fn revision(&self, bytes: &[u8], end: usize) -> Option<Document> {
let key = (end, fingerprint(&bytes[..end]));
@ -261,7 +364,8 @@ pub fn list(doc: &Document, bytes: &[u8], trust: &TrustStore) -> Vec<SignatureIn
list_cached(doc, bytes, trust, &DigestCache::default())
}
/// [`list`], reusing digests from `cache`.
/// [`list`], reusing digests and standalone-timestamp discovery from `cache`. Field/widget
/// structure and validation against `bytes` and `trust` are read again on every call.
pub fn list_cached(doc: &Document, bytes: &[u8], trust: &TrustStore, cache: &DigestCache) -> Vec<SignatureInfo> {
let pages = annot_pages(doc);
let mut out = Vec::new();
@ -311,14 +415,14 @@ pub fn list_cached(doc: &Document, bytes: &[u8], trust: &TrustStore, cache: &Dig
out.push(info);
}
// Standalone document timestamps (ISO 32000-2 §12.8.2.2) live outside AcroForm fields.
for num in doc.object_numbers() {
let r = ObjRef { num, generation: doc.generation(num) };
let candidates = cache.timestamp_candidates(doc).into_iter().map(|r| r.num).collect();
for (r, o) in doc.scan_objects_subset(candidates) {
if field_values.contains(&r) {
continue;
}
let o = doc.get(r);
let Ok(o) = o else { continue };
let Some(d) = o.as_dict() else { continue };
if d.name(b"Type") != Some(b"Sig") || d.name(b"SubFilter") != Some(b"ETSI.RFC3161") || !d.contains(b"ByteRange") {
if !is_document_timestamp(&o) {
continue;
}
let mut info = SignatureInfo {
@ -707,9 +811,7 @@ pub(crate) fn signature_contents(doc: &Document) -> Vec<Vec<u8>> {
}
}
}
for num in doc.object_numbers() {
let r = ObjRef { num, generation: doc.generation(num) };
let o = doc.get(r);
for (r, o) in doc.scan_objects() {
let Some(d) = o.as_dict() else { continue };
if d.name(b"Type") == Some(b"Sig") && d.name(b"SubFilter") == Some(b"ETSI.RFC3161") {
add(d, Some(r));

352
crates/sign/src/pdf_discovery_tests.rs Normal file
View File

@ -0,0 +1,352 @@
use super::*;
use pdfcraft_cos::{Algorithm, NewEncryption, XrefEntry, write_full, write_incremental};
use std::sync::atomic::Ordering;
const STAMP: &str = "<< /Type /Sig /SubFilter /ETSI.RFC3161 /ByteRange [0 1 2 1] >>";
fn fixture(extra: &[&str]) -> Document {
let mut objects = vec!["<< /Type /Catalog /Pages 2 0 R /Unknown 3 0 R >>", "<< /Type /Pages /Kids [] /Count 0 >>"];
objects.extend_from_slice(extra);
let mut bytes = b"%PDF-1.7\n".to_vec();
let mut offsets = Vec::new();
for (i, object) in objects.iter().enumerate() {
offsets.push(bytes.len());
bytes.extend_from_slice(format!("{} 0 obj\n{object}\nendobj\n", i + 1).as_bytes());
}
let xref = bytes.len();
bytes.extend_from_slice(format!("xref\n0 {}\n0000000000 65535 f \n", objects.len() + 1).as_bytes());
for offset in offsets {
bytes.extend_from_slice(format!("{offset:010} 00000 n \n").as_bytes());
}
bytes.extend_from_slice(format!("trailer\n<< /Size {} /Root 1 0 R >>\nstartxref\n{xref}\n%%EOF\n", objects.len() + 1).as_bytes());
Document::open(Arc::new(bytes)).unwrap()
}
fn stamp() -> Dict {
pdfcraft_cos::Lexer::new(STAMP.as_bytes(), 0).object().unwrap().as_dict().unwrap().clone()
}
fn scans(cache: &DigestCache) -> usize {
cache.discovery_scans.load(Ordering::Relaxed)
}
#[test]
fn unchanged_source_and_clones_skip_the_full_discovery_scan() {
let doc = fixture(&[STAMP]);
let cache = DigestCache::default();
let expected = vec![ObjRef::new(3, 0)];
assert_eq!(cache.timestamp_candidates(&doc), expected);
assert_eq!(cache.timestamp_candidates(&doc), expected);
assert_eq!(cache.timestamp_candidates(&doc.clone()), expected);
assert_eq!(scans(&cache), 1);
let listed = list_cached(&doc, doc.bytes(), &TrustStore::default(), &cache);
assert_eq!(listed.len(), 1, "a timestamp needs no AcroForm");
assert!(listed[0].doc_timestamp);
assert_eq!(scans(&cache), 1);
}
#[test]
fn empty_discovery_is_cached_until_an_unknown_dictionary_becomes_a_timestamp() {
let mut doc = fixture(&["<< /Unknown /Preserved >>"]);
let cache = DigestCache::default();
assert!(cache.timestamp_candidates(&doc).is_empty());
assert!(cache.timestamp_candidates(&doc).is_empty());
doc.update_dict(ObjRef::new(2, 0), |d| d.set(b"Rotate".to_vec(), Object::Int(90))).unwrap();
assert!(cache.timestamp_candidates(&doc).is_empty());
doc.set(ObjRef::new(3, 0), stamp());
assert_eq!(cache.timestamp_candidates(&doc), vec![ObjRef::new(3, 0)]);
assert_eq!(scans(&cache), 1);
}
#[test]
fn dictionary_edits_additions_frees_and_undo_redo_reconcile_candidates() {
let original = fixture(&[STAMP, "<< /Unknown /Preserved >>"]);
let cache = DigestCache::default();
let old = ObjRef::new(3, 0);
let unknown = ObjRef::new(4, 0);
assert_eq!(cache.timestamp_candidates(&original), vec![old]);
let mut edited = original.clone();
edited.update_dict(ObjRef::new(2, 0), |d| d.set(b"Rotate".to_vec(), Object::Int(90))).unwrap();
assert_eq!(cache.timestamp_candidates(&edited), vec![old]);
edited
.update_dict(old, |d| {
d.remove(b"ByteRange");
})
.unwrap();
edited.set(unknown, stamp());
let added = edited.add(stamp());
assert_eq!(cache.timestamp_candidates(&edited), vec![unknown, added]);
let before_free = edited.clone();
edited.free(unknown);
edited.free(added);
assert!(cache.timestamp_candidates(&edited).is_empty());
assert_eq!(cache.timestamp_candidates(&before_free), vec![unknown, added]);
assert_eq!(cache.timestamp_candidates(&original), vec![old]);
assert!(cache.timestamp_candidates(&edited).is_empty());
assert_eq!(scans(&cache), 1);
}
#[test]
fn first_discovery_on_an_edited_snapshot_does_not_hide_original_candidates() {
let original = fixture(&[STAMP]);
let mut edited = original.clone();
edited.free(ObjRef::new(3, 0));
let cache = DigestCache::default();
assert!(cache.timestamp_candidates(&edited).is_empty());
assert_eq!(cache.timestamp_candidates(&original), vec![ObjRef::new(3, 0)]);
assert_eq!(scans(&cache), 1);
}
#[test]
fn field_membership_is_refreshed_even_when_discovery_is_cached() {
let mut doc = fixture(&[STAMP]);
let cache = DigestCache::default();
assert!(list_cached(&doc, doc.bytes(), &TrustStore::default(), &cache)[0].doc_timestamp);
let mut field = Dict::new();
field.set(b"FT".to_vec(), Object::name("Sig"));
field.set(b"T".to_vec(), Object::String(PdfString::literal("Approval")));
field.set(b"V".to_vec(), Object::Ref(ObjRef::new(3, 0)));
let field = doc.add(field);
let mut form = Dict::new();
form.set(b"Fields".to_vec(), Object::Array(vec![Object::Ref(field)]));
doc.update_dict(doc.root().unwrap(), |d| d.set(b"AcroForm".to_vec(), Object::Dict(form))).unwrap();
let listed = list_cached(&doc, doc.bytes(), &TrustStore::default(), &cache);
assert_eq!(listed.len(), 1, "field values must not be duplicated as standalone timestamps");
assert_eq!(listed[0].field, "Approval");
assert!(!listed[0].doc_timestamp);
doc.update_dict(doc.root().unwrap(), |d| {
d.remove(b"AcroForm");
})
.unwrap();
assert!(list_cached(&doc, doc.bytes(), &TrustStore::default(), &cache)[0].doc_timestamp);
assert_eq!(scans(&cache), 1);
}
#[test]
fn reopened_incremental_and_renumbered_sources_get_fresh_discovery() {
let mut doc = fixture(&["<< /Ignored true >>", STAMP]);
doc.update_dict(doc.root().unwrap(), |d| d.set(b"Unknown".to_vec(), Object::Ref(ObjRef::new(4, 0)))).unwrap();
let cache = DigestCache::default();
assert_eq!(cache.timestamp_candidates(&doc), vec![ObjRef::new(4, 0)]);
let incremental = Document::open(Arc::new(write_incremental(&doc, &SaveOptions::default()).unwrap())).unwrap();
assert_ne!(doc.source_identity(), incremental.source_identity());
assert_eq!(cache.timestamp_candidates(&incremental), vec![ObjRef::new(4, 0)]);
let full = Document::open(Arc::new(write_full(&doc, &SaveOptions::default()).unwrap())).unwrap();
let renumbered = full.get(full.root().unwrap()).as_dict().unwrap().reference(b"Unknown").unwrap();
assert_ne!(renumbered.num, 4);
assert_eq!(cache.timestamp_candidates(&full), vec![renumbered]);
let reopened_same_bytes = Document::open(full.bytes().clone()).unwrap();
assert_ne!(full.source_identity(), reopened_same_bytes.source_identity());
assert_eq!(cache.timestamp_candidates(&reopened_same_bytes), vec![renumbered]);
assert_eq!(scans(&cache), 4);
}
#[test]
fn discovery_retains_neither_document_bytes_nor_old_sources() {
let cache = DigestCache::default();
let doc = fixture(&[STAMP]);
let bytes = Arc::downgrade(doc.bytes());
let source = doc.source_identity();
cache.timestamp_candidates(&doc);
drop(doc);
assert!(bytes.upgrade().is_none());
for _ in 0..64 {
let next = fixture(&["<< /Unknown true >>"]);
assert_ne!(source, next.source_identity());
assert!(cache.timestamp_candidates(&next).is_empty());
}
assert_eq!(scans(&cache), 65);
}
#[test]
fn reconstructed_and_unreadable_objects_are_not_cached_as_complete() {
let doc = fixture(&[STAMP]);
let mut damaged = doc.bytes().as_ref().clone();
damaged[0] = b'!';
let repaired = Document::open(Arc::new(damaged)).unwrap();
assert!(!repaired.repair_log().is_empty());
let cache = DigestCache::default();
for _ in 0..2 {
assert_eq!(cache.timestamp_candidates(&repaired), vec![ObjRef::new(3, 0)]);
}
// A single invalid keyword immediately before endobj is deliberately recovered as
// Null by parse_indirect. Use a stream without endstream to exercise a real read error.
let unreadable = fixture(&[STAMP, "<< /Length 0 >>\nstream\nunterminated"]);
assert!(unreadable.repair_log().is_empty());
assert!(unreadable.try_get(4).is_err());
assert!(unreadable.scan_objects_checked().any(|(r, object)| r.num == 4 && object.is_err()));
for _ in 0..2 {
assert_eq!(cache.timestamp_candidates(&unreadable), vec![ObjRef::new(3, 0)]);
assert!(cache.discovery.lock().unwrap().is_none(), "incomplete discovery must not be published");
}
assert_eq!(scans(&cache), 4);
assert!(cache.discovery.lock().unwrap().is_none());
}
#[test]
fn indirect_length_edits_and_frees_take_the_full_scan_fallback() {
let original = fixture(&[STAMP, "<< /Length 5 0 R >>\nstream\nabc\nendstream", "3"]);
let cache = DigestCache::default();
assert_eq!(cache.timestamp_candidates(&original), vec![ObjRef::new(3, 0)]);
let mut edited = original.clone();
edited.set(ObjRef::new(5, 0), Object::Int(0));
assert_eq!(cache.timestamp_candidates(&edited), vec![ObjRef::new(3, 0)]);
assert!(cache.discovery.lock().unwrap().is_none());
edited.free(ObjRef::new(5, 0));
assert_eq!(cache.timestamp_candidates(&edited), vec![ObjRef::new(3, 0)]);
assert_eq!(cache.timestamp_candidates(&original), vec![ObjRef::new(3, 0)]);
assert_eq!(scans(&cache), 4);
}
#[test]
fn object_stream_changes_cannot_reuse_old_candidate_membership() {
let original = fixture(&[STAMP]);
let original = Document::open(Arc::new(write_full(&original, &SaveOptions::default()).unwrap())).unwrap();
let cache = DigestCache::default();
let candidates = cache.timestamp_candidates(&original);
assert_eq!(candidates.len(), 1);
let Some(XrefEntry::InStream { stream, .. }) = original.xref_entry(candidates[0].num) else { panic!("fixture must pack the timestamp") };
let stream_ref = ObjRef::new(stream, original.generation(stream));
let old_stream = original.get(stream_ref);
let Object::Stream(old_stream) = old_stream.as_ref() else { panic!("missing object stream") };
let mut decoded = old_stream.decoded().unwrap();
let at = decoded.windows(b"ETSI.RFC3161".len()).position(|s| s == b"ETSI.RFC3161").unwrap();
decoded[at] = b'X';
let mut edited = original.clone();
edited.set(stream_ref, Stream::flate(old_stream.dict.clone(), &decoded));
assert!(cache.timestamp_candidates(&edited).is_empty());
assert!(cache.discovery.lock().unwrap().is_none());
edited.free(stream_ref);
assert!(cache.timestamp_candidates(&edited).is_empty());
assert_eq!(cache.timestamp_candidates(&original), candidates);
assert_eq!(scans(&cache), 4);
}
#[test]
fn decode_limits_and_password_contexts_cannot_share_discovery() {
let mut doc = fixture(&[STAMP]);
doc.set_encryption(&NewEncryption {
algorithm: Algorithm::Aes128,
user_password: "user",
owner_password: "owner",
permissions: -1,
encrypt_metadata: true,
seed: [29; 32],
})
.unwrap();
let bytes = Arc::new(write_full(&doc, &SaveOptions::default()).unwrap());
assert!(Document::open_with_password(bytes.clone(), Some("wrong")).is_err());
let user = Document::open_with_password(bytes.clone(), Some("user")).unwrap();
let owner = Document::open_with_password(bytes.clone(), Some("owner")).unwrap();
let limited = Document::open_with_stream_limit(bytes, Some("owner"), 1024 * 1024).unwrap();
assert_ne!(user.source_identity(), owner.source_identity());
assert_ne!(owner.source_identity(), limited.source_identity());
let cache = DigestCache::default();
for opened in [&user, &owner, &limited] {
assert_eq!(cache.timestamp_candidates(opened).len(), 1);
assert_eq!(cache.timestamp_candidates(opened).len(), 1);
}
assert_eq!(scans(&cache), 3);
let mut unprotected = owner.clone();
unprotected.remove_encryption();
assert_eq!(cache.timestamp_candidates(&unprotected).len(), 1);
assert!(cache.discovery.lock().unwrap().is_none());
let reopened = Document::open(Arc::new(write_full(&unprotected, &SaveOptions::default()).unwrap())).unwrap();
assert_eq!(cache.timestamp_candidates(&reopened).len(), 1);
assert_eq!(scans(&cache), 5);
}
#[test]
fn candidate_cache_is_bounded_without_truncating_results() {
let extras = vec![STAMP; DISCOVERY_CACHE_LIMIT + 1];
let doc = fixture(&extras);
let cache = DigestCache::default();
for _ in 0..2 {
assert_eq!(cache.timestamp_candidates(&doc).len(), extras.len());
assert!(cache.discovery.lock().unwrap().is_none());
}
assert_eq!(scans(&cache), 2);
}
#[test]
fn overlay_cache_is_bounded_and_new_documents_take_the_fallback() {
let mut doc = fixture(&[STAMP]);
for _ in 0..=DISCOVERY_CACHE_LIMIT {
doc.add(Dict::new());
}
let cache = DigestCache::default();
for _ in 0..2 {
assert_eq!(cache.timestamp_candidates(&doc), vec![ObjRef::new(3, 0)]);
assert!(cache.discovery.lock().unwrap().is_none());
}
let mut fresh = Document::new_empty();
let timestamp = fresh.add(stamp());
for _ in 0..2 {
assert_eq!(cache.timestamp_candidates(&fresh), vec![timestamp]);
assert!(cache.discovery.lock().unwrap().is_none());
}
assert_eq!(scans(&cache), 4);
}
/// Local, read-only corpus diagnostic; intentionally separate from ordinary timing.
/// The source PDF is never copied into the repository or written by this probe.
#[test]
#[ignore = "set PDFCRAFT_SIGNATURE_PROBE_SOURCE to a local PDF; parent coordinates resource use"]
fn real_file_discovery_cache_diagnostic() {
let path = std::env::var_os("PDFCRAFT_SIGNATURE_PROBE_SOURCE").expect("PDFCRAFT_SIGNATURE_PROBE_SOURCE");
let bytes = Arc::new(std::fs::read(path).unwrap());
let doc = Document::open(bytes.clone()).unwrap();
let mut objects = 0usize;
let mut candidates = 0usize;
let mut read_errors = 0usize;
let mut failed_numbers = Vec::new();
for (reference, object) in doc.scan_objects_checked() {
objects += 1;
match object {
Ok(object) => candidates += usize::from(is_document_timestamp(&object)),
Err(_) => {
read_errors += 1;
if failed_numbers.len() < 8 {
failed_numbers.push(reference.num);
}
}
}
}
eprintln!(
"signature-discovery-input\tbytes={}\tobjects={objects}\tcandidates={candidates}\tread_errors={read_errors}\tfirst_failed_numbers={failed_numbers:?}\trepairs={}\trevisions={}",
bytes.len(),
doc.repair_log().len(),
doc.revisions().len()
);
let cache = DigestCache::default();
let trust = TrustStore::default();
let first = list_cached(&doc, &bytes, &trust, &cache);
let cached_candidates = cache.discovery.lock().unwrap().as_ref().map(|entry| entry.candidates.len());
eprintln!("signature-discovery-initial\tfull_scans={}\tcached_candidates={cached_candidates:?}\tsignatures={}", scans(&cache), first.len());
assert_eq!(read_errors, 0, "input has unreadable objects; complete discovery must not be published");
assert_eq!(cached_candidates, Some(candidates), "input did not qualify for discovery caching");
assert_eq!(scans(&cache), 1);
for _ in 0..3 {
assert_eq!(list_cached(&doc, &bytes, &trust, &cache).len(), first.len());
}
eprintln!("signature-discovery-unchanged\tfull_scans={}", scans(&cache));
assert_eq!(scans(&cache), 1);
let mut edited = doc.clone();
edited.update_dict(edited.root().unwrap(), |d| d.set(b"PdfcraftSignatureCacheProbe".to_vec(), Object::Int(1))).unwrap();
assert_eq!(list_cached(&edited, &bytes, &trust, &cache).len(), first.len());
assert_eq!(list_cached(&doc, &bytes, &trust, &cache).len(), first.len());
assert_eq!(list_cached(&edited, &bytes, &trust, &cache).len(), first.len());
eprintln!("signature-discovery-edit-undo-redo\tfull_scans={}", scans(&cache));
assert_eq!(scans(&cache), 1);
let added = edited.add(stamp());
assert_eq!(list_cached(&edited, &bytes, &trust, &cache).len(), first.len() + 1);
edited.free(added);
assert_eq!(list_cached(&edited, &bytes, &trust, &cache).len(), first.len());
assert_eq!(list_cached(&doc, &bytes, &trust, &cache).len(), first.len());
eprintln!("signature-discovery-add-free-undo\tfull_scans={}", scans(&cache));
assert_eq!(scans(&cache), 1);
let reopened = Document::open(bytes.clone()).unwrap();
assert_eq!(list_cached(&reopened, &bytes, &trust, &cache).len(), first.len());
eprintln!("signature-discovery-reopen\tfull_scans={}", scans(&cache));
assert_eq!(scans(&cache), 2);
}

67
crates/sign/tests/pdf.rs
View File

@ -40,6 +40,29 @@ fn open(b: &[u8]) -> Document {
Document::open(Arc::new(b.to_vec())).unwrap()
}
#[test]
fn standalone_timestamp_scan_survives_cache_batches_and_object_streams() {
let mut doc = open(&fixture());
let filler: Vec<_> = (0..300).map(|i| Object::Ref(doc.add(Object::Int(i)))).collect();
let mut stamp = pdfcraft_cos::Dict::new();
stamp.set(b"Type".to_vec(), Object::name("Sig"));
stamp.set(b"SubFilter".to_vec(), Object::name("ETSI.RFC3161"));
stamp.set(b"ByteRange".to_vec(), Object::Array(vec![Object::Int(0), Object::Int(1), Object::Int(2), Object::Int(1)]));
stamp.set(b"Contents".to_vec(), Object::String(PdfString::literal("invalid-token")));
let stamp = doc.add(Object::Dict(stamp));
doc.update_dict(doc.root().unwrap(), |catalog| {
catalog.set(b"UninspectedObjects".to_vec(), Object::Array(filler));
catalog.set(b"UninspectedTimestamp".to_vec(), Object::Ref(stamp));
})
.unwrap();
let bytes = pdfcraft_cos::write_full(&doc, &SaveOptions::default()).unwrap();
let doc = open(&bytes);
assert!(doc.object_numbers().len() > 300);
let listed = signatures(&doc, &bytes, &TrustStore::default());
assert!(listed.iter().any(|s| s.doc_timestamp && s.status != Status::Valid));
assert!(listed.iter().any(|s| s.field == "Approval" && !s.signed));
}
fn opts() -> SignOptions {
SignOptions {
page: 0,
@ -270,6 +293,50 @@ fn a_document_timestamp_covers_the_file_and_validates() {
assert_eq!(stamp.status, Status::Valid, "{:?}", stamp.details);
}
#[test]
fn cached_discovery_still_validates_current_timestamp_bytes_ranges_and_trust() {
let tsa = TestTsa {
id: pkcs12::open(&data("rsa-aes.p12"), "test").unwrap(),
time: Time { year: 2026, month: 10, day: 6, hour: 12, minute: 0, second: 0 },
};
let mut source = open(&fixture());
source
.update_dict(source.root().unwrap(), |d| {
d.remove(b"AcroForm");
})
.unwrap();
let bytes = pdfcraft_sign::timestamp_document(&source, &tsa, "D:20261006120000Z").unwrap();
let doc = open(&bytes);
let cache = pdfcraft_sign::DigestCache::default();
let trusted = TrustStore { certs: vec![tsa.id.certificate.clone()] };
let check = |doc: &Document, bytes: &[u8], trust: &TrustStore| {
let listed = pdfcraft_sign::pdf::list_cached(doc, bytes, trust, &cache);
assert_eq!(listed.len(), 1);
assert!(listed[0].doc_timestamp);
listed.into_iter().next().unwrap()
};
assert_eq!(check(&doc, &bytes, &TrustStore::default()).status, Status::Unknown);
assert_eq!(check(&doc, &bytes, &trusted).status, Status::Valid);
assert_eq!(check(&doc, &bytes, &TrustStore::default()).status, Status::Unknown);
let mut tampered = bytes.clone();
let at = tampered.windows(13).position(|w| w == b"Contract text").unwrap();
tampered[at] = b'K';
assert_eq!(check(&doc, &tampered, &trusted).status, Status::Invalid);
assert_eq!(check(&doc, &[], &trusted).status, Status::Invalid);
let stamp = doc.scan_objects().find(|(_, o)| o.as_dict().is_some_and(|d| d.name(b"SubFilter") == Some(b"ETSI.RFC3161"))).unwrap().0;
let mut edited = doc.clone();
edited.update_dict(stamp, |d| d.set(b"ByteRange".to_vec(), Object::Array(vec![Object::Int(-1)]))).unwrap();
assert_eq!(check(&edited, &bytes, &trusted).status, Status::Invalid);
assert_eq!(check(&doc, &bytes, &trusted).status, Status::Valid);
// Even a malformed token in the same byte ranges must be read again after discovery.
let br = doc.get(stamp).as_dict().unwrap().get(b"ByteRange").unwrap().as_array().unwrap().clone();
let gap_start = br[1].as_int().unwrap() as usize;
tampered = bytes.clone();
tampered[gap_start + 1] = b'F';
tampered[gap_start + 2] = b'F';
assert_eq!(check(&doc, &tampered, &trusted).status, Status::Invalid);
}
#[test]
fn embedding_ltv_evidence_adds_a_dss_and_keeps_signatures_valid() {
use pdfcraft_sign::dss::{self, Evidence};

816
crates/ui-egui/src/canvas.rs
View File

@ -33,6 +33,33 @@ const TILE: u32 = 1024;
/// Longest side of the low-resolution backdrop drawn under tiles.
const BASE_SIDE: f32 = 2048.0;
const THUMB_W: f32 = 132.0;
const THUMB_H: f32 = 176.0;
const THUMB_BYTES: usize = 24 * 1024 * 1024;
const PAGE_BYTES: usize = 128 * 1024 * 1024;
const TILE_BYTES: usize = 128 * 1024 * 1024;
const UPLOAD_BYTES_PER_FRAME: usize = 16 * 1024 * 1024;
const RESULTS_PER_FRAME: usize = 8;
/// A defensive ceiling for unusually large viewports / n-up previews.
const MAX_THUMB_DEMAND: usize = 256;
/// At most this many whole-page rasters are cached beyond the ones on screen.
const MAX_CACHED_PAGES: usize = 24;
fn rgba_bytes(width: usize, height: usize) -> usize {
width.saturating_mul(height).saturating_mul(4)
}
fn texture_bytes(tex: &TextureHandle) -> usize {
let [w, h] = tex.size();
rgba_bytes(w, h)
}
/// Visible grid rows, plus one adjacent row in either direction. Work is independent of
/// document length, including after a large scroll jump.
fn thumbnail_rows(top: f32, bottom: f32, row_height: f32, rows: usize) -> Range<usize> {
let first = (top.max(0.0) / row_height).floor() as usize;
let end = (bottom.max(0.0) / row_height).ceil() as usize;
first.saturating_sub(1).min(rows)..end.saturating_add(1).min(rows)
}
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum Fit {
@ -183,6 +210,21 @@ struct PageTex {
tex: TextureHandle,
}
/// Retained RGBA texture accounting for automation and performance regressions. These
/// surface bytes exclude driver overhead and temporary upload buffers; they are not RSS.
#[derive(Clone, Copy, Debug, Default, serde::Serialize)]
pub struct RasterMemory {
pub thumbnail_count: usize,
pub thumbnail_bytes: usize,
pub page_count: usize,
pub page_bytes: usize,
pub tile_count: usize,
pub tile_bytes: usize,
pub thumbnail_demand: usize,
/// Last submitted missing results; workers may have completed some since submission.
pub pending_requests: usize,
}
pub struct DocView {
pub id: DocId,
pub zoom: f32,
@ -221,7 +263,13 @@ pub struct DocView {
errors: HashMap<usize, String>,
/// When each still-missing page was first shown, to flag unusually slow renders.
waiting_since: HashMap<usize, f64>,
thumbs: HashMap<usize, TextureHandle>,
thumbs: HashMap<usize, PageTex>,
/// This frame's consumers, with visible cells ahead of prefetch.
thumb_demand: HashMap<usize, bool>,
frame_queue: Vec<RenderRequest>,
/// Pages on screen this frame. Their rasters and tiles are always admitted: the viewport
/// bounds them, and the byte and count limits apply to prefetch only.
frame_visible: HashSet<usize>,
/// Thumbnails that are out of date (still shown until their replacement arrives).
stale_thumbs: HashSet<usize>,
/// Sharp tiles of large pages: (page, scale tag, tile x, tile y) → texture. Tiles of an
@ -349,6 +397,19 @@ pub enum ViewAction {
}
impl DocView {
pub fn raster_memory(&self) -> RasterMemory {
RasterMemory {
thumbnail_count: self.thumbs.len(),
thumbnail_bytes: self.thumbs.values().fold(0usize, |n, p| n.saturating_add(texture_bytes(&p.tex))),
page_count: self.pages.len(),
page_bytes: self.pages.values().fold(0usize, |n, p| n.saturating_add(texture_bytes(&p.tex))),
tile_count: self.tiles.len(),
tile_bytes: self.tiles.values().fold(0usize, |n, tex| n.saturating_add(texture_bytes(tex))),
thumbnail_demand: self.thumb_demand.len(),
pending_requests: self.last_queue.len(),
}
}
/// Where `page` is drawn on screen this frame (`None` when it is not on screen).
pub fn page_screen_rect(&self, page: usize) -> Option<Rect> {
self.screen_xforms.iter().find(|(p, _)| *p == page).map(|(_, xf)| xf.rect)
@ -403,6 +464,9 @@ impl DocView {
errors: HashMap::new(),
waiting_since: HashMap::new(),
thumbs: HashMap::new(),
thumb_demand: HashMap::new(),
frame_queue: Vec::new(),
frame_visible: HashSet::new(),
stale_thumbs: HashSet::new(),
tiles: HashMap::new(),
texts: HashMap::new(),
@ -597,7 +661,7 @@ impl DocView {
/// A page's thumbnail texture, when rendered (the print preview uses them).
pub(crate) fn thumb_id(&self, page: usize) -> Option<egui::TextureId> {
self.thumbs.get(&page).map(|t| t.id())
self.thumbs.get(&page).map(|t| t.tex.id())
}
pub(crate) fn page_text(&self, page: usize) -> Option<Arc<PageText>> {
@ -705,7 +769,7 @@ impl DocView {
}
pub fn thumb(&self, page: usize) -> Option<&TextureHandle> {
self.thumbs.get(&page)
self.thumbs.get(&page).map(|t| &t.tex)
}
pub fn go_to_page(&mut self, page: usize) {
@ -980,58 +1044,263 @@ impl DocView {
}
}
/// Pull finished renders into textures.
pub fn receive(&mut self, ctx: &egui::Context, pool: &RenderPool) {
let mut got = false;
// Inline (single-threaded, e.g. web) rendering happens inside try_recv: one page per frame.
let budget = if pool.is_inline() { 1 } else { usize::MAX };
let mut n = 0;
while n < budget
&& let Some(r) = pool.try_recv()
{
n += 1;
got = true;
if r.request.kind == RequestKind::Text {
match r.text {
Some(t) => {
self.texts.insert(r.request.page, t);
self.refresh_find_page(r.request.page);
}
None => {
log::warn!("page {} text: {}", r.request.page + 1, r.error.unwrap_or_default());
self.text_failed.insert(r.request.page);
}
/// Start collecting demand from every consumer before scheduling or uploading results.
pub(crate) fn begin_render_frame(&mut self) {
self.thumb_demand.clear();
self.frame_queue.clear();
self.frame_visible.clear();
}
pub(crate) fn need_thumbnail(&mut self, page: usize, visible: bool) {
if page >= self.page_count {
return;
}
if let Some(priority) = self.thumb_demand.get_mut(&page) {
*priority |= visible;
} else if self.thumb_demand.len() < MAX_THUMB_DEMAND {
self.thumb_demand.insert(page, visible);
} else if visible && let Some(prefetch) = self.thumb_demand.iter().find_map(|(&p, &v)| (!v).then_some(p)) {
self.thumb_demand.remove(&prefetch);
self.thumb_demand.insert(page, true);
}
}
/// Only the displayed document owns raster textures. Tabs keep their document and view
/// state, but do not each retain a separate full application texture allowance.
pub(crate) fn suspend_rendering(&mut self, pool: &RenderPool) {
self.begin_render_frame();
self.pages.clear();
self.tiles.clear();
self.thumbs.clear();
self.grid_pages.clear();
self.stale_thumbs.clear();
self.waiting_since.clear();
// Signature previews own a separate renderer and image/background textures.
self.signature_drag = Default::default();
if !self.last_queue.is_empty() {
pool.set_queue(Vec::new());
self.last_queue.clear();
}
// Results already finished for this tab would otherwise sit in its pool, rasters and
// all, until the tab is shown again. (An inline pool renders inside try_recv: skip it.)
if !pool.is_inline() {
while pool.try_recv().is_some() {}
}
}
fn thumbnail_requests(&self, info: &DocInfo, ppp: f32) -> Vec<RenderRequest> {
let mut pages: Vec<_> = self.thumb_demand.iter().map(|(&p, &visible)| (p, visible)).collect();
pages.sort_unstable_by_key(|&(p, visible)| (!visible, p));
// At high DPI or in a very large viewport every demanded cell still fits the same
// byte allowance. Both dimensions are capped, including tall/wide page boxes.
// The count is rounded up to a power of two: the scale is part of each request's tag,
// so a density that followed every row scrolling in or out would re-render them all.
let per_thumb = THUMB_BYTES / pages.len().max(1).next_power_of_two();
let density = ppp.clamp(0.25, 4.0).min((per_thumb as f32 / (4.0 * THUMB_W * THUMB_H)).sqrt() * 0.98);
pages
.into_iter()
.filter_map(|(page, _)| {
let p = info.pages.get(page)?;
let scale = (THUMB_W / p.width.max(1.0)).min(THUMB_H / p.height.max(1.0)) * density;
Some(RenderRequest { page, kind: RequestKind::Pixels, tile: None, scale, tag: THUMB_TAG | u64::from(scale.to_bits()) })
})
.collect()
}
/// Admit current demand within explicit byte limits. Cached and pending surfaces share
/// the allowance, so an evicted page can be requested again without a permanent done set.
fn prepare_render_queue(&mut self, info: &DocInfo, ppp: f32) -> Vec<RenderRequest> {
let requests = std::mem::take(&mut self.frame_queue);
let mut queue = Vec::new();
let mut pages = HashSet::new();
let mut tiles = HashSet::new();
let mut tile_tags = HashSet::new();
let mut grid = Vec::new();
let (mut page_bytes, mut tile_bytes) = (0usize, 0usize);
for req in requests.iter().filter(|r| r.kind == RequestKind::Pixels) {
let Some(p) = info.pages.get(req.page) else { continue };
// Organize-grid renders: only the cells in view that need one (organize_grid).
if req.tag & GRID_TAG != 0 {
grid.push(*req);
continue;
}
// On-screen work is always admitted (the viewport bounds it); the limits only
// decide how much prefetch fits beside it.
let visible = self.frame_visible.contains(&req.page);
if let Some(tile) = req.tile {
let key = (req.page, req.tag, tile.x / TILE, tile.y / TILE);
let bytes = rgba_bytes(tile.w as usize, tile.h as usize);
if (!visible && tile_bytes.saturating_add(bytes) > TILE_BYTES) || !tiles.insert(key) {
continue;
}
tile_bytes = tile_bytes.saturating_add(bytes);
tile_tags.insert(req.tag);
if !self.tiles.contains_key(&key) {
queue.push(*req);
}
continue;
}
if let Some(e) = r.error {
log::warn!("page {}: {e}", r.request.page + 1);
self.errors.insert(r.request.page, e);
continue;
}
let img = texture_image([r.width as usize, r.height as usize], r.rgba);
let page = r.request.page;
if let Some(t) = r.request.tile {
let tex = ctx.load_texture(format!("tile-{:?}-{page}-{}-{}", self.id, t.x, t.y), img, TextureOptions::LINEAR);
self.tiles.insert((page, r.request.tag, t.x / TILE, t.y / TILE), tex);
continue;
}
if r.request.tag & GRID_TAG != 0 {
let tex = ctx.load_texture(format!("grid-{:?}-{page}", self.id), img, TextureOptions::LINEAR);
self.grid_pages.insert(page, (r.width, tex));
} else if r.request.tag & THUMB_TAG != 0 {
let tex = ctx.load_texture(format!("thumb-{:?}-{page}", self.id), img, TextureOptions::LINEAR);
self.thumbs.insert(page, tex);
self.stale_thumbs.remove(&page);
} else {
let tex = ctx.load_texture(format!("page-{:?}-{page}", self.id), img, TextureOptions::LINEAR);
self.pages.insert(page, PageTex { tag: r.request.tag, tex });
self.signature_drag.page_received(page);
self.waiting_since.remove(&page);
let bytes = rgba_bytes(device_pixels(p.width, req.scale) as usize, device_pixels(p.height, req.scale) as usize);
if (!visible && (page_bytes.saturating_add(bytes) > PAGE_BYTES || pages.len() >= MAX_CACHED_PAGES)) || !pages.insert(req.page) {
continue;
}
page_bytes = page_bytes.saturating_add(bytes);
if self.pages.get(&req.page).is_none_or(|p| p.tag != req.tag) {
queue.push(*req);
}
}
}
if got {
ctx.request_repaint();
// Pages no longer demanded stay cached while the allowance has room, nearest to the
// current page first, so turning back a page shows it at once instead of rendering.
let mut undemanded: Vec<usize> = self.pages.keys().copied().filter(|p| !pages.contains(p)).collect();
undemanded.sort_by_key(|p| p.abs_diff(self.current));
let mut kept = pages.len();
for page in undemanded {
let bytes = self.pages.get(&page).map_or(0, |p| texture_bytes(&p.tex));
if kept < MAX_CACHED_PAGES && page_bytes.saturating_add(bytes) <= PAGE_BYTES {
page_bytes = page_bytes.saturating_add(bytes);
kept += 1;
pages.insert(page);
}
}
self.pages.retain(|p, _| pages.contains(p));
// Tiles of an earlier zoom (not a demanded scale) stay while the canvas keeps them, a
// bounded few under a page whose current tiles are still missing (OLD_TILES).
self.tiles.retain(|key, _| tiles.contains(key) || !tile_tags.contains(&key.1));
queue.extend(grid);
let thumbs = self.thumbnail_requests(info, ppp);
let mut thumb_bytes = 0usize;
let mut admitted = HashSet::new();
for req in thumbs {
let Some(p) = info.pages.get(req.page) else { continue };
let bytes = rgba_bytes(device_pixels(p.width, req.scale) as usize, device_pixels(p.height, req.scale) as usize);
if thumb_bytes.saturating_add(bytes) > THUMB_BYTES || self.errors.contains_key(&req.page) {
continue;
}
thumb_bytes += bytes;
admitted.insert(req.page);
if self.thumbs.get(&req.page).is_none_or(|t| t.tag != req.tag) || self.stale_thumbs.contains(&req.page) {
queue.push(req);
}
}
self.thumbs.retain(|p, _| admitted.contains(p));
self.stale_thumbs.retain(|p| admitted.contains(p));
// Visible main pages and thumbnails take precedence over background search work.
queue.extend(requests.into_iter().filter(|r| r.kind == RequestKind::Text));
queue
}
pub(crate) fn finish_render_frame(&mut self, ctx: &egui::Context, info: &DocInfo, pool: &RenderPool) {
let queue = self.prepare_render_queue(info, ctx.pixels_per_point());
if queue != self.last_queue {
pool.set_queue(queue.clone());
self.last_queue = queue;
}
self.receive(ctx, pool);
if !self.last_queue.is_empty() {
ctx.request_repaint_after(std::time::Duration::from_millis(30));
}
}
/// Pull a bounded batch of current results into textures. Demand is committed first, so
/// offscreen results and obsolete scale/configuration tags never allocate a ColorImage.
pub fn receive(&mut self, ctx: &egui::Context, pool: &RenderPool) {
let budget = if pool.is_inline() { 1 } else { RESULTS_PER_FRAME };
let mut uploaded = 0usize;
#[cfg(not(target_arch = "wasm32"))]
let started = std::time::Instant::now();
for _ in 0..budget {
let Some(r) = pool.try_recv() else { break };
uploaded = uploaded.saturating_add(self.receive_result(ctx, r));
// One whole-page upload may exceed the frame allowance; it must still progress.
if uploaded >= UPLOAD_BYTES_PER_FRAME {
break;
}
#[cfg(not(target_arch = "wasm32"))]
if started.elapsed() >= std::time::Duration::from_millis(4) {
break;
}
}
}
fn receive_result(&mut self, ctx: &egui::Context, r: pdfcraft_render::RenderedPage) -> usize {
if !self.last_queue.contains(&r.request) {
return 0;
}
ctx.request_repaint();
if r.request.kind == RequestKind::Text {
match r.text {
Some(t) => {
self.texts.insert(r.request.page, t);
self.refresh_find_page(r.request.page);
}
None => {
log::warn!("page {} text: {}", r.request.page + 1, r.error.unwrap_or_default());
self.text_failed.insert(r.request.page);
}
}
return 0;
}
if let Some(e) = r.error {
log::warn!("page {}: {e}", r.request.page + 1);
self.errors.insert(r.request.page, e);
return 0;
}
let page = r.request.page;
let bytes = rgba_bytes(r.width as usize, r.height as usize);
if bytes != r.rgba.len() || !self.admit_texture(r.request, bytes) {
return 0;
}
// The length was checked above, so the renderer's buffer becomes the texture data as is.
let img = texture_image([r.width as usize, r.height as usize], r.rgba);
if let Some(t) = r.request.tile {
let tex = ctx.load_texture(format!("tile-{:?}-{page}-{}-{}", self.id, t.x, t.y), img, TextureOptions::LINEAR);
self.tiles.insert((page, r.request.tag, t.x / TILE, t.y / TILE), tex);
} else if r.request.tag & GRID_TAG != 0 {
let tex = ctx.load_texture(format!("grid-{:?}-{page}", self.id), img, TextureOptions::LINEAR);
self.grid_pages.insert(page, (r.width, tex));
} else if r.request.tag & THUMB_TAG != 0 {
let tex = ctx.load_texture(format!("thumb-{:?}-{page}", self.id), img, TextureOptions::LINEAR);
self.thumbs.insert(page, PageTex { tag: r.request.tag, tex });
self.stale_thumbs.remove(&page);
} else {
let tex = ctx.load_texture(format!("page-{:?}-{page}", self.id), img, TextureOptions::LINEAR);
self.pages.insert(page, PageTex { tag: r.request.tag, tex });
self.signature_drag.page_received(page);
self.waiting_since.remove(&page);
}
bytes
}
/// Check real dimensions before conversion, including stale textures kept during edits.
/// Dropping an old handle releases its backend allocation after the current frame.
fn admit_texture(&mut self, req: RenderRequest, bytes: usize) -> bool {
// An on-screen page or tile is always taken (the viewport bounds it): it may evict
// off-screen rasters but never another on-screen one, and is never refused.
let visible = req.tag & THUMB_TAG == 0 && self.frame_visible.contains(&req.page);
// Organize-grid renders are bounded by the cells in view (organize_grid).
if req.tag & GRID_TAG != 0 {
return true;
}
if let Some(t) = req.tile {
let key = (req.page, req.tag, t.x / TILE, t.y / TILE);
self.tiles.remove(&key);
let retained = self.tiles.values().fold(0usize, |n, tex| n.saturating_add(texture_bytes(tex)));
visible || bytes <= TILE_BYTES.saturating_sub(retained)
} else {
let thumb = req.tag & THUMB_TAG != 0;
let (cache, limit) = if thumb { (&mut self.thumbs, THUMB_BYTES) } else { (&mut self.pages, PAGE_BYTES) };
cache.remove(&req.page);
let mut retained = cache.values().fold(0usize, |n, p| n.saturating_add(texture_bytes(&p.tex)));
// A zoom/DPI change can temporarily retain larger stale images than the new
// demand estimates. Evict those first, then let the current demand refill them.
let on_screen = |p: &usize| !thumb && self.frame_visible.contains(p);
while retained.saturating_add(bytes) > limit {
let Some(page) = cache.keys().copied().filter(|p| !on_screen(p)).max_by_key(|p| p.abs_diff(self.current)) else { break };
if let Some(old) = cache.remove(&page) {
retained = retained.saturating_sub(texture_bytes(&old.tex));
}
}
visible || bytes <= limit.saturating_sub(retained)
}
}
@ -1382,7 +1651,6 @@ pub fn document_area(app: &mut PdfCraftApp, index: usize, ui: &mut egui::Ui) {
ui.centered_and_justified(|ui| ui.label(tl!("This document has no pages.")));
return;
}
let want_thumbs = app.right == Some(RightPanel::Pages) || app.views[index].organize || app.dialog == Some(crate::Dialog::Print);
// The Prepare a form panel is open (or a field tool is picked): fields are edited, not filled.
let preparing = app.is_preparing();
// Edit a PDF: added text and images can be selected, moved and edited.
@ -1402,7 +1670,7 @@ pub fn document_area(app: &mut PdfCraftApp, index: usize, ui: &mut egui::Ui) {
// No dialog, close prompt or palette over the page: only then does page input count.
let unobstructed = app.dialog.is_none() && app.close_request.is_none() && !app.palette_open;
if view.organize {
organize_grid(view, info, &doc.renderer, doc.allows_assembly(), doc.dirty, unobstructed, ui, &t);
organize_grid(view, info, doc.allows_assembly(), doc.dirty, unobstructed, ui, &t);
return;
}
@ -1671,6 +1939,12 @@ pub fn document_area(app: &mut PdfCraftApp, index: usize, ui: &mut egui::Ui) {
} else {
(scale, tag)
};
// Under tiles, a whole-page raster sharper than the backdrop (from before the page
// grew past the tiling size) is a better backdrop: keep demanding it as it is.
match view.pages.get(&i).filter(|p| tiled && p.tag != STALE_TAG && tag_scale(p.tag) >= want_scale) {
Some(p) => wanted.push((i, tag_scale(p.tag), p.tag, None)),
None => wanted.push((i, want_scale, want_tag, None)),
}
match view.pages.get(&i) {
Some(p) => {
if !tiled && p.tag == want_tag {
@ -1679,15 +1953,8 @@ pub fn document_area(app: &mut PdfCraftApp, index: usize, ui: &mut egui::Ui) {
// A backdrop, or a raster at an older scale until the new one arrives.
xf.paint_image(painter, p.tex.id(), 0.0, 0.0, 1.0, 1.0);
}
// Under tiles, a whole-page raster sharper than the backdrop (from before
// the page grew past the tiling size) is a better backdrop: keep it.
let sharper = tiled && p.tag != STALE_TAG && tag_scale(p.tag) >= want_scale;
if p.tag != want_tag && !sharper {
wanted.push((i, want_scale, want_tag, None));
}
}
None => {
wanted.push((i, want_scale, want_tag, None));
let now = ui.input(|inp| inp.time);
let since = *view.waiting_since.entry(i).or_insert(now);
let msg = if now - since > 6.0 { "Still rendering — this page is unusually complex…" } else { "Rendering…" };
@ -1727,19 +1994,17 @@ pub fn document_area(app: &mut PdfCraftApp, index: usize, ui: &mut egui::Ui) {
if w == 0 || h == 0 {
continue;
}
match view.tiles.get(&(i, tag, tx, ty)) {
Some(tex) => {
let (fw, fh) = (dw as f32, dh as f32);
xf.texel_aligned([dw as usize, dh as usize], ppp).paint_image(
painter,
tex.id(),
x as f32 / fw,
y as f32 / fh,
(x + w) as f32 / fw,
(y + h) as f32 / fh,
);
}
_ => wanted.push((i, scale, tag, Some(Tile { x, y, w, h }))),
wanted.push((i, scale, tag, Some(Tile { x, y, w, h })));
if let Some(tex) = view.tiles.get(&(i, tag, tx, ty)) {
let (fw, fh) = (dw as f32, dh as f32);
xf.texel_aligned([dw as usize, dh as usize], ppp).paint_image(
painter,
tex.id(),
x as f32 / fw,
y as f32 / fh,
(x + w) as f32 / fw,
(y + h) as f32 / fh,
);
}
}
}
@ -2157,16 +2422,10 @@ pub fn document_area(app: &mut PdfCraftApp, index: usize, ui: &mut egui::Ui) {
view.auto_scroll.paint(ui, avail);
// Bound texture memory: keep sharp rasters only near the current page.
if view.pages.len() > 24 {
let cur = view.current;
view.pages.retain(|&p, _| p.abs_diff(cur) <= 8);
}
// Schedule renders: visible pages first (nearest the current page), then thumbnails.
// Collect all main-page demand, including cached surfaces, for byte admission later.
let (mut wanted, visible_now) = out.inner;
let cur = view.current;
// Nearest pages first; for each page, the backdrop before its tiles.
wanted.sort_by_key(|w| ((w.0 as isize - cur as isize).unsigned_abs(), w.3.is_some()));
wanted.sort_by_key(|w| (!visible_now.contains(&w.0), w.0.abs_diff(cur), w.3.is_some()));
// Tiles of far-away pages are useless: free them. Tiles of other zooms stay only while a
// current tile of their page is still missing, at most `OLD_TILES` of them, nearest scale first.
let missing: HashSet<usize> = wanted.iter().filter(|w| w.3.is_some()).map(|w| w.0).collect();
@ -2190,21 +2449,8 @@ pub fn document_area(app: &mut PdfCraftApp, index: usize, ui: &mut egui::Ui) {
text_pages.extend(rest);
}
queue.extend(text_pages.into_iter().map(|page| RenderRequest { page, kind: RequestKind::Text, tile: None, scale: 1.0, tag: TEXT_TAG }));
if want_thumbs {
let s = THUMB_W * ppp / info.pages.iter().map(|p| p.width).fold(1.0, f32::max);
for page in 0..info.pages.len() {
if (!view.thumbs.contains_key(&page) || view.stale_thumbs.contains(&page)) && !view.errors.contains_key(&page) {
queue.push(RenderRequest { page, kind: RequestKind::Pixels, tile: None, scale: s, tag: THUMB_TAG });
}
}
}
if queue != view.last_queue {
doc.renderer.set_queue(queue.clone());
view.last_queue = queue;
}
if !view.last_queue.is_empty() {
ui.ctx().request_repaint_after(std::time::Duration::from_millis(30));
}
view.frame_queue = queue;
view.frame_visible = visible_now.iter().copied().collect();
if let Some((pos, text)) = hover_text {
egui::Area::new(egui::Id::new("canvas-hover")).order(egui::Order::Tooltip).fixed_pos(pos + vec2(14.0, 16.0)).show(ui.ctx(), |ui| {
@ -2985,17 +3231,7 @@ fn gap_button(ui: &mut egui::Ui, gap: usize, at: Pos2, height: f32, label: Strin
resp.on_hover_text(label).clicked()
}
#[allow(clippy::too_many_arguments)]
fn organize_grid(
view: &mut DocView,
info: &DocInfo,
pool: &RenderPool,
editable: bool,
dirty: bool,
auto_scroll_enabled: bool,
ui: &mut egui::Ui,
t: &Tokens,
) {
fn organize_grid(view: &mut DocView, info: &DocInfo, editable: bool, dirty: bool, auto_scroll_enabled: bool, ui: &mut egui::Ui, t: &Tokens) {
let ppp = ui.ctx().pixels_per_point();
// The page image scales with the zoom; the padding and the page number don't.
let cell = vec2(146.0 * view.grid_zoom + 44.0, 194.0 * view.grid_zoom + 56.0);
@ -3011,12 +3247,13 @@ fn organize_grid(
Vec2::ZERO
};
let middle_gesture = view.auto_scroll.blocks_input();
let mut cells: Vec<(usize, Rect)> = Vec::with_capacity(info.pages.len());
let mut cells: Vec<(usize, Rect)> = Vec::new();
// The pages in view that are drawn larger than their thumbnails, and those of them that
// need a sharper render than they have.
let (mut in_view, mut sharper): (Vec<usize>, Vec<RenderRequest>) = (Vec::new(), Vec::new());
let mut drop = false;
egui::ScrollArea::vertical().auto_shrink([false, false]).show(ui, |ui| {
// The source cell can scroll out of the virtualized rows during a drag.
let mut drop = view.org_drag.is_some() && ui.input(|i| i.pointer.primary_released());
egui::ScrollArea::vertical().auto_shrink([false, false]).show_viewport(ui, |ui, clip| {
if middle_gesture {
let opacity = ui.opacity();
ui.disable();
@ -3029,12 +3266,14 @@ fn organize_grid(
let cols = ((ui.available_width() - 40.0) / cell.x).floor().max(1.0) as usize;
let rows = info.pages.len().div_ceil(cols);
let left = (ui.available_width() - cols as f32 * cell.x) / 2.0;
for row in 0..rows {
let (row_rect, _) = ui.allocate_exact_size(vec2(ui.available_width(), cell.y), Sense::hover());
let (grid, _) = ui.allocate_exact_size(vec2(ui.available_width(), rows as f32 * cell.y), Sense::hover());
for row in thumbnail_rows(clip.top() - 20.0, clip.bottom() - 20.0, cell.y, rows) {
let row_rect = Rect::from_min_size(grid.min + vec2(0.0, row as f32 * cell.y), vec2(grid.width(), cell.y));
for col in 0..cols {
let i = row * cols + col;
let Some(p) = info.pages.get(i) else { break };
let c = Rect::from_min_size(pos2(row_rect.left() + left + col as f32 * cell.x, row_rect.top()), cell);
view.need_thumbnail(i, c.intersects(ui.clip_rect()));
let resp = ui.interact(c, ui.id().with(("org", i)), if editable { Sense::click_and_drag() } else { Sense::click() });
cells.push((i, c));
if let Some((_, above)) = anchor.filter(|(page, _)| *page == i) {
@ -3070,7 +3309,7 @@ fn organize_grid(
// A page drawn larger than its thumbnail gets a render at the size drawn, while
// it is in view; until that arrives the thumbnail stands in.
let want = size.x * ppp;
let thumb = view.thumbs.get(&i);
let thumb = view.thumbs.get(&i).map(|t| &t.tex);
let thumb_ok = thumb.is_some_and(|t| t.size()[0] as f32 >= want * GRID_SHARP.start());
let sharp = view.grid_pages.get(&i).filter(|_| !thumb_ok);
if !thumb_ok && ui.is_rect_visible(c) {
@ -3127,6 +3366,7 @@ fn organize_grid(
}
}
if let Some((i, c)) = cells.last()
&& i + 1 == info.pages.len()
&& gap_button(ui, i + 1, pos2(c.right(), mid(c)), cell.y - 56.0, tl!("Insert a file at the end").to_string(), t)
{
view.pending_action = Some(ViewAction::InsertFromFileAt(i + 1));
@ -3207,15 +3447,8 @@ fn organize_grid(
// Sharp renders are kept only for the pages in view, so their memory stays small however
// long the document is. They come first; the thumbnails of the other pages follow.
view.grid_pages.retain(|page, _| in_view.contains(page));
let s = THUMB_W * ppp / info.pages.iter().map(|p| p.width).fold(1.0, f32::max);
let thumbs = (0..info.pages.len())
.filter(|p| (!view.thumbs.contains_key(p) || view.stale_thumbs.contains(p)) && !view.errors.contains_key(p))
.map(|page| RenderRequest { page, kind: RequestKind::Pixels, tile: None, scale: s, tag: THUMB_TAG });
let queue: Vec<RenderRequest> = sharper.into_iter().chain(thumbs).collect();
if queue != view.last_queue {
pool.set_queue(queue.clone());
view.last_queue = queue;
}
// Sharp renders of the pages in view join this frame's demand ahead of the thumbnails.
view.frame_queue.extend(sharper);
if let Some(p) = open_page {
view.organize = false;
view.go_to_page(p);
@ -3285,6 +3518,345 @@ trailer << /Root 1 0 R >>
DocView::new(DocId(1), &info, ViewDefaults { layout, ..Default::default() })
}
fn thumbnail_info(count: usize) -> DocInfo {
DocInfo {
pages: (0..count)
.map(|i| {
let (width, height) = match i % 3 {
0 => (612.0, 792.0),
1 => (200.0, 4000.0),
_ => (4000.0, 200.0),
};
pdfcraft_render::PageInfo { width, height, label: (i + 1).to_string(), crop: [0.0, 0.0, width, height], rotation: 0 }
})
.collect(),
..Default::default()
}
}
#[test]
fn ten_thousand_pages_only_demand_viewport_and_prefetch_at_both_densities() {
let info = thumbnail_info(10_000);
let mut v = DocView::new(DocId(1), &info, ViewDefaults::default());
let cols = 4;
let rows = info.pages.len().div_ceil(cols);
for ppp in [1.0, 2.0] {
for row in [0, 1234, rows - 1] {
v.begin_render_frame();
let range = thumbnail_rows(row as f32 * 250.0, row as f32 * 250.0 + 750.0, 250.0, rows);
for p in range.start * cols..(range.end * cols).min(info.pages.len()) {
v.need_thumbnail(p, p / cols == row);
}
let queue = v.prepare_render_queue(&info, ppp);
assert!(queue.len() <= 5 * cols, "only three visible rows and two prefetch rows");
assert_eq!(queue.first().unwrap().page / cols, row, "visible cells precede prefetch");
let bytes: usize = queue
.iter()
.map(|r| {
let p = &info.pages[r.page];
let (w, h) = (device_pixels(p.width, r.scale), device_pixels(p.height, r.scale));
assert!(w <= (THUMB_W * ppp).ceil() as u32);
assert!(h <= (THUMB_H * ppp).ceil() as u32);
rgba_bytes(w as usize, h as usize)
})
.sum();
assert!(bytes <= THUMB_BYTES);
assert_eq!(queue, v.prepare_render_queue(&info, ppp), "unchanged demand reconstructs the same work");
}
}
}
#[test]
fn oversized_thumbnail_viewport_scales_all_admitted_cells_to_one_budget() {
let info = thumbnail_info(10_000);
let mut v = DocView::new(DocId(1), &info, ViewDefaults::default());
for page in 0..MAX_THUMB_DEMAND {
v.need_thumbnail(page, true);
}
let queue = v.prepare_render_queue(&info, 2.0);
assert_eq!(queue.len(), MAX_THUMB_DEMAND);
let bytes: usize = queue
.iter()
.map(|r| {
let p = &info.pages[r.page];
rgba_bytes(device_pixels(p.width, r.scale) as usize, device_pixels(p.height, r.scale) as usize)
})
.sum();
assert!(bytes <= THUMB_BYTES);
}
fn solid_result(request: RenderRequest, w: u32, h: u32) -> pdfcraft_render::RenderedPage {
pdfcraft_render::RenderedPage {
request,
width: w,
height: h,
rgba: pdfcraft_render::Pixels::from(vec![u32::MAX; rgba_bytes(w as usize, h as usize) / 4]),
text: None,
error: None,
millis: 0,
}
}
#[test]
fn thumbnails_release_on_scroll_panel_close_and_request_again_after_eviction() {
let ctx = egui::Context::default();
let info = thumbnail_info(10_000);
let mut v = DocView::new(DocId(1), &info, ViewDefaults::default());
v.need_thumbnail(0, true);
let first = v.prepare_render_queue(&info, 1.0);
v.last_queue = first.clone();
assert_eq!(v.receive_result(&ctx, solid_result(first[0], 132, 171)), 132 * 171 * 4);
assert!(v.prepare_render_queue(&info, 1.0).is_empty());
v.begin_render_frame();
v.need_thumbnail(9999, true);
v.prepare_render_queue(&info, 1.0);
assert!(v.thumbs.is_empty(), "jump releases old handles");
v.begin_render_frame();
v.need_thumbnail(0, true);
assert_eq!(v.prepare_render_queue(&info, 1.0), first, "an evicted result must be requested again");
v.begin_render_frame();
assert!(v.prepare_render_queue(&info, 1.0).is_empty(), "closing consumers cancels thumbnail work");
}
#[test]
fn stale_results_are_rejected_before_conversion_after_dpi_invalidation_and_suspend() {
let ctx = egui::Context::default();
let info = thumbnail_info(1);
let mut v = DocView::new(DocId(1), &info, ViewDefaults::default());
v.need_thumbnail(0, true);
let first = v.prepare_render_queue(&info, 1.0)[0];
v.last_queue = v.prepare_render_queue(&info, 2.0);
// Valid pixels: only the request gate can keep these out (a size check can't).
let obsolete = || solid_result(first, 132, 171);
assert_eq!(v.receive_result(&ctx, obsolete()), 0);
assert!(v.thumbs.is_empty(), "a result for the old DPI is not uploaded");
// The same result is uploaded while its request is current.
v.last_queue = vec![first];
assert_eq!(v.receive_result(&ctx, obsolete()), 132 * 171 * 4);
v.thumbs.clear();
v.invalidate_content();
assert_eq!(v.receive_result(&ctx, obsolete()), 0);
assert!(v.thumbs.is_empty(), "nor after the content changed");
let pool = RenderPool::new_inline(Arc::new(Vec::new()), Default::default());
v.last_queue = vec![first];
v.suspend_rendering(&pool);
assert_eq!(v.receive_result(&ctx, obsolete()), 0);
assert!(v.thumbs.is_empty(), "nor after the tab was suspended");
}
#[test]
fn editing_after_layout_preserves_stale_page_until_replacement() {
let ctx = egui::Context::default();
let info = thumbnail_info(1);
let mut v = DocView::new(DocId(1), &info, ViewDefaults::default());
let req = RenderRequest { scale: 1.0, tag: 1000, ..Default::default() };
v.last_queue = vec![req];
v.receive_result(&ctx, solid_result(req, 10, 10));
v.frame_queue = vec![req];
v.page_changed(0);
assert_eq!(v.prepare_render_queue(&info, 1.0), vec![req]);
assert_eq!(v.pages.get(&0).unwrap().tag, STALE_TAG);
}
#[test]
fn page_and_tile_admission_accounts_for_bytes_before_count_limits() {
let mut info = thumbnail_info(40);
for p in &mut info.pages {
p.width = 4000.0;
p.height = 4000.0;
}
let mut v = DocView::new(DocId(1), &info, ViewDefaults::default());
v.frame_queue = (0..40).map(|page| RenderRequest { page, scale: 1.0, tag: 1000, ..Default::default() }).collect();
let queue = v.prepare_render_queue(&info, 1.0);
assert_eq!(queue.len(), 2, "128 MiB admits two 4000-square pages, not 24");
v.frame_queue = (0..40)
.map(|page| RenderRequest { page, scale: 1.0, tag: 1000, tile: Some(Tile { x: 0, y: 0, w: 1024, h: 1024 }), ..Default::default() })
.collect();
assert_eq!(v.prepare_render_queue(&info, 1.0).len(), 32);
assert!(!v.admit_texture(RenderRequest::default(), PAGE_BYTES + 1));
}
#[test]
fn on_screen_pages_and_tiles_are_admitted_beyond_the_prefetch_allowance() {
let ctx = egui::Context::default();
let mut info = thumbnail_info(40);
for p in &mut info.pages {
p.width = 2000.0;
p.height = 2000.0;
}
let mut v = DocView::new(DocId(1), &info, ViewDefaults::default());
// Ten 16 MB pages on screen (a zoomed-out HiDPI view) exceed the 128 MiB allowance.
v.frame_visible = (0..10).collect();
v.frame_queue = (0..40).map(|page| RenderRequest { page, scale: 1.0, tag: 1000, ..Default::default() }).collect();
let queue = v.prepare_render_queue(&info, 1.0);
let pages: Vec<usize> = queue.iter().map(|r| r.page).collect();
assert_eq!(pages, (0..10).collect::<Vec<_>>(), "every on-screen page, and no prefetch past the allowance");
v.last_queue = queue.clone();
for req in &queue {
assert!(v.receive_result(&ctx, solid_result(*req, 2000, 2000)) > 0, "on-screen page {} uploaded", req.page);
}
assert_eq!(v.pages.len(), 10, "none evicted another on-screen page");
// A 5K viewport needs more 1024-pixel tiles than the tile allowance holds.
v.frame_queue = (0..40)
.map(|x| RenderRequest { page: 0, scale: 1.0, tag: 1000, tile: Some(Tile { x: x * TILE, y: 0, w: 1024, h: 1024 }), ..Default::default() })
.collect();
assert_eq!(v.prepare_render_queue(&info, 1.0).len(), 40);
}
#[test]
fn a_page_turned_back_to_is_still_cached() {
let ctx = egui::Context::default();
let info = thumbnail_info(10);
let mut v = DocView::new(DocId(1), &info, ViewDefaults::default());
let req = |page| RenderRequest { page, scale: 1.0, tag: 1000, ..Default::default() };
v.frame_visible = [0].into();
v.frame_queue = vec![req(0)];
v.last_queue = v.prepare_render_queue(&info, 1.0);
v.receive_result(&ctx, solid_result(req(0), 100, 130));
// Single-page view moves on to page 1: page 0 is no longer demanded.
v.begin_render_frame();
v.current = 1;
v.frame_visible = [1].into();
v.frame_queue = vec![req(1)];
assert_eq!(v.prepare_render_queue(&info, 1.0), vec![req(1)]);
assert!(v.pages.contains_key(&0), "kept while the allowance has room");
// And back: nothing to render.
v.begin_render_frame();
v.current = 0;
v.frame_visible = [0].into();
v.frame_queue = vec![req(0)];
assert!(v.prepare_render_queue(&info, 1.0).is_empty());
}
#[test]
fn thumbnail_tags_do_not_change_as_a_row_scrolls_in_or_out() {
let info = thumbnail_info(10_000);
let mut v = DocView::new(DocId(1), &info, ViewDefaults::default());
let tags = |v: &mut DocView, cells: usize| {
v.begin_render_frame();
for page in 0..cells {
v.need_thumbnail(page, true);
}
v.thumbnail_requests(&info, 2.0).into_iter().map(|r| (r.page, r.tag)).collect::<HashMap<_, _>>()
};
// 2x on a large screen: about 70 cells, give or take a row of six.
let (a, b) = (tags(&mut v, 70), tags(&mut v, 76));
assert!((0..70).all(|p| a[&p] == b[&p]), "a row more or less re-renders nothing");
}
#[test]
fn many_open_documents_share_one_raster_allowance_and_close_releases_handles() {
let ctx = egui::Context::default();
let mut app = PdfCraftApp::new();
let bytes = app.session.create_blank(612.0, 792.0, 1).unwrap();
for index in 0..6 {
app.open_bytes(&format!("synthetic-{index}.pdf"), None, bytes.as_ref().clone()).unwrap();
let view = app.views.last_mut().unwrap();
let request = RenderRequest { tag: THUMB_TAG, ..Default::default() };
view.last_queue = vec![request];
view.receive_result(&ctx, solid_result(request, 132, 171));
view.need_thumbnail(0, true);
}
app.finish_render_frame(&ctx);
let total: usize = app.views.iter().map(|v| v.thumbs.values().map(|p| texture_bytes(&p.tex)).sum::<usize>()).sum();
assert!(total <= 132 * 171 * 4);
assert!(app.views.iter().take(5).all(|v| v.thumbs.is_empty() && v.last_queue.is_empty()));
app.close_tab(5);
assert!(app.views.iter().all(|v| v.thumbs.is_empty()));
}
#[test]
fn suspended_signature_preview_releases_layers_and_rebuilds_without_editing() {
let ctx = egui::Context::default();
let mut png = std::io::Cursor::new(Vec::new());
image::RgbaImage::from_pixel(12, 4, image::Rgba([20, 30, 40, 255])).write_to(&mut png, image::ImageFormat::Png).unwrap();
let image = pdfcraft_engine::SignatureImage::from_bytes(png.get_ref()).unwrap();
let mut session = pdfcraft_engine::Session::new();
let blank = session.create_blank(300.0, 400.0, 1).unwrap();
let id = session.open_new("signature-preview.pdf", blank).unwrap();
let info = session.get(id).unwrap().info.pages[0].clone();
session.apply(id, image.edit(0, &info, [20.0, 100.0], false, "Test").unwrap()).unwrap();
let doc = session.get(id).unwrap();
let bytes = doc.bytes.clone();
let generation = doc.edit_generation();
let mut v = DocView::new(id, &doc.info, ViewDefaults::default());
v.comments.selected = Some((0, 0));
v.signature_drag.prepare(&ctx, doc, v.comments.selected, 1.0);
assert!(v.signature_drag.contains(0, 0));
v.suspend_rendering(&doc.renderer);
assert!(!v.signature_drag.contains(0, 0), "inactive tabs release the preview pool and textures");
assert_eq!(v.comments.selected, Some((0, 0)), "the user's selection survives suspension");
v.signature_drag.prepare(&ctx, doc, v.comments.selected, 1.0);
assert!(v.signature_drag.contains(0, 0), "reactivating the view rebuilds the selected image's preview");
assert!(Arc::ptr_eq(&bytes, &doc.bytes));
assert_eq!(doc.edit_generation(), generation);
assert_eq!(doc.can_undo(), Some("Add signature"));
assert!(doc.dirty);
}
#[test]
fn virtualized_organizer_keeps_insert_gaps_at_document_positions() {
use egui_kittest::kittest::Queryable;
let mut h = egui_kittest::Harness::builder().with_size(vec2(1400.0, 900.0)).build_eframe(|_cc| {
let mut app = PdfCraftApp::new();
let bytes = app.session.create_blank(300.0, 400.0, 100).unwrap();
app.open_bytes("organize-gaps.pdf", None, bytes.as_ref().clone()).unwrap();
app.views[0].organize = true;
app
});
h.run_steps(4);
assert!(h.state().views[0].thumb_demand.len() < 50);
assert_eq!(h.query_all_by_label("Insert a file at the end").count(), 0, "the last virtualized cell is not the final page");
let center = h.state().views[0].viewport_rect().center();
h.hover_at(center);
h.event(egui::Event::MouseWheel {
unit: egui::MouseWheelUnit::Point,
delta: vec2(0.0, -100_000.0),
phase: egui::TouchPhase::Move,
modifiers: egui::Modifiers::NONE,
});
// Well past egui's longest scroll animation (about 0.3 s).
h.run_steps(40);
h.get_by_label("Insert a file at the end");
let before_last = h.get_by_label("Insert a file before page 100").rect().center();
h.hover_at(before_last);
h.run_steps(2);
assert_eq!(h.state().views[0].grid_gap, Some(99), "external drops use the document index after virtualized scrolling");
assert!(h.state().views[0].thumb_demand.len() < 50);
}
#[test]
fn real_shell_pages_organizer_and_print_only_collect_current_thumbnail_consumers() {
let mut h = egui_kittest::Harness::builder().with_size(vec2(1400.0, 900.0)).build_eframe(|_cc| {
let mut app = PdfCraftApp::new();
let bytes = app.session.create_blank(612.0, 792.0, 10_000).unwrap();
app.open_bytes("synthetic-ten-thousand.pdf", None, bytes.as_ref().clone()).unwrap();
app.right = Some(RightPanel::Pages);
app
});
h.run_steps(4);
assert!((1..20).contains(&h.state().views[0].thumb_demand.len()));
h.state_mut().right = None;
h.state_mut().views[0].organize = true;
h.run_steps(2);
assert!((1..50).contains(&h.state().views[0].thumb_demand.len()));
h.state_mut().views[0].organize = false;
h.state_mut().dialog = Some(crate::Dialog::Print);
h.state_mut().print_draft.sheet = 9999;
h.run_steps(2);
assert_eq!(h.state().views[0].thumb_demand.keys().copied().collect::<Vec<_>>(), vec![9999]);
h.state_mut().print_draft.handling = crate::PrintHandling::Booklet;
h.state_mut().print_draft.sheet = 0;
h.run_steps(2);
assert_eq!(h.state().views[0].thumb_demand.len(), 2, "booklet only needs its two placed pages");
h.state_mut().dialog = None;
h.run_steps(2);
assert!(h.state().views[0].thumb_demand.is_empty());
assert!(h.state().views[0].thumbs.is_empty());
}
/// One mouse-wheel notch (a line) at `at` seconds.
fn notch(v: &mut DocView, dy: f32, at: f64) -> bool {
v.single_page_wheel(egui::MouseWheelUnit::Line, dy, egui::TouchPhase::Move, at, true)

1
crates/ui-egui/src/control.rs
View File

@ -557,6 +557,7 @@ impl Host for crate::PdfCraftApp {
// Pages picked in the organize grid or the Pages panel (empty: the current page).
"selected_pages": v.selected.iter().map(|p| p + 1).collect::<Vec<_>>(),
"auto_scrolling": v.auto_scrolling(),
"raster_memory": v.raster_memory(),
"viewport": [v.viewport_rect().min.x, v.viewport_rect().min.y, v.viewport_rect().max.x, v.viewport_rect().max.y],
"find_open": v.find.is_some(),
"page_errors": v.page_errors().iter().map(|(p, e)| json!({ "page": p + 1, "error": e })).collect::<Vec<_>>(),

8
crates/ui-egui/src/dialogs.rs
View File

@ -774,9 +774,11 @@ pub fn show(app: &mut PdfCraftApp, ctx: &egui::Context) {
let Some(doc) = app.session.get(id) else { return };
let sizes: Vec<(f64, f64)> = doc.info.pages.iter().map(|p| (p.width as f64, p.height as f64)).collect();
let labels: Vec<String> = doc.info.pages.iter().map(|p| p.label.clone()).collect();
let thumbs: std::collections::HashMap<usize, egui::TextureId> =
(0..sizes.len()).filter_map(|p| app.views[i].thumb_id(p).map(|t| (p, t))).collect();
let (go, cancel) = crate::print_ui::body(ui, &mut app.print_draft, &t, &sizes, &labels, &|p| thumbs.get(&p).copied());
let view = &mut app.views[i];
let (go, cancel) = crate::print_ui::body(ui, &mut app.print_draft, &t, &sizes, &labels, &mut |p| {
view.need_thumbnail(p, true);
view.thumb_id(p)
});
print_go = go;
close = go || cancel;
return;

32
crates/ui-egui/src/lib.rs
View File

@ -113,7 +113,7 @@ mod widgets;
use pdfcraft_engine::{DocId, Session};
pub use canvas::DocView;
pub use canvas::{DocView, RasterMemory};
pub use editing::{CloseRequest, SaveTarget};
pub use files::{ExtractDraft, FilePurpose, FileRequest, RotateDraft, SplitDraft, SplitMode, SplitPlan};
pub use recovery::{AUTOSAVE_SECS, RecoveryMeta, RecoveryStore};
@ -1762,12 +1762,6 @@ impl eframe::App for PdfCraftApp {
self.process_file_requests();
#[cfg(not(target_arch = "wasm32"))]
self.process_picked();
// Pull finished renders into textures for every open document.
for view in &mut self.views {
if let Some(doc) = self.session.get(view.id) {
view.receive(ctx, &doc.renderer);
}
}
}
fn ui(&mut self, ui: &mut egui::Ui, _frame: &mut eframe::Frame) {
@ -1778,6 +1772,9 @@ impl eframe::App for PdfCraftApp {
ctx.request_repaint();
return;
}
for view in &mut self.views {
view.begin_render_frame();
}
// The window shows the active document's name (or title, if it asks for that).
let title = self
.active
@ -1804,6 +1801,7 @@ impl eframe::App for PdfCraftApp {
// Notices too: a refused field value or a failed save must be seen in full screen.
self.show_progress(&ctx);
widgets::toast(self, &ctx);
self.finish_render_frame(&ctx);
return;
}
chrome::tab_strip(self, ui);
@ -1828,5 +1826,25 @@ impl eframe::App for PdfCraftApp {
dialogs::show(self, &ctx);
self.show_progress(&ctx);
widgets::toast(self, &ctx);
self.finish_render_frame(&ctx);
}
}
impl PdfCraftApp {
fn finish_render_frame(&mut self, ctx: &egui::Context) {
// Retire hidden documents before admitting the visible document's textures, so the
// three cache limits apply to the application, regardless of how many tabs are open.
for (i, view) in self.views.iter_mut().enumerate() {
if Some(i) != self.active
&& let Some(doc) = self.session.get(view.id)
{
view.suspend_rendering(&doc.renderer);
}
}
if let Some(view) = self.active.and_then(|i| self.views.get_mut(i))
&& let Some(doc) = self.session.get(view.id)
{
view.finish_render_frame(ctx, &doc.info, &doc.renderer);
}
}
}

3
crates/ui-egui/src/panels.rs
View File

@ -917,6 +917,9 @@ fn pages(ui: &mut egui::Ui, t: &Tokens, info: &DocInfo, view: &mut crate::DocVie
let sense = if editable { Sense::click_and_drag() } else { Sense::click() };
let (rect, resp) = ui.allocate_exact_size(vec2(w + 16.0, h + 16.0), sense);
rows.push(rect);
if rect.intersects(ui.clip_rect().expand(200.0)) {
view.need_thumbnail(i, rect.intersects(ui.clip_rect()));
}
// Only the primary button drags pages (the middle button scrolls, the secondary one opens the menu).
if resp.drag_started_by(egui::PointerButton::Primary) {
if !view.target_pages().contains(&i) {

2
crates/ui-egui/src/print_ui.rs
View File

@ -341,7 +341,7 @@ pub(crate) fn body(
t: &Tokens,
sizes: &[(f64, f64)],
labels: &[String],
thumb: &dyn Fn(usize) -> Option<egui::TextureId>,
thumb: &mut dyn FnMut(usize) -> Option<egui::TextureId>,
) -> (bool, bool) {
ui.set_width(820.0);
ui.label(egui::RichText::new(tl!("Print")).font(theme::semibold(18.0)));

62
crates/ui-egui/tests/control.rs
View File

@ -163,6 +163,50 @@ fn start_autoscroll(h: &mut Harness<'static, PdfCraftApp>, c: &ControlClient) ->
p
}
/// Measure document movement from rendered geometry without requiring an offscreen grid
/// cell to remain instantiated. Learn the grid's column count and row pitch before scrolling;
/// any currently rendered cell then identifies the same content origin. This tests actual
/// layout displacement, independently of the autoscroll velocity calculation.
#[cfg(target_os = "linux")]
struct CanvasPosition {
grid: Option<(usize, f32)>,
}
#[cfg(target_os = "linux")]
impl CanvasPosition {
fn new(h: &Harness<'static, PdfCraftApp>) -> Self {
let grid = h.state().views[0].organize.then(|| {
let first = h.get_by_label("Page 1").rect().top();
let (page, next_row) = Self::grid_cells(h).into_iter().find(|(_, top)| *top > first + 1.0).expect("a second grid row");
(page - 1, next_row - first)
});
Self { grid }
}
fn grid_cells(h: &Harness<'static, PdfCraftApp>) -> Vec<(usize, f32)> {
use egui_kittest::kittest::NodeT;
let mut cells: Vec<_> = h
.query_all_by_label_contains("Page ")
.filter_map(|node| {
let page = node.accesskit_node().label()?.strip_prefix("Page ")?.parse::<usize>().ok()?;
Some((page, node.rect().top()))
})
.collect();
cells.sort_unstable_by_key(|(page, _)| *page);
cells
}
fn top(&self, h: &Harness<'static, PdfCraftApp>) -> f32 {
match self.grid {
Some((columns, row_pitch)) => {
let (page, top) = Self::grid_cells(h).into_iter().next().expect("a rendered grid cell");
top - ((page - 1) / columns) as f32 * row_pitch
}
None => h.state().views[0].page_screen_rect(0).expect("the zoomed viewer page remains visible").top(),
}
}
}
#[cfg(target_os = "linux")]
#[test]
fn middle_click_autoscroll_latches_has_a_dead_zone_and_scrolls_both_directions() {
@ -245,14 +289,12 @@ fn autoscroll_uses_the_initial_click_position_when_input_arrives_in_one_frame()
#[cfg(target_os = "linux")]
#[test]
fn farther_from_the_click_scrolls_faster_in_the_viewer_and_page_grid() {
use egui_kittest::kittest::Queryable;
for organize in [false, true] {
let (mut h, c) = harness_pages(40);
h.state_mut().views[0].organize = organize;
h.run_steps(3);
let top = |h: &Harness<'static, PdfCraftApp>| {
if organize { h.get_by_label("Page 1").rect().top() } else { h.state().views[0].page_screen_rect(0).unwrap().top() }
};
let position = CanvasPosition::new(&h);
let top = |h: &Harness<'static, PdfCraftApp>| position.top(h);
let p = start_autoscroll(&mut h, &c);
ok(&mut h, &c, "ui.move", json!({ "x": p.x, "y": p.y + 30.0 }));
let before_near = top(&h);
@ -276,7 +318,6 @@ fn farther_from_the_click_scrolls_faster_in_the_viewer_and_page_grid() {
#[cfg(target_os = "linux")]
#[test]
fn autoscroll_uses_elapsed_frame_time_and_preserves_fractional_motion_in_both_views() {
use egui_kittest::kittest::Queryable;
for (organize, scale, zoom) in [(false, 1.0, "100"), (false, 2.0, "100"), (false, 1.0, "400"), (true, 1.0, "100"), (true, 2.0, "100")] {
for frames in [30, 60, 120, 144] {
let (mut h, c) = harness_pages(40);
@ -285,9 +326,8 @@ fn autoscroll_uses_elapsed_frame_time_and_preserves_fractional_motion_in_both_vi
h.state_mut().views[0].organize = organize;
h.run_steps(3);
assert_eq!(h.ctx.pixels_per_point(), scale);
let top = |h: &Harness<'static, PdfCraftApp>| {
if organize { h.get_by_label("Page 1").rect().top() } else { h.state().views[0].page_screen_rect(0).unwrap().top() }
};
let position = CanvasPosition::new(&h);
let top = |h: &Harness<'static, PdfCraftApp>| position.top(h);
let step = |h: &mut Harness<'static, PdfCraftApp>| {
// Deliberately differ from the harness's predicted frame interval: scrolling
// must follow the elapsed time rather than the display's predicted rate.
@ -421,15 +461,15 @@ fn the_click_that_stops_autoscroll_preserves_the_page_selection() {
#[cfg(target_os = "linux")]
#[test]
fn organize_pages_supports_autoscroll_without_selecting_or_reordering_pages() {
use egui_kittest::kittest::Queryable;
let (mut h, c) = harness_pages(40);
h.state_mut().execute("page.organize");
h.run_steps(3);
let top = h.get_by_label("Page 1").rect().top();
let position = CanvasPosition::new(&h);
let top = position.top(&h);
let p = start_autoscroll(&mut h, &c);
ok(&mut h, &c, "ui.move", json!({ "x": p.x, "y": p.y + 80.0 }));
h.run_steps(8);
assert!(h.get_by_label("Page 1").rect().top() < top - 30.0, "the organize grid scrolls");
assert!(position.top(&h) < top - 30.0, "the organize grid scrolls");
assert!(h.state().views[0].selected.is_empty());
assert!(h.state().views[0].org_drag.is_none());
ok(&mut h, &c, "ui.key", json!({ "key": "Escape" }));

2
vendor/README.md vendored
View File

@ -12,7 +12,7 @@ the fix. Vendoring copyleft code is never allowed (plan/adr/0001).
| hayro | 0.7.1 | Apache-2.0 OR MIT | (1) `RenderSettings::{x_offset, y_offset}` to render one tile of a large page; (2) images over `MAX_IMAGE_PIXELS` (2^28) are skipped (a fuzzed `/W 4294967295` hung in resampling); (3) `tiling_cell_scale` keeps a tiling pattern's cell pixmap within 3000 px a side whatever its `/XStep`/`/YStep` (a fuzzed step far beyond `/BBox` allocated 2 GB); (4) stroke widths clamped to 4 × (canvas width + height) device pixels (a fuzzed `/LW 9223372036854775807` allocated 10 GB in stroke expansion); (5) `usable_dash_pattern`: a stroke whose dash pattern could cut it into more than `MAX_DASHES_PER_STROKE` (250,000) pieces, counting kurbo's restart of the pattern at every subpath and `ClosePath`, is drawn solid (a fuzzed `/D [[11] 0]` on a line from x = 92234775807 allocated over 5 GB, `[0.000001] 0 d` on a 20 pt line over 10 GB); (6) a dash pattern with a negative or non-finite entry, or a period that isn't positive, is drawn solid (`[-1 -1] 0 d` looped for ever in kurbo); (7) `render_size` and `render_into`: render into a buffer the caller allocates (PdfCraft renders into pixel-aligned memory that the GUI takes over as texture data); `render` is `render_into` on a new pixmap | (1) `raster::tests::tiles_match_full_render`; (2) `raster::tests::absurd_image_dimensions_are_skipped`; (3) `raster::tests::tiling_cells_stay_small_whatever_the_step`; (4) `raster::tests::absurd_predictor_columns_and_line_widths_render`; (5) `raster::tests::dash_patterns_that_cut_a_stroke_into_billions_of_pieces_render`; (6) `raster::tests::invalid_dash_patterns_terminate_and_are_drawn_solid`; (7) `raster::tests::pixels_are_a_plain_renders_bytes_in_a_buffer_a_gui_can_take_over` |
| hayro-jbig2 | 0.3.0 | Apache-2.0 OR MIT | (1) at most 2^28 pixels per bitmap, page bitmap, halftone grid and grayscale image (each side could be 65535: 4.3 billion pixels decoded one by one); (2) at most 2^26 symbol pixels per symbol dictionary (a fuzzed dictionary decoded for minutes); (3) a symbol dictionary's height classes are capped at one per new symbol and its export runs at `2 × symbols + 1`, each plus `EXTRA_EMPTY_STEPS` (1024): an empty height class or a zero-length export run doesn't advance, and past the end of its data the arithmetic decoder can decode them for ever (one changed byte hung the renderer) | (1) `raster::tests::jbig2_regions_of_billions_of_pixels_are_refused`; (2) the nightly `cargo xtask fuzz` finding, which now finishes in about a second; (3) `raster::tests::jbig2_symbol_dictionaries_that_never_advance_terminate` |
| lopdf | 0.45.0 | MIT | (1) PNG predictor rows longer than the data are refused before their buffers are allocated (a fuzzed `/Columns 4294967295` allocated two 4 GiB rows), and empty frames return without row buffers; (2) predictor row widths use saturating arithmetic or reject overflow; (3) sub-byte TIFF Predictor 2 scratch is bounded by the samples present in the input instead of the declared `/Colors`, aligned with [lopdf#589](https://github.com/J-F-Liu/lopdf/pull/589) | (1) `inspect::tests::png_predictor_rows_longer_than_the_data_are_refused_before_allocating`, `inspect::tests::empty_png_predictor_frames_do_not_allocate_rows`; (2) `inspect::tests::tiff_subbyte_predictor_row_width_overflow_is_refused`; (3) `inspect::tests::tiff_subbyte_predictor_scratch_is_bounded_by_available_samples`, `inspect::tests::tiff_subbyte_predictors_keep_components_rows_and_padding` |
| hayro-syntax | 0.7.2 | Apache-2.0 OR MIT | (1) Page-tree cycle guard in `resolve_pages` (visited set + depth cap): a `/Kids` loop among compressed objects overflowed the stack and aborted the process; (2) `ccitt_size_ok`: CCITT images wider than 2^20 columns or over 2^28 pixels are not decoded (a fuzzed `/Columns 4294967295` allocated 4 GiB); (3) Flate/LZW predictor row widths are saturating `usize`, a zero width is refused and a row longer than the data is not allocated (a fuzzed `/Columns 9223372036854775807` wrapped to a 2^61-byte allocation and aborted); (4) the inline-image end-of-data check (`EI`) doesn't nest inside its own look-ahead and examines at most 256 candidates and steps (50 images with "EI" in their data ran for over ten minutes); (5) a stream whose `/Length` is wrong is recovered only when `stream` is the next token — scanning ahead to a later object's `stream` made an indirect appearance-state dictionary (Quartz checkboxes: `/AP /N` → `<< /Yes … /Off … >>`) render as a non-form, so the widget was skipped; (6) JPX rescaling uses an exact integer sample range for 1–32-bit components (32 shifted past u32, and floating-point maxima at 25–31 bits wrapped to zero), and refuses zero-sized rows; (7) `decode_limit`: Flate (through flate2 and the fallback inflater), LZW and RunLength never write past `MAX_DECODED_STREAM` (256 MiB), or, for the last filter of an image of known size, past twice its pixel bytes (a predictor tag byte per byte), at most 1 GiB; `Stream::decoded_within` for budgeted callers; a page's `/Contents` array shares one `MAX_DECODED_STREAM` (a 2 MB page whose content stream or 8 × 8 image inflated to 2 GB took the renderer past 3 GB in under a second; naming the stream twenty times in `/Contents`, 5 GB); (8) high-precision numbers (jsPDF MediaBox coordinates with over 19 digits) parse without overflowing the u64 mantissa: fractional digits past it are dropped, integer digits past it scale the value (a real, kept finite) | (1) `cos/tests/writer.rs::page_tree_loops_in_object_streams_do_not_crash_readers`; (2) `raster::tests::absurd_mask_and_fax_sizes_are_refused_before_decoding`; (3) `raster::tests::absurd_predictor_columns_and_line_widths_render`; (4) `raster::tests::inline_images_with_ei_in_their_data_parse_in_linear_time`; (5) `raster::tests::indirect_appearance_state_dict_is_not_another_stream`; (6) `raster::tests::jpx_stream_bit_depth_boundaries_do_not_panic`, `raster::tests::jpx_32_bit_image_does_not_prevent_other_page_content_from_rendering`, `raster::tests::jpx_gray_rgb_and_alpha_keep_eight_bit_samples`, `raster::tests::jpx_streams_reject_out_of_range_bit_depths`; (7) `raster::tests::decompression_bombs_stop_at_the_stream_limit`, `raster::tests::page_contents_and_nested_forms_share_one_budget`; (8) `inspect::tests::jspdf_high_precision_mediabox_does_not_overflow_dimensions`, `object::number::tests::high_precision_float_does_not_overflow_mantissa` (hayro-syntax) |
| hayro-syntax | 0.7.2 | Apache-2.0 OR MIT | (1) Page-tree cycle guard in `resolve_pages` (visited set + depth cap): a `/Kids` loop among compressed objects overflowed the stack and aborted the process; (2) `ccitt_size_ok`: CCITT images wider than 2^20 columns or over 2^28 pixels are not decoded (a fuzzed `/Columns 4294967295` allocated 4 GiB); (3) Flate/LZW predictor row widths are saturating `usize`, a zero width is refused and a row longer than the data is not allocated (a fuzzed `/Columns 9223372036854775807` wrapped to a 2^61-byte allocation and aborted); (4) the inline-image end-of-data check (`EI`) doesn't nest inside its own look-ahead and examines at most 256 candidates and steps (50 images with "EI" in their data ran for over ten minutes); (5) a stream whose `/Length` is wrong is recovered only when `stream` is the next token — scanning ahead to a later object's `stream` made an indirect appearance-state dictionary (Quartz checkboxes: `/AP /N` → `<< /Yes … /Off … >>`) render as a non-form, so the widget was skipped; (6) JPX rescaling uses an exact integer sample range for 1–32-bit components (32 shifted past u32, and floating-point maxima at 25–31 bits wrapped to zero), and refuses zero-sized rows; (7) `decode_limit`: Flate (through flate2 and the fallback inflater), LZW and RunLength never write past `MAX_DECODED_STREAM` (256 MiB), or, for the last filter of an image of known size, past twice its pixel bytes (a predictor tag byte per byte), at most 1 GiB; `Stream::decoded_within` for budgeted callers; a page's `/Contents` array shares one `MAX_DECODED_STREAM` (a 2 MB page whose content stream or 8 × 8 image inflated to 2 GB took the renderer past 3 GB in under a second; naming the stream twenty times in `/Contents`, 5 GB); (8) high-precision numbers (jsPDF MediaBox coordinates with over 19 digits) parse without overflowing the u64 mantissa: fractional digits past it are dropped, integer digits past it scale the value (a real, kept finite); (9) large dictionary indexes are built on first use and shared across repeated reads within each PDF (#307); the validated-byte cache is bounded, reader contexts remain separate, repairs and inline/content dictionaries bypass sharing, and small dictionaries stay eager; (10) decoded object-stream slots publish once under contention, cold loading rejects recursive dependencies, and xref repair serializes without holding recursive read guards; this permits one parser per native render pool with private worker caches; (11) `sync.rs`: a poisoned lock is recovered instead of unwrapped (a worker panic no longer cascades into every reader of the same PDF), and xref repair takes a blocking write lock instead of asserting a non-blocking one; (12) `SegmentList` (`util.rs`) returns `None` for an out-of-range index instead of panicking | (1) `cos/tests/writer.rs::page_tree_loops_in_object_streams_do_not_crash_readers`; (2) `raster::tests::absurd_mask_and_fax_sizes_are_refused_before_decoding`; (3) `raster::tests::absurd_predictor_columns_and_line_widths_render`; (4) `raster::tests::inline_images_with_ei_in_their_data_parse_in_linear_time`; (5) `raster::tests::indirect_appearance_state_dict_is_not_another_stream`; (6) `raster::tests::jpx_stream_bit_depth_boundaries_do_not_panic`, `raster::tests::jpx_32_bit_image_does_not_prevent_other_page_content_from_rendering`, `raster::tests::jpx_gray_rgb_and_alpha_keep_eight_bit_samples`, `raster::tests::jpx_streams_reject_out_of_range_bit_depths`; (7) `raster::tests::decompression_bombs_stop_at_the_stream_limit`, `raster::tests::page_contents_and_nested_forms_share_one_budget`; (8) `inspect::tests::jspdf_high_precision_mediabox_does_not_overflow_dimensions`, `object::number::tests::high_precision_float_does_not_overflow_mantissa` (hayro-syntax); (9) `object::dict::lazy_index_tests`, `raster::tests::large_resource_indexes_preserve_pixels_and_text`; (10) `data::tests`, `xref::tests::concurrent_cold_object_stream_reads_all_succeed`, `raster::tests::shared_parser_is_lazy_and_initializes_once_under_contention`; (11) `data::tests::panicking_initializer_wakes_waiters_and_prevents_retries`, `xref::tests::concurrent_cold_object_stream_reads_all_succeed`; (12) `util::tests::segment_list_rejects_out_of_range_indices` |
| winit | 0.30.13 | Apache-2.0 | (1) on Windows 11 (build 22000 and later), `WM_DPICHANGED` applies the rect Windows suggests. It used to nudge the window onto the monitor `MonitorFromWindow` names, which during a drag is still the monitor being left; the nudge sent another `WM_DPICHANGED`, so a window dragged to a monitor with another scale factor grew and stayed on the wrong monitor at the wrong scale (#324). Backport of winit master 488c036a ([winit#4341](https://github.com/rust-windowing/winit/pull/4341)), which no 0.30 release has; Windows 10 keeps the old adjustment, as on winit master. The published crate's asset files (`docs/res/*.svg`, `examples/data/*.png`) are left out, so its `window` example doesn't build | (1) `pdfcraft` `tests::winit_carries_the_windows_11_monitor_scale_fix` keeps the patch in the build; the drag itself needs two monitors with different scale factors on Windows 11 |
Temporary: hayro is the bootstrap renderer and lopdf the bootstrap inspector (ADR-0004); M2.6 replaces them with our own devices and `cos`/`model`.

181
vendor/hayro-syntax/src/data.rs vendored
View File

@ -67,6 +67,7 @@ pub(crate) struct Data {
// 32 segments are more than enough as we can't have more objects than this.
decoded: SegmentList<Option<Vec<u8>>, 32>,
map: Mutex<HashMap<ObjectIdentifier, usize>>,
loading: LoadGate,
}
impl Debug for Data {
@ -82,6 +83,7 @@ impl Data {
data,
decoded: SegmentList::new(),
map: Mutex::new(HashMap::new()),
loading: LoadGate::default(),
}
}
@ -92,22 +94,171 @@ impl Data {
/// Get access to the data of a decoded object stream.
pub(crate) fn get_with(&self, id: ObjectIdentifier, ctx: &ReaderContext<'_>) -> Option<&[u8]> {
if let Some(&idx) = self.map.get().get(&id) {
self.decoded.get(idx)?.as_deref()
} else {
// Block scope to keep the lock short-lived.
let idx = {
let mut locked = self.map.get();
let idx = locked.len();
locked.insert(id, idx);
idx
};
self.decoded
.get_or_init(idx, || {
let stream = ctx.xref().get_with::<Stream<'_>>(id, ctx)?;
stream.decoded().ok().map(Cow::into_owned)
// PdfCraft patch: (#307, see vendor/README.md hayro-syntax (10)) register exactly once:
// racing misses must not allocate different slots
// for the same id (or reuse the resulting hole for another stream).
let idx = {
let mut map = self.map.get();
let next = map.len();
*map.entry(id).or_insert(next)
};
if let Some(decoded) = self.decoded.get(idx) {
return decoded.as_deref();
}
// Only cold initialization is serialized. Nested streams may depend on
// one another; the gate rejects cycles without blocking on our own cell.
let _loading = self.loading.enter(id)?;
self.decoded
.get_or_init(idx, || {
let stream = ctx.xref().get_with::<Stream<'_>>(id, ctx)?;
stream.decoded().ok().map(Cow::into_owned)
})?
.as_deref()
}
}
/// PdfCraft patch: (#307) one cold initializer at a time prevents cross-thread dependency deadlocks.
/// The owner may recursively load different streams, up to a fixed depth. No
/// state mutex is held while parsing or decoding input. no_std is single-threaded
/// (its Arc/locks are Rc/RefCell), so it needs the same cycle/depth checks only.
#[derive(Default)]
struct LoadGate {
state: Mutex<LoadState>,
#[cfg(feature = "std")]
ready: std::sync::Condvar,
}
#[derive(Default)]
struct LoadState {
active: Vec<ObjectIdentifier>,
#[cfg(feature = "std")]
owner: Option<std::thread::ThreadId>,
failed: bool,
}
impl LoadGate {
fn enter(&self, id: ObjectIdentifier) -> Option<LoadGuard<'_>> {
let mut state = self.state.get();
#[cfg(feature = "std")]
while !state.failed
&& state
.owner
.is_some_and(|owner| owner != std::thread::current().id())
{
state = self.ready.wait(state).unwrap_or_else(|p| p.into_inner());
}
if state.failed || state.active.len() >= 64 || state.active.contains(&id) {
return None;
}
#[cfg(feature = "std")]
{
state.owner = Some(std::thread::current().id());
}
state.active.push(id);
Some(LoadGuard(self))
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn recursive_loads_reject_cycles_and_limit_depth() {
let gate = LoadGate::default();
let mut guards = Vec::new();
for n in 0..64 {
guards.push(
gate.enter(ObjectIdentifier::new(n, 0))
.expect("nested distinct stream"),
);
}
assert!(gate.enter(ObjectIdentifier::new(64, 0)).is_none());
assert!(gate.enter(ObjectIdentifier::new(0, 0)).is_none());
while guards.pop().is_some() {}
assert!(gate.enter(ObjectIdentifier::new(0, 0)).is_some());
}
#[cfg(feature = "std")]
#[test]
fn competing_dependency_chains_cannot_wait_on_each_other() {
let gate = Arc::new(LoadGate::default());
let barrier = Arc::new(std::sync::Barrier::new(16));
let jobs: Vec<_> = (0..16)
.map(|n| {
let gate = gate.clone();
let barrier = barrier.clone();
std::thread::spawn(move || {
barrier.wait();
let (a, b) = (
ObjectIdentifier::new(n % 2, 0),
ObjectIdentifier::new(1 - n % 2, 0),
);
let _a = gate.enter(a).unwrap();
let _b = gate.enter(b).unwrap();
assert!(gate.enter(a).is_none());
})
.as_deref()
})
.collect();
for job in jobs {
job.join().unwrap();
}
}
#[cfg(feature = "std")]
#[test]
fn panicking_initializer_wakes_waiters_and_prevents_retries() {
use std::{
panic::{AssertUnwindSafe, catch_unwind},
sync::mpsc,
time::Duration,
};
let gate = Arc::new(LoadGate::default());
let (ready_tx, ready_rx) = mpsc::channel();
let (done_tx, done_rx) = mpsc::channel();
let mut jobs = Vec::new();
let result = catch_unwind(AssertUnwindSafe(|| {
let _guard = gate.enter(ObjectIdentifier::new(1, 0)).unwrap();
for _ in 0..8 {
let (gate, ready, done) = (gate.clone(), ready_tx.clone(), done_tx.clone());
jobs.push(std::thread::spawn(move || {
ready.send(()).unwrap();
done.send(gate.enter(ObjectIdentifier::new(2, 0)).is_none())
.unwrap();
}));
}
for _ in 0..8 {
ready_rx.recv_timeout(Duration::from_secs(2)).unwrap();
}
panic!("injected decode failure");
}));
assert!(result.is_err());
for _ in 0..8 {
assert!(done_rx.recv_timeout(Duration::from_secs(2)).unwrap());
}
for job in jobs {
job.join().unwrap();
}
assert!(gate.enter(ObjectIdentifier::new(1, 0)).is_none());
assert!(gate.enter(ObjectIdentifier::new(3, 0)).is_none());
}
}
struct LoadGuard<'a>(&'a LoadGate);
impl Drop for LoadGuard<'_> {
fn drop(&mut self) {
let mut state = self.0.state.get();
state.active.pop();
#[cfg(feature = "std")]
{
// OnceLock itself permits a retry after a panic. The parser does not:
// wake all waiters and fail closed until the caller replaces the Pdf.
state.failed |= std::thread::panicking();
if state.active.is_empty() {
state.owner = None;
}
self.0.ready.notify_all();
}
}
}

258
vendor/hayro-syntax/src/object/dict.rs vendored
View File

@ -6,8 +6,9 @@ use crate::object::{Name, ObjectIdentifier};
use crate::object::{Object, ObjectLike};
use crate::reader::Reader;
use crate::reader::{Readable, ReaderContext, ReaderExt, Skippable};
use crate::sync::Arc;
use crate::sync::FxHashMap;
use crate::sync::{Arc, OnceLock};
use alloc::boxed::Box;
use alloc::format;
use alloc::vec::Vec;
use core::fmt::{Debug, Formatter};
@ -128,10 +129,27 @@ impl<'a> Dict<'a> {
}
}
fn offsets(&self) -> Option<&FxHashMap<Name<'a>, usize>> {
fn offsets(&self) -> Option<&FxHashMap<Name<'static>, usize>> {
match &self.0 {
Inner::Empty => None,
Inner::Some(repr) => Some(&repr.offsets),
Inner::Some(repr) => Some(repr.offsets.get_or_init(|| {
// PdfCraft patch: (#307) large resource dictionaries are read for every
// page at open, but most are never looked up. Index them only on demand.
// The first pass already validated the same immutable bytes with this
// context. Keep the same parser, including duplicate-key and repair rules.
let mut offsets = new_offsets();
let mut reader = Reader::new(repr.data);
let (start, end): (Option<&[u8]>, &[u8]) = if repr.inline {
(None, b"ID")
} else {
(Some(b"<<"), b">>")
};
let _ = parse_dict_with(&mut reader, &repr.ctx, start, end, |name, offset, _| {
offsets.insert(name.into_owned(), offset);
Some(())
});
offsets
})),
}
}
}
@ -190,36 +208,114 @@ impl<'a> Readable<'a> for Dict<'a> {
}
}
// PdfCraft patch: (#307) retain the fast single-pass index for ordinary small
// dictionaries. Above this entry count, validate without retaining a hash table.
const EAGER_DICT_ENTRIES: usize = 32;
fn new_offsets() -> FxHashMap<Name<'static>, usize> {
#[cfg(feature = "std")]
return FxHashMap::with_capacity_and_hasher(8, rustc_hash::FxBuildHasher);
#[cfg(not(feature = "std"))]
FxHashMap::new()
}
pub(crate) type SharedIndex = Arc<OnceLock<FxHashMap<Name<'static>, usize>>>;
#[derive(Debug)]
pub(crate) struct CachedDict {
data: Box<[u8]>,
offsets: SharedIndex,
}
// PdfCraft patch: share a validated large dictionary's index across pages. The cache
// belongs to the xref, owns its keys, and is bounded even on documents with many unique
// dictionaries. Comparing the bytes also makes address reuse harmless. No reader context
// is cached: decryption, object identity and reference-cycle tracking stay with each read.
#[derive(Debug, Default)]
pub(crate) struct DictCache {
entries: FxHashMap<usize, Arc<CachedDict>>,
bytes: usize,
}
impl DictCache {
fn get(&self, data: &[u8]) -> Option<Arc<CachedDict>> {
self.entries
.get(&(data.as_ptr() as usize))
.filter(|cached| data.starts_with(&cached.data))
.cloned()
}
fn insert(&mut self, data: &[u8], offsets: SharedIndex) {
const MAX_BYTES: usize = 8 << 20;
if self.entries.len() < 256 && data.len() <= MAX_BYTES.saturating_sub(self.bytes) {
self.bytes += data.len();
self.entries.insert(
data.as_ptr() as usize,
Arc::new(CachedDict {
data: data.into(),
offsets,
}),
);
}
}
}
fn read_inner<'a>(
r: &mut Reader<'a>,
ctx: &ReaderContext<'a>,
start_tag: Option<&[u8]>,
end_tag: &[u8],
) -> Option<Dict<'a>> {
// TODO: Figure out how to
// 1) Make dictionaries easily cloneable without wrapping in Arc.
// 2) Maybe have an efficient per-document allocator pool for hashmaps
// that can be reused.
#[cfg(feature = "std")]
let mut offsets = FxHashMap::with_capacity_and_hasher(8, rustc_hash::FxBuildHasher);
#[cfg(not(feature = "std"))]
let mut offsets = FxHashMap::new();
let cacheable = start_tag.is_some() && !ctx.in_content_stream();
if cacheable {
let tail = r.tail()?;
if let Some(cached) = ctx
.xref()
.with_dict_cache(|cache| cache.get(tail))
.flatten()
{
let data = r.read_bytes(cached.data.len())?;
return Some(Dict(Inner::Some(Arc::new(Repr {
data,
offsets: cached.offsets.clone(),
inline: false,
ctx: ctx.clone(),
}))));
}
}
let mut offsets = Some(new_offsets());
let data = parse_dict_with(
let (data, repaired) = parse_dict_with(
r,
ctx,
start_tag,
end_tag,
#[inline]
|name, offset, _| {
offsets.insert(name, offset);
if let Some(index) = &mut offsets {
index.insert(name.into_owned(), offset);
if index.len() > EAGER_DICT_ENTRIES {
offsets = None;
}
}
Some(())
},
)?;
let index = Arc::new(OnceLock::new());
if let Some(offsets) = offsets {
let _ = index.set(offsets);
}
// Repair may read objects through the current reference context; only cache the
// context-independent validating path. Inline/content dictionaries stay uncached.
if cacheable && !repaired && data.len() >= 1024 {
ctx.xref()
.with_dict_cache(|cache| cache.insert(data, index.clone()));
}
Some(Dict(Inner::Some(Arc::new(Repr {
data,
offsets,
offsets: index,
inline: start_tag.is_none(),
ctx: ctx.clone(),
}))))
}
@ -242,7 +338,7 @@ pub(crate) fn probe_dict<'a>(
let mut has_root = false;
let mut has_type = false;
let data = parse_dict_with(
let (data, _) = parse_dict_with(
r,
ctx,
start_tag,
@ -272,12 +368,13 @@ fn parse_dict_with<'a, F>(
start_tag: Option<&[u8]>,
end_tag: &[u8],
mut on_entry: F,
) -> Option<&'a [u8]>
) -> Option<(&'a [u8], bool)>
where
F: FnMut(Name<'a>, usize, &Reader<'a>) -> Option<()>,
{
let dict_data = r.tail()?;
let start_offset = r.offset();
let mut repaired = false;
// Inline image dictionaries don't start with '<<'.
if let Some(start_tag) = start_tag {
@ -292,13 +389,14 @@ where
r.forward_tag(end_tag)?;
let end_offset = r.offset() - start_offset;
break Some(&dict_data[..end_offset]);
break Some((&dict_data[..end_offset], repaired));
} else {
let Some(name) = r.read_without_context::<Name<'_>>() else {
if start_tag.is_some() {
// In case there is garbage in-between, be lenient and just try to skip it.
// But only do this if we are parsing a proper dictionary as opposed to an
// inline dictionary.
repaired = true;
r.read::<Object<'_>>(ctx)?;
continue;
} else {
@ -330,7 +428,8 @@ object!(Dict<'a>, Dict);
struct Repr<'a> {
data: &'a [u8],
offsets: FxHashMap<Name<'a>, usize>,
offsets: SharedIndex,
inline: bool,
ctx: ReaderContext<'a>,
}
@ -348,6 +447,129 @@ impl<'a> Readable<'a> for InlineImageDict<'a> {
}
}
#[cfg(test)]
mod lazy_index_tests {
use super::*;
use crate::object::{Array, FromBytes};
use alloc::string::String;
fn entries() -> String {
let mut data = String::new();
for i in 0..100 {
data.push_str(&format!("/Key{i} {i} "));
}
// Escaped and long names, duplicate keys, null and nested values keep
// exactly the same semantics after the second (index-building) pass.
data.push_str("/Key99 123 /Esc#61ped 7 /ThisNameIsLongerThanTheInlineNameBuffer 9 /Nil null /Nested << /N [1 2 3] >> ");
data
}
#[test]
fn large_indexes_are_deferred_and_clones_share_the_result() {
let data = format!("<< {} >>", entries());
let dict = Dict::from_bytes(data.as_bytes()).unwrap();
let Inner::Some(repr) = &dict.0 else {
panic!("nonempty")
};
assert!(repr.offsets.get().is_none());
let cloned = dict.clone();
assert_eq!(cloned.get::<i32>(b"Key99"), Some(123));
assert!(repr.offsets.get().is_some());
assert_eq!(dict.len(), 104);
assert_eq!(dict.get::<i32>(b"Escaped"), Some(7));
assert_eq!(
dict.get::<i32>(b"ThisNameIsLongerThanTheInlineNameBuffer"),
Some(9)
);
assert!(dict.contains_key(b"Nil"));
assert!(!dict.contains_key(b"Missing"));
assert_eq!(
dict.get::<Dict<'_>>(b"Nested")
.unwrap()
.get::<Array<'_>>(b"N")
.unwrap()
.iter::<i32>()
.collect::<Vec<_>>(),
[1, 2, 3]
);
assert_eq!(dict.keys().count(), dict.len());
assert_eq!(dict.entries().count(), dict.len());
assert!(core::ptr::eq(
dict.offsets().unwrap(),
cloned.offsets().unwrap()
));
}
#[test]
fn small_indexes_stay_eager_and_large_inline_dicts_work() {
let dict = Dict::from_bytes(b"<< /Key 42 >>").unwrap();
let Inner::Some(repr) = &dict.0 else {
panic!("nonempty")
};
assert!(repr.offsets.get().is_some());
assert_eq!(dict.get::<i32>(b"Key"), Some(42));
let data = format!("{} ID", entries());
let mut reader = Reader::new(data.as_bytes());
let inline = InlineImageDict::read(&mut reader, &ReaderContext::dummy()).unwrap();
assert_eq!(inline.0.get::<i32>(b"Key99"), Some(123));
assert_eq!(inline.0.len(), 104);
}
#[test]
fn large_dicts_are_still_validated_before_any_lookup() {
let data = format!("<< {} /Broken [", entries());
assert!(Dict::from_bytes(data.as_bytes()).is_none());
}
#[test]
fn repeated_resource_reads_share_indexes_but_keep_object_context() {
let data = format!(
"%PDF-1.7\n1 0 obj << /Type /Catalog /Pages 2 0 R >> endobj\n2 0 obj << /Type /Pages /Kids [3 0 R 4 0 R] /Count 2 >> endobj\n3 0 obj << /Type /Page /Parent 2 0 R /MediaBox [0 0 100 100] /Resources 5 0 R >> endobj\n4 0 obj << /Type /Page /Parent 2 0 R /MediaBox [0 0 100 100] /Resources 5 0 R >> endobj\n5 0 obj << /XObject << {} >> >> endobj\ntrailer << /Root 1 0 R >>\n%%EOF",
entries()
);
let pdf = crate::Pdf::new(data.into_bytes()).unwrap();
let a = pdf
.xref()
.get::<Dict<'_>>(ObjectIdentifier::new(5, 0))
.unwrap()
.get::<Dict<'_>>(b"XObject")
.unwrap();
let b = pdf
.xref()
.get::<Dict<'_>>(ObjectIdentifier::new(5, 0))
.unwrap()
.get::<Dict<'_>>(b"XObject")
.unwrap();
let (Inner::Some(a_repr), Inner::Some(b_repr)) = (&a.0, &b.0) else {
panic!("nonempty")
};
assert!(Arc::ptr_eq(&a_repr.offsets, &b_repr.offsets));
assert_eq!(a.obj_id(), Some(ObjectIdentifier::new(5, 0)));
assert_eq!(a.get::<i32>(b"Key99"), Some(123));
assert_eq!(b.get::<i32>(b"Escaped"), Some(7));
assert!(core::ptr::eq(a.offsets().unwrap(), b.offsets().unwrap()));
}
#[test]
fn index_cache_is_bounded_and_rejects_changed_or_truncated_bytes() {
let mut cache = DictCache::default();
let mut data = format!("<< {} >>", entries()).into_bytes();
cache.insert(&data, Arc::new(OnceLock::new()));
assert!(cache.get(&data).is_some());
assert!(cache.get(&data[..data.len() - 1]).is_none());
data[3] = b'Z';
assert!(cache.get(&data).is_none());
let buffers: Vec<_> = (0..300)
.map(|i| format!("<< /Key{i} 1 >>").into_bytes())
.collect();
for data in &buffers {
cache.insert(data, Arc::new(OnceLock::new()));
}
assert_eq!(cache.entries.len(), 256);
assert!(cache.bytes <= 8 << 20);
}
}
/// A collection of possible keys in a PDF dictionary. Copied and adapted from `PDFBox`.
#[allow(missing_docs)]
pub mod keys {

10
vendor/hayro-syntax/src/object/name.rs vendored
View File

@ -72,6 +72,16 @@ impl Ord for Name<'_> {
}
impl<'a> Name<'a> {
// PdfCraft patch: (#307) lazy dictionary indexes cannot borrow through a
// OnceLock without making the enclosing PDF pages invariant in their lifetime.
// Ordinary PDF keys fit inline, so owning these bytes needs no allocation.
pub(crate) fn into_owned(self) -> Name<'static> {
Name(NameInner::Owned(match self.0 {
NameInner::Borrowed(data) => SmallVec::from_slice(data),
NameInner::Owned(data) => data,
}))
}
/// Create a new name from a sequence of bytes.
#[inline]
pub fn new(data: &'a [u8]) -> Option<Self> {

28
vendor/hayro-syntax/src/sync.rs vendored
View File

@ -60,7 +60,9 @@ pub(crate) trait MutexExt<T> {
#[cfg(feature = "std")]
impl<T> MutexExt<T> for Mutex<T> {
fn get(&self) -> MutexGuard<'_, T> {
self.lock().unwrap()
// PdfCraft patch: (#307) cached values remain owned after an unwinding caller; do not cascade a
// worker panic into unrelated readers. Initialization has its own failure state.
self.lock().unwrap_or_else(|poisoned| poisoned.into_inner())
}
}
@ -73,22 +75,20 @@ impl<T> MutexExt<T> for Mutex<T> {
pub(crate) trait RwLockExt<T> {
fn get(&self) -> RwLockReadGuard<'_, T>;
fn try_get(&self) -> Option<RwLockReadGuard<'_, T>>;
fn try_put(&self) -> Option<RwLockWriteGuard<'_, T>>;
fn put(&self) -> RwLockWriteGuard<'_, T>;
}
#[cfg(feature = "std")]
impl<T> RwLockExt<T> for RwLock<T> {
// PdfCraft patch: (#307) poisoned locks are recovered rather than unwrapped, and the
// non-blocking try_get/try_put (which repair asserted could not fail) become blocking.
fn get(&self) -> RwLockReadGuard<'_, T> {
self.read().unwrap()
self.read().unwrap_or_else(|poisoned| poisoned.into_inner())
}
fn try_get(&self) -> Option<RwLockReadGuard<'_, T>> {
self.try_read().ok()
}
fn try_put(&self) -> Option<RwLockWriteGuard<'_, T>> {
self.try_write().ok()
fn put(&self) -> RwLockWriteGuard<'_, T> {
self.write()
.unwrap_or_else(|poisoned| poisoned.into_inner())
}
}
@ -98,11 +98,7 @@ impl<T> RwLockExt<T> for RwLock<T> {
self.borrow()
}
fn try_get(&self) -> Option<RwLockReadGuard<'_, T>> {
Some(self.borrow())
}
fn try_put(&self) -> Option<RwLockWriteGuard<'_, T>> {
Some(self.borrow_mut())
fn put(&self) -> RwLockWriteGuard<'_, T> {
self.borrow_mut()
}
}

46
vendor/hayro-syntax/src/util.rs vendored
View File

@ -139,8 +139,8 @@ pub(crate) fn findr_needle(haystack: &[u8], needle: &[u8]) -> Option<usize> {
/// Indices should be used in order. Usage of higher indices implies more memory
/// usage (even if lower indices are not in use).
///
/// The capacity is limited at 2^C - 1. Calling `get_or_init` with a higher
/// index will lead to a panic.
/// The capacity is limited at 2^C - 1. Out-of-range indices return `None` (PdfCraft patch:
/// upstream panics; #307 reaches this with object-stream slot numbers from the file).
pub(crate) struct SegmentList<T, const C: usize>([OnceLock<Box<[OnceLock<T>]>>; C]);
impl<T, const C: usize> SegmentList<T, C> {
@ -149,32 +149,28 @@ impl<T, const C: usize> SegmentList<T, C> {
}
pub(crate) fn get(&self, i: usize) -> Option<&T> {
let (s, k) = self.locate(i);
let segment = self.0[s as usize].get()?;
let (s, k) = self.locate(i)?;
let segment = self.0.get(s as usize)?.get()?;
segment.get(k)?.get()
}
#[track_caller]
pub(crate) fn get_or_init(&self, i: usize, f: impl FnOnce() -> T) -> &T {
let (s, k) = self.locate(i);
let segment = self
.0
.get(s as usize)
.expect("segment list is out of capacity")
.get_or_init(|| {
(0..2_usize.pow(s))
.map(|_| OnceLock::new())
.collect::<Vec<_>>()
.into_boxed_slice()
});
segment[k].get_or_init(f)
pub(crate) fn get_or_init(&self, i: usize, f: impl FnOnce() -> T) -> Option<&T> {
let (s, k) = self.locate(i)?;
let segment = self.0.get(s as usize)?.get_or_init(|| {
(0..2_usize.pow(s))
.map(|_| OnceLock::new())
.collect::<Vec<_>>()
.into_boxed_slice()
});
Some(segment.get(k)?.get_or_init(f))
}
fn locate(&self, i: usize) -> (u32, usize) {
let power = (i + 2).next_power_of_two() / 2;
fn locate(&self, i: usize) -> Option<(u32, usize)> {
let power = i.checked_add(2)?.checked_next_power_of_two()? / 2;
let s = power.trailing_zeros();
let k = i - ((1 << s) - 1);
(s, k)
Some((s, k))
}
}
@ -201,6 +197,16 @@ mod tests {
}
}
#[test]
fn segment_list_rejects_out_of_range_indices() {
let list = SegmentList::<u8, 2>::new();
assert_eq!(list.get_or_init(2, || 7), Some(&7));
for index in [3, 20, usize::MAX - 1, usize::MAX] {
assert_eq!(list.get(index), None);
assert_eq!(list.get_or_init(index, || 9), None);
}
}
#[test]
fn test_findr_needle() {
assert_eq!(findr_needle(b"abc foo def foo xyz", b"foo"), Some(12));

256
vendor/hayro-syntax/src/xref.rs vendored
View File

@ -10,7 +10,7 @@ use crate::object::dict::keys::{
AUTHOR, CREATION_DATE, CREATOR, ENCRYPT, FIRST, ID, INDEX, INFO, KEYWORDS, MOD_DATE, N,
OCPROPERTIES, PAGES, PREV, PRODUCER, ROOT, SIZE, SUBJECT, TITLE, TYPE, VERSION, W, XREF_STM,
};
use crate::object::dict::probe_dict;
use crate::object::dict::{DictCache, probe_dict};
use crate::object::indirect::IndirectObject;
use crate::object::{Array, MaybeRef};
use crate::object::{DateTime, Dict};
@ -18,7 +18,7 @@ use crate::object::{Object, ObjectLike};
use crate::pdf::PdfVersion;
use crate::reader::Reader;
use crate::reader::{Readable, ReaderContext, ReaderExt};
use crate::sync::{Arc, FxHashMap, RwLock, RwLockExt};
use crate::sync::{Arc, FxHashMap, Mutex, RwLock, RwLockExt};
use crate::trivia::is_white_space_character;
use crate::util::findr_needle;
use crate::{PdfData, object};
@ -274,6 +274,21 @@ const DUMMY_XREF: XRef = XRef(Inner::Dummy);
pub struct XRef(Inner);
impl XRef {
// PdfCraft patch: keep the bounded dictionary-index cache local to this PDF.
pub(crate) fn with_dict_cache<T>(&self, f: impl FnOnce(&mut DictCache) -> T) -> Option<T> {
let Inner::Some(repr) = &self.0 else {
return None;
};
#[cfg(feature = "std")]
let mut cache = repr
.dict_cache
.lock()
.unwrap_or_else(|poison| poison.into_inner());
#[cfg(not(feature = "std"))]
let mut cache = repr.dict_cache.try_borrow_mut().ok()?;
Some(f(&mut cache))
}
fn new(
data: PdfData,
xref_map: XrefMap,
@ -294,6 +309,7 @@ impl XRef {
metadata: Arc::new(Metadata::default()),
trailer_data,
password: password.to_vec(),
dict_cache: Arc::new(Mutex::new(DictCache::default())),
})));
// We read the trailer twice, once to determine the encryption used and then a second
@ -381,16 +397,6 @@ impl XRef {
Ok(xref)
}
fn is_repaired(&self) -> bool {
match &self.0 {
Inner::Dummy => false,
Inner::Some(r) => {
let locked = r.map.get();
locked.repaired
}
}
}
pub(crate) fn dummy() -> &'static Self {
&DUMMY_XREF
}
@ -479,15 +485,21 @@ impl XRef {
pub(crate) fn repair(&self) {
let Inner::Some(r) = &self.0 else {
unreachable!();
return;
};
let mut locked = r.map.try_put().unwrap();
assert!(!locked.repaired);
// All callers release read guards before following references/repairing.
// Fallback reconstruction uses its own dummy/temporary xref, never this
// locked map. Serialize the one repair and let concurrent readers wait.
let mut locked = r.map.put();
if locked.repaired {
return;
}
// Set first so an unwinding fallback cannot trigger a reparse storm.
// Assignment below publishes a complete replacement atomically.
locked.repaired = true;
let (xref_map, _) = fallback_xref_map(r.data.get(), &r.password);
locked.xref_map = xref_map;
locked.repaired = true;
}
#[inline]
@ -541,7 +553,7 @@ impl XRef {
return None;
};
let locked = repr.map.try_get().unwrap();
let locked = repr.map.get();
let mut r = Reader::new(repr.data.get().as_ref());
@ -550,6 +562,7 @@ impl XRef {
// shall be treated as a reference to the null object.
None
})?;
let repaired = locked.repaired;
drop(locked);
let mut ctx = ctx.clone();
@ -574,7 +587,7 @@ impl XRef {
};
// The xref table is broken, try to repair if not already repaired.
if self.is_repaired() {
if repaired {
error!(
"attempt was made at repairing xref, but object {id:?} still couldn't be read"
);
@ -599,6 +612,16 @@ impl XRef {
return None;
}
// Stream objects cannot themselves be stored in object streams
// (PDF 7.5.7). Reject malformed chains before recursively loading
// them, including A-in-B/B-in-A and arbitrarily deep chains.
if !matches!(
repr.map.get().xref_map.get(&obj_stream_id),
Some(EntryType::Normal(_))
) {
return None;
}
let stream = self.get_with::<Stream<'_>>(obj_stream_id, &ctx)?;
let data = repr.data.get_with(obj_stream_id, &ctx)?;
let object_stream = ObjectStream::new(stream, data, &ctx)?;
@ -674,6 +697,7 @@ impl TrailerData {
#[derive(Debug, Clone)]
struct SomeRepr {
dict_cache: Arc<Mutex<DictCache>>,
data: Arc<Data>,
map: Arc<RwLock<MapRepr>>,
metadata: Arc<Metadata>,
@ -1119,6 +1143,200 @@ fn parse_metadata(info_dict: &Dict<'_>) -> Metadata {
mod tests {
use super::*;
// Contributor-original, MIT OR Apache-2.0. Matches the bounded, uncompressed
// synthetic regression used to reproduce the cold-publication race. A second
// stream optionally checks that racing registration never aliases two ids.
fn object_stream_fixture(second: bool) -> Vec<u8> {
let mut out = b"%PDF-1.5\n".to_vec();
let mut offsets = alloc::collections::BTreeMap::new();
fn object(
out: &mut Vec<u8>,
offsets: &mut alloc::collections::BTreeMap<u32, usize>,
n: u32,
body: &[u8],
) {
offsets.insert(n, out.len());
out.extend_from_slice(format!("{n} 0 obj\n").as_bytes());
out.extend_from_slice(body);
out.extend_from_slice(b"\nendobj\n");
}
object(
&mut out,
&mut offsets,
1,
b"<< /Type /Catalog /Pages 2 0 R >>",
);
object(
&mut out,
&mut offsets,
2,
b"<< /Type /Pages /Count 1 /Kids [3 0 R] >>",
);
object(
&mut out,
&mut offsets,
3,
b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 100 100] >>",
);
let mut payload = b"20 0 << /Marker 123 >>".to_vec();
payload.resize(payload.len() + 4 * 1024 * 1024, b' ');
let mut stream = format!(
"<< /Type /ObjStm /N 1 /First 5 /Length {} >>\nstream\n",
payload.len()
)
.into_bytes();
stream.extend(payload);
stream.extend_from_slice(b"\nendstream");
object(&mut out, &mut offsets, 10, &stream);
if second {
let payload = b"21 0 << /Marker 456 >>";
let mut stream = format!(
"<< /Type /ObjStm /N 1 /First 5 /Length {} >>\nstream\n",
payload.len()
)
.into_bytes();
stream.extend_from_slice(payload);
stream.extend_from_slice(b"\nendstream");
object(&mut out, &mut offsets, 12, &stream);
}
let xref = out.len();
offsets.insert(11, xref);
let size = if second { 22 } else { 21 };
let mut rows = Vec::new();
for number in 0..size {
let (kind, a, b): (u8, u32, u16) = match number {
20 => (2, 10, 0),
21 => (2, 12, 0),
n => offsets
.get(&n)
.map_or((0, 0, 65535), |offset| (1, *offset as u32, 0)),
};
rows.push(kind);
rows.extend_from_slice(&a.to_be_bytes());
rows.extend_from_slice(&b.to_be_bytes());
}
let mut stream = format!(
"<< /Type /XRef /Size {size} /Root 1 0 R /W [1 4 2] /Length {} >>\nstream\n",
rows.len()
)
.into_bytes();
stream.extend(rows);
stream.extend_from_slice(b"\nendstream");
object(&mut out, &mut offsets, 11, &stream);
out.extend_from_slice(format!("startxref\n{xref}\n%%EOF\n").as_bytes());
out
}
#[cfg(feature = "std")]
#[test]
fn concurrent_cold_object_stream_reads_all_succeed() {
use std::sync::Barrier;
let bytes = Arc::new(object_stream_fixture(true));
for _ in 0..10 {
let pdf = Arc::new(crate::Pdf::new(bytes.clone()).unwrap());
let barrier = Arc::new(Barrier::new(16));
let jobs: Vec<_> = (0..16)
.map(|_| {
let (pdf, barrier) = (pdf.clone(), barrier.clone());
std::thread::spawn(move || {
barrier.wait();
for (id, marker) in [(20, 123), (21, 456), (20, 123)] {
let dict = pdf
.xref()
.get::<Dict<'_>>(ObjectIdentifier::new(id, 0))
.unwrap();
assert_eq!(dict.get::<i32>(b"Marker"), Some(marker));
}
})
})
.collect();
for job in jobs {
job.join().unwrap();
}
}
}
#[test]
fn recursive_object_stream_locations_fail_without_recursing() {
let pdf = crate::Pdf::new(object_stream_fixture(true)).unwrap();
let Inner::Some(repr) = &pdf.xref().0 else {
panic!("real xref")
};
{
let mut map = repr.map.put();
map.xref_map
.insert(ObjectIdentifier::new(10, 0), EntryType::ObjStream(12, 0));
map.xref_map
.insert(ObjectIdentifier::new(12, 0), EntryType::ObjStream(10, 0));
}
for id in [10, 12, 20, 21] {
assert!(
pdf.xref()
.get::<Dict<'_>>(ObjectIdentifier::new(id, 0))
.is_none()
);
}
}
#[cfg(feature = "std")]
#[test]
fn concurrent_repair_and_read_retries_the_repaired_map() {
let pdf = Arc::new(crate::Pdf::new(object_stream_fixture(false)).unwrap());
let Inner::Some(repr) = &pdf.xref().0 else {
panic!("real xref")
};
repr.map
.put()
.xref_map
.insert(ObjectIdentifier::new(1, 0), EntryType::Normal(0));
let barrier = Arc::new(std::sync::Barrier::new(16));
let jobs: Vec<_> = (0..16)
.map(|_| {
let (pdf, barrier) = (pdf.clone(), barrier.clone());
std::thread::spawn(move || {
barrier.wait();
for _ in 0..8 {
let root = pdf
.xref()
.get::<Dict<'_>>(ObjectIdentifier::new(1, 0))
.unwrap();
assert!(root.get_ref(PAGES).is_some());
pdf.xref().repair(); // already repaired is an idempotent no-op
}
})
})
.collect();
for job in jobs {
job.join().unwrap();
}
assert!(repr.map.get().repaired);
}
#[cfg(feature = "std")]
#[test]
fn poisoned_repair_lock_does_not_cascade_or_retry_repair() {
use std::panic::{AssertUnwindSafe, catch_unwind};
let pdf = crate::Pdf::new(object_stream_fixture(false)).unwrap();
let Inner::Some(repr) = &pdf.xref().0 else {
panic!("real xref")
};
assert!(
catch_unwind(AssertUnwindSafe(|| {
let mut map = repr.map.put();
map.repaired = true; // same state as an unwinding fallback before publication
panic!("injected repair failure");
}))
.is_err()
);
pdf.xref().repair();
assert!(
pdf.xref()
.get::<Dict<'_>>(ObjectIdentifier::new(1, 0))
.is_some()
);
assert!(repr.map.get().repaired);
}
#[test]
fn circular_prev_chain() {
let mut pdf = b"%PDF-1.0\n1 0 obj\n<< /Type /Catalog /Pages 2 0 R >>\nendobj\n".to_vec();