diff --git a/docs/cjk.md b/docs/cjk.md new file mode 100644 index 0000000..534b36c --- /dev/null +++ b/docs/cjk.md @@ -0,0 +1,273 @@ +# CJK typesetting plan + +Goal: the Chinese, Japanese and Korean typesetting that InDesign offers with its East Asian +feature set (the "Japanese/CJK" feature set, `Feature Set Locale Setting` 257), so that a +DesignCraft document can be set to the standard of a Japanese, Chinese or Korean publisher, +round-trip through IDML, and export correct PDF and EPUB. + +It builds on the CJK import and composition described in [cjk-typography.md](cjk-typography.md) +(composite fonts, kinsoku sets, aki, tsume, jidori, hanging punctuation, em alignment and leading +models, kenten characters, tate-chu-yoko offsets, preserved mojikumi tables) and plans what +remains. + +## Sources (clean-room) + +Behaviour comes from public specifications, not from InDesign's implementation: + +- W3C *Requirements for Japanese Text Layout* (JLREQ): character classes, line composition, + spacing (aki), line breaking (kinsoku), ruby, kenten, warichu, tate-chu-yoko, grids. +- W3C *Requirements for Chinese Text Layout* (CLREQ) and *Korean* (KLREQ): punctuation placement + for Simplified and Traditional Chinese, Korean word-based line breaking. +- JIS X 4051 (published standard; JLREQ follows it). +- OpenType: `vmtx`/`vhea`/`VORG`, `BASE`, the `vert`/`vrt2`/`vkrn`/`vpal`/`palt`/`halt`/`vhal`, + `locl`, `ruby`, width (`fwid`/`hwid`/`pwid`/`qwid`/`twid`) and form (`jp78`/`jp83`/`jp90`/`jp04`, + `trad`, `expt`, `nlck`, `hojo`) features. +- The published IDML specification for attribute names and values. + +InDesign itself is only observed black-box (feature names, which options exist); no wording, +presets or tables are copied. Our default mojikumi and kinsoku tables are built from JLREQ. + +## Where we are (2026-10) + +| Area | State | +|---|---| +| Vertical frames, upright CJK, rotated Latin | done (per frame; glyphs turned by transforms) | +| `vert`/`vrt2` in vertical frames | done (applied to every run, Latin included); Horizontal Kana (`hkna`) becomes `vkna` | +| Kinsoku | named (hard, soft, Chinese, Korean) and custom kinsoku sets, bunri-kinshi, rensuuji, hanging punctuation (regular, force); without a set, fixed no-start / no-end lists and a break anywhere between CJK characters; kinsoku type (push in / push out priorities) kept but not applied | +| Mojikumi | tables and paragraph references kept through IDML; not applied (Preflight says so) | +| Aki, tsume, jidori | done ([cjk-typography.md](cjk-typography.md)) | +| Character alignment, leading model | em box top / centre / bottom, ICF from ascender and descender, roman baseline; aki above / below, centre (centre down as centre) | +| Tate-chu-yoko | manual, with offsets; no auto, 3+ digits overflow the em | +| Ruby | group ruby only, fixed 50 % size, no options, doesn't affect leading | +| Kenten | the IDML kinds and a custom character, drawn as characters; no position, size, alignment, colour or font | +| Composite fonts | done: model, `style.compositeFont.*`, shaping, IDML, font list, replace and Preflight; no dialog | +| CJK fonts | Japanese faces (Shippori Mincho first) from the optional craft-fonts build input; language-blind fallback; no per-document fonts | +| Everything else below | missing | + +Known bugs to fix first: + +- Korean breaks between every syllable (it should break at spaces by default). +- Only Arabic, Persian, Urdu, Hebrew and English language names reach the shaper as languages + it knows, so `locl` never applies for the others (Japanese, Chinese, Korean, Turkish…); Chinese + and Korean text get Japanese glyph forms and the Japanese fallback font. +- Vertical advances are the horizontal ones and upright glyphs turn about a guessed centre + (`adv × 0.38`), so proportional and non-square glyphs sit wrong in vertical text. +- A justified CJK line has no stretch except Single Word Justification. + +## Architecture + +### Character classes + +A `cjk::class_of(char, &Context) -> Class` in `designcraft-compose` with the JLREQ classes +(opening brackets, closing brackets, hyphens, dividing punctuation, middle dots, full stops, +commas, inseparable characters, iteration marks, prolonged sound mark, small kana, prefixed and +postfixed abbreviations, full-width ideographic space, hiragana, katakana, math symbols and +operators, ideographs, Western characters, numerals, ruby bases, warichu brackets, …). Language +and the paragraph's punctuation style decide the ambiguous ones (e.g. Chinese full-width +punctuation centred vs corner-placed). + +### Spacing: mojikumi + +A *mojikumi set* is a table indexed by (class before, class after) giving the space between them +as a min/optimum/max triple in em fractions, plus line-start and line-end rules. Sets are +document resources (`Styles::mojikumi_tables`, read from and written to IDML with their override +rows), referenced by name from the paragraph's `mojikumi`; what remains is applying them. We +ship a few built-in sets derived from JLREQ (no line-start indent, one-em indent, half-width +punctuation, full-width punctuation…), all original data, for the IDML built-in names. + +Shaping keeps the glyphs' own advances; mojikumi is applied after shaping as *aki* glue between +glyphs: a full-width bracket's empty half becomes negative aki (compression) and the composer +can widen or narrow each gap within its range. This keeps `palt`-style proportional fonts +working (tsume removes the side bearings before mojikumi runs). + +### Line breaking: kinsoku and the Japanese composer + +- Kinsoku sets (`doc::cjk::Kinsoku`: characters not allowed at line start, at line end, that may + hang, and inseparable pairs), the paragraph's set, hanging (none / regular / force), + bunri-kinshi and rensuuji are in place: they constrain breaks in both breakers. Remaining: the + built-in hard and soft sets from JLREQ's classes (today's named sets are common punctuation + lists), and the kinsoku type (push in first / push out first / push out only / prioritise the + adjustment amount), which needs the Japanese composer's push-in and push-out candidates. +- A `Composer::Japanese` (paragraph) and `Composer::JapaneseSingleLine` beside the Western + composers. In `breaker::items` the Japanese composer emits: + - boxes per glyph, + - mojikumi glue between glyphs with priority tiers (compress punctuation first, then + expand inter-ideograph space, then Western/CJK spacing, then letter-spacing), + - kinsoku penalties (infinite inside prohibited pairs; push-in candidates scored by + compression needed), + - hanging punctuation as zero-width boxes allowed past the measure. + Knuth–Plass already takes tiered stretch (`TIER_COST`), so this plugs into `kp_pass`. +- Justification in `layout_line` distributes by the same tiers; the last line follows the + paragraph's last-line alignment; CJK lines justify by default (JLREQ). +- Korean: the Japanese composer with word-based breaking (spaces) by default; per-paragraph + option for character-based breaking. + +### Vertical text, properly + +- Read `vmtx`, `vhea`, `VORG` and the ideographic em box (`BASE` `ideo`/`idtp`, else OS/2 typo + metrics). `FontFace` gets `v_advance(gid)`, `v_origin(gid)` and `em_box()`. +- Shape upright runs with direction `TopToBottom` so `vert`, `vkrn` and `vpal` apply and the + shaper reports vertical advances; rotated (Latin) runs stay horizontal and turn as a whole. + `vrt2` only for runs that are rotated, never with an extra turn. +- Glyph placement in vertical lines uses the vertical origin, not a guessed centre. +- Story direction per story (IDML `StoryOrientation`) rather than per frame; frames of one story + agree. +- Arrow keys and carets follow the vertical line direction. + +### Annotations and inline structures + +All set after line layout like today's ruby, but able to affect line height: + +- **Ruby**: per-character (mono), group and jukugo ruby; alignment (1-2-1, centre, left/right, + equal space, JIS rule), position (above/right, below/left), size, font, colour, x/y scale, + offsets, overhang onto neighbours (none / ½ / 1 ruby character / ½ base / unlimited), automatic + scaling, spacing to the base, "ruby adjusts leading". +- **Kenten**: position, size, alignment, colour, font (the kinds and a custom character are + there). +- **Warichu**: an inline run set in 2+ smaller lines inside one line height, with size, line + spacing, alignment and breaking rules. +- **Tate-chu-yoko**: scale-to-fit-em, automatic TCY for runs of N digits (with or without + Latin), rensuji (offsets are there). +- **Shatai** (oblique in the CJK sense: angle and magnification, keeping the em box), + **character rotation**. +- **Character alignment and leading model**: ICF top and bottom from the font's ideographic + character face (`BASE` `icft`/`icfb`) rather than its ascender and descender, and centre down + as its own model. + +Jidori, tsume, aki before and after, em-box alignment and the other leading models are in place +([cjk-typography.md](cjk-typography.md)). + +### Grids + +- **Layout grid** (per page / parent): character size, characters per line, lines per column, + columns and gutter, origin; drawn like the baseline grid; snap target. +- **Frame grids**: text frames that show a character grid (horizontal or vertical), with a + character count, named grids as document resources (font, size, scale, aki between characters + and lines, alignment, view options), grid alignment of paragraphs (em-box centre, ICF top/bottom, + roman baseline, gyoudori: lines spanning N grid lines), grid tracking. +- Frame Grid tools (horizontal and vertical) and Object › Frame Type (text frame ↔ frame grid). + +### Fonts + +- **Composite fonts** (a named font made of a base font plus fonts for character sets, with size, + baseline, x/y scale and centre adjustments) are resolved before shaping, read from and written + to IDML and listed, replaced and preflighted by their member fonts; PDF embeds the member fonts. + Remaining: the Composite Fonts dialog, and the predefined character classes (kanji, kana, + full-width symbols, punctuation, Latin, numerals) beside custom sets. +- **Language-aware fallback**: Japanese, Simplified Chinese, Traditional Chinese and Korean each + have their own fallback chain of system fonts (Japanese: the craft-fonts Japanese faces first); + `locl` follows the character language (IDML language names). +- **No CJK fonts in the repo.** CJK fonts are large (15–25 MB each even subset) and documents + should name the fonts they use. The Japanese UI and document faces come from the optional + craft-fonts build input when DesignCraft is built with it (release builds are). +- **Document fonts**: fonts in a `Document Fonts` folder beside an opened `.designcraft` or + `.idml` file are available to that document only, ahead of system fonts, and unloaded when it + closes. File › Package copies the document's fonts into the package's `Document Fonts` folder + (font licences permitting: OS/2 `fsType` restricted-licence fonts are listed, not copied), so a + packaged CJK document opens with its fonts on another machine. On the web, the fonts of a + dropped or opened package folder load the same way. + +### Editing and output + +- Find/Change: kana-sensitive and width-sensitive matching; full/half-width and + hiragana/katakana conversion (Type › Change Case). +- Numbering styles: kanji, full-width, circled, parenthesised. +- Typographer's quotes per language (Simplified/Traditional Chinese, Korean). +- Character count (Info panel) by CJK conventions. +- EPUB/HTML: `writing-mode: vertical-rl`, ``, `text-combine-upright`, `text-emphasis`. +- PDF: correct `ActualText` for ruby/warichu; vertical glyph placement as above. +- IDML: every attribute below, import and export, round-trip tested. + +## Feature set (UI) + +Preferences › Type gets a *Feature Set* choice (Western, CJK). CJK shows the East Asian +controls in the Character and Paragraph panels, styles dialogs and menus (Kinsoku, Mojikumi, +Composite Fonts, Named Grids, Layout Grid), makes the Japanese composer the default for new +documents and offers the frame grid tools. Western hides them; documents keep their settings +either way (opening a CJK document in Western mode loses nothing). Every feature stays reachable +as a command regardless of the feature set (agents, MCP). + +## IDML attributes to support + +Already read and written: `KentenKind`, `KentenCustomCharacter`, `RubyFlag`, `RubyString`, +`Tatechuyoko`, `TatechuyokoXOffset`, `TatechuyokoYOffset`, `Jidori`, `Tsume`, `LeadingAki`, +`TrailingAki`, `CharacterAlignment`, `LeadingModel`, `GlyphForm`, `OTFProportionalMetrics`, +`OTFHVKana`, `OTFRomanItalics`; paragraph `KinsokuSet`, `KinsokuType`, `KinsokuHangType`, +`BunriKinshi`, `Rensuuji`, `MojikumiTable`, `TreatIdeographicSpaceAsSpace`; resources +`KinsokuTable`, `MojikumiTable`, `CompositeFont`, `StoryPreference` direction. + +Character: `KentenFont`, `KentenFontStyle`, `KentenSize`, `KentenPosition`, `KentenAlignment`, +`KentenCharacterSet`, `KentenXScale`, `KentenYScale`, `KentenFillColor`/`Tint`, `RubyType`, +`RubyAlignment`, `RubyPosition`, `RubyFont`, `RubyFontStyle`, `RubyFontSize`, `RubyXScale`, +`RubyYScale`, `RubyXOffset`, `RubyYOffset`, `RubyOverhang`, `RubyParentSpacing`, +`RubyParentOverhangAmount`, `RubyAutoAlign`, `RubyAutoScaling`, `RubyAutoTcyDigits`, `Rensuji`, +`CharacterRotation`, `ShataiMagnification`, `ShataiDegreeAngle`, `ShataiAdjustRotation`, +`ShataiAdjustTsume`, `Warichu`, `WarichuLines`, `WarichuSize`, `WarichuLineSpacing`, +`WarichuAlignment`, `WarichuCharsBeforeBreak`, `WarichuCharsAfterBreak`. + +Paragraph: `Composer` (the Japanese composers), `GridAlignment` (all values), +`GridAlignFirstLineOnly`, `GridGyoudori`, `ParagraphGyoudori`, `CjkGridTracking`, `AutoTcy`, +`AutoTcyIncludeRoman`, `RotateSingleByteCharacters`, `ScaleAffectsLineHeight`. + +Resources: `MojikumiTableSet`, `NamedGrid`, layout grid and frame grid preferences, +`StoryPreference` orientation. + +Exact names and enumerations are checked against the IDML specification as each phase lands. + +## Phases + +Each phase lands as reviewable commits with tests (compose tests for layout, engine tests for +commands, IDML round trips, a rendered sample page) and `cargo xtask ci`. + +| # | Phase | Contents | Estimate (agent hours) | +|---|---|---|---| +| 1 | Foundations | language passed to the shaper (`locl`); Simplified/Traditional Chinese and Korean languages; language-aware CJK fallback; document fonts folder; Korean word breaking; `vmtx`/`VORG`/em box and vertical shaping; story-level direction; vertical caret keys | 12–17 | +| 2 | Japanese composer | character classes; mojikumi applied (built-in sets and the document's tables) as aki glue; JLREQ kinsoku sets and kinsoku types (push in / push out); Japanese paragraph and single-line composers; CJK justification and last line; Chinese punctuation styles; IDML | 20–30 | +| 3 | Inline CJK | full ruby, kenten options, warichu, auto TCY and scale-to-fit, rensuji, shatai, character rotation, ICF alignment from the font; IDML | 20–30 | +| 4 | Grids | layout grid, frame grids and tools, named grids, grid alignment and gyoudori, grid tracking, character count; IDML | 20–30 | +| 5 | Fonts | Composite Fonts dialog and predefined character classes; CJK OpenType features in the OpenType menu | 5–8 | +| 6 | Editing and output | kana/width-sensitive Find/Change, width and kana conversion, CJK numbering, quotes, EPUB/HTML vertical text and ruby | 10–15 | +| 7 | Feature set UI | Preferences › Feature Set, panels and dialogs for every attribute, Japanese defaults for new documents, translations | 15–20 | + +Total: roughly 100–150 agent hours. + +### Phase 1 in detail + +1. `CharProps.language` reaches `designcraft_fonts::shape` (harfrust buffer language) and the + word cache key; tests: a font with `locl` for JAN vs ZHS draws different glyphs for U+76F4. +2. Language list: "Chinese: Simplified", "Chinese: Traditional", "Korean", "Japanese" (IDML + names), with quotes and spelling/hyphenation off for them. +3. Fallback chains by language in `FontDb::fallback_for` (system fonts per platform; the + craft-fonts Japanese faces first for Japanese); tests with stand-in fonts and a fake catalog. +4. Document fonts: opening a document loads the fonts in its `Document Fonts` folder for that + document; Package copies used fonts there (licence permitting) and reports the ones it + couldn't copy; tests with a synthetic package folder. +5. Korean: `cjk_break_between` doesn't break between Hangul syllables unless the paragraph asks + for character breaking; test with a Korean sentence. +6. `FontFace::v_advance/v_origin/em_box` from `vmtx`/`vhea`/`VORG`/`BASE`; upright glyphs in + vertical lines use them; tests against synthetic fonts with and without `vmtx` (without, the + em box), and Shippori Mincho's tables when built with craft-fonts. +7. Upright runs shaped top-to-bottom (`vert`, `vkrn`, `vpal` apply); `vrt2` never on turned + runs; test that a full-width comma takes its vertical form and position. +8. Story direction at story level (frames of a story agree; IDML maps both ways); arrow keys move + along the vertical line. + +## Decisions + +Each follows what an InDesign user expects. + +1. **Feature set gating.** East Asian controls are hidden in the Western feature set and shown in + the CJK feature set, as in InDesign. Every feature stays reachable as a command either way. +2. **Korean line breaking** is word-based by default (KLREQ), with a per-paragraph option to break + between syllables. +3. **PDF vertical text** keeps transformed horizontal glyphs (correct-looking, copyable via + ActualText) rather than vertical CID fonts (`Identity-V`). +4. **No CJK fonts in the repo**; documents carry theirs in `Document Fonts`, scoped to the + document as in InDesign. DesignCraft embeds Japanese UI and document faces only when built with + the optional craft-fonts input (release builds are). +5. **Package copies every font that draws text**, fallback fonts included, so a package + reproduces what is on screen. +6. **Korean fallback is serif first** (Myungjo), as Chinese and Japanese fall back to Song and + Mincho: the serif faces CJK editions of layout software default to. +7. **Language names on IDML import** accept the common spellings of Simplified and Traditional + Chinese, since the exact names InDesign writes are unconfirmed without a sample file.