CJK: plan for the East Asian feature set
Phased plan for professional CJK typesetting (the East Asian feature set): sources (JLREQ, CLREQ, KLREQ, JIS X 4051, OpenType, IDML), current state and known bugs, architecture (character classes, mojikumi, kinsoku and the Japanese composer, vertical metrics, annotations, grids, composite fonts), IDML attributes, phases with estimates and open decisions. It builds on the CJK composition of docs/cjk-typography.md (composite fonts, kinsoku sets, aki, tsume, jidori, hanging punctuation, em alignment and leading models, kenten characters, tate-chu-yoko offsets) and plans what remains.
This commit is contained in:
aiongg
parent
02c5f61173
commit
d6a67bdcd1
|
|
@ -0,0 +1,273 @@
|
|||
# CJK typesetting plan
|
||||
|
||||
Goal: the Chinese, Japanese and Korean typesetting that InDesign offers with its East Asian
|
||||
feature set (the "Japanese/CJK" feature set, `Feature Set Locale Setting` 257), so that a
|
||||
DesignCraft document can be set to the standard of a Japanese, Chinese or Korean publisher,
|
||||
round-trip through IDML, and export correct PDF and EPUB.
|
||||
|
||||
It builds on the CJK import and composition described in [cjk-typography.md](cjk-typography.md)
|
||||
(composite fonts, kinsoku sets, aki, tsume, jidori, hanging punctuation, em alignment and leading
|
||||
models, kenten characters, tate-chu-yoko offsets, preserved mojikumi tables) and plans what
|
||||
remains.
|
||||
|
||||
## Sources (clean-room)
|
||||
|
||||
Behaviour comes from public specifications, not from InDesign's implementation:
|
||||
|
||||
- W3C *Requirements for Japanese Text Layout* (JLREQ): character classes, line composition,
|
||||
spacing (aki), line breaking (kinsoku), ruby, kenten, warichu, tate-chu-yoko, grids.
|
||||
- W3C *Requirements for Chinese Text Layout* (CLREQ) and *Korean* (KLREQ): punctuation placement
|
||||
for Simplified and Traditional Chinese, Korean word-based line breaking.
|
||||
- JIS X 4051 (published standard; JLREQ follows it).
|
||||
- OpenType: `vmtx`/`vhea`/`VORG`, `BASE`, the `vert`/`vrt2`/`vkrn`/`vpal`/`palt`/`halt`/`vhal`,
|
||||
`locl`, `ruby`, width (`fwid`/`hwid`/`pwid`/`qwid`/`twid`) and form (`jp78`/`jp83`/`jp90`/`jp04`,
|
||||
`trad`, `expt`, `nlck`, `hojo`) features.
|
||||
- The published IDML specification for attribute names and values.
|
||||
|
||||
InDesign itself is only observed black-box (feature names, which options exist); no wording,
|
||||
presets or tables are copied. Our default mojikumi and kinsoku tables are built from JLREQ.
|
||||
|
||||
## Where we are (2026-10)
|
||||
|
||||
| Area | State |
|
||||
|---|---|
|
||||
| Vertical frames, upright CJK, rotated Latin | done (per frame; glyphs turned by transforms) |
|
||||
| `vert`/`vrt2` in vertical frames | done (applied to every run, Latin included); Horizontal Kana (`hkna`) becomes `vkna` |
|
||||
| Kinsoku | named (hard, soft, Chinese, Korean) and custom kinsoku sets, bunri-kinshi, rensuuji, hanging punctuation (regular, force); without a set, fixed no-start / no-end lists and a break anywhere between CJK characters; kinsoku type (push in / push out priorities) kept but not applied |
|
||||
| Mojikumi | tables and paragraph references kept through IDML; not applied (Preflight says so) |
|
||||
| Aki, tsume, jidori | done ([cjk-typography.md](cjk-typography.md)) |
|
||||
| Character alignment, leading model | em box top / centre / bottom, ICF from ascender and descender, roman baseline; aki above / below, centre (centre down as centre) |
|
||||
| Tate-chu-yoko | manual, with offsets; no auto, 3+ digits overflow the em |
|
||||
| Ruby | group ruby only, fixed 50 % size, no options, doesn't affect leading |
|
||||
| Kenten | the IDML kinds and a custom character, drawn as characters; no position, size, alignment, colour or font |
|
||||
| Composite fonts | done: model, `style.compositeFont.*`, shaping, IDML, font list, replace and Preflight; no dialog |
|
||||
| CJK fonts | Japanese faces (Shippori Mincho first) from the optional craft-fonts build input; language-blind fallback; no per-document fonts |
|
||||
| Everything else below | missing |
|
||||
|
||||
Known bugs to fix first:
|
||||
|
||||
- Korean breaks between every syllable (it should break at spaces by default).
|
||||
- Only Arabic, Persian, Urdu, Hebrew and English language names reach the shaper as languages
|
||||
it knows, so `locl` never applies for the others (Japanese, Chinese, Korean, Turkish…); Chinese
|
||||
and Korean text get Japanese glyph forms and the Japanese fallback font.
|
||||
- Vertical advances are the horizontal ones and upright glyphs turn about a guessed centre
|
||||
(`adv × 0.38`), so proportional and non-square glyphs sit wrong in vertical text.
|
||||
- A justified CJK line has no stretch except Single Word Justification.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Character classes
|
||||
|
||||
A `cjk::class_of(char, &Context) -> Class` in `designcraft-compose` with the JLREQ classes
|
||||
(opening brackets, closing brackets, hyphens, dividing punctuation, middle dots, full stops,
|
||||
commas, inseparable characters, iteration marks, prolonged sound mark, small kana, prefixed and
|
||||
postfixed abbreviations, full-width ideographic space, hiragana, katakana, math symbols and
|
||||
operators, ideographs, Western characters, numerals, ruby bases, warichu brackets, …). Language
|
||||
and the paragraph's punctuation style decide the ambiguous ones (e.g. Chinese full-width
|
||||
punctuation centred vs corner-placed).
|
||||
|
||||
### Spacing: mojikumi
|
||||
|
||||
A *mojikumi set* is a table indexed by (class before, class after) giving the space between them
|
||||
as a min/optimum/max triple in em fractions, plus line-start and line-end rules. Sets are
|
||||
document resources (`Styles::mojikumi_tables`, read from and written to IDML with their override
|
||||
rows), referenced by name from the paragraph's `mojikumi`; what remains is applying them. We
|
||||
ship a few built-in sets derived from JLREQ (no line-start indent, one-em indent, half-width
|
||||
punctuation, full-width punctuation…), all original data, for the IDML built-in names.
|
||||
|
||||
Shaping keeps the glyphs' own advances; mojikumi is applied after shaping as *aki* glue between
|
||||
glyphs: a full-width bracket's empty half becomes negative aki (compression) and the composer
|
||||
can widen or narrow each gap within its range. This keeps `palt`-style proportional fonts
|
||||
working (tsume removes the side bearings before mojikumi runs).
|
||||
|
||||
### Line breaking: kinsoku and the Japanese composer
|
||||
|
||||
- Kinsoku sets (`doc::cjk::Kinsoku`: characters not allowed at line start, at line end, that may
|
||||
hang, and inseparable pairs), the paragraph's set, hanging (none / regular / force),
|
||||
bunri-kinshi and rensuuji are in place: they constrain breaks in both breakers. Remaining: the
|
||||
built-in hard and soft sets from JLREQ's classes (today's named sets are common punctuation
|
||||
lists), and the kinsoku type (push in first / push out first / push out only / prioritise the
|
||||
adjustment amount), which needs the Japanese composer's push-in and push-out candidates.
|
||||
- A `Composer::Japanese` (paragraph) and `Composer::JapaneseSingleLine` beside the Western
|
||||
composers. In `breaker::items` the Japanese composer emits:
|
||||
- boxes per glyph,
|
||||
- mojikumi glue between glyphs with priority tiers (compress punctuation first, then
|
||||
expand inter-ideograph space, then Western/CJK spacing, then letter-spacing),
|
||||
- kinsoku penalties (infinite inside prohibited pairs; push-in candidates scored by
|
||||
compression needed),
|
||||
- hanging punctuation as zero-width boxes allowed past the measure.
|
||||
Knuth–Plass already takes tiered stretch (`TIER_COST`), so this plugs into `kp_pass`.
|
||||
- Justification in `layout_line` distributes by the same tiers; the last line follows the
|
||||
paragraph's last-line alignment; CJK lines justify by default (JLREQ).
|
||||
- Korean: the Japanese composer with word-based breaking (spaces) by default; per-paragraph
|
||||
option for character-based breaking.
|
||||
|
||||
### Vertical text, properly
|
||||
|
||||
- Read `vmtx`, `vhea`, `VORG` and the ideographic em box (`BASE` `ideo`/`idtp`, else OS/2 typo
|
||||
metrics). `FontFace` gets `v_advance(gid)`, `v_origin(gid)` and `em_box()`.
|
||||
- Shape upright runs with direction `TopToBottom` so `vert`, `vkrn` and `vpal` apply and the
|
||||
shaper reports vertical advances; rotated (Latin) runs stay horizontal and turn as a whole.
|
||||
`vrt2` only for runs that are rotated, never with an extra turn.
|
||||
- Glyph placement in vertical lines uses the vertical origin, not a guessed centre.
|
||||
- Story direction per story (IDML `StoryOrientation`) rather than per frame; frames of one story
|
||||
agree.
|
||||
- Arrow keys and carets follow the vertical line direction.
|
||||
|
||||
### Annotations and inline structures
|
||||
|
||||
All set after line layout like today's ruby, but able to affect line height:
|
||||
|
||||
- **Ruby**: per-character (mono), group and jukugo ruby; alignment (1-2-1, centre, left/right,
|
||||
equal space, JIS rule), position (above/right, below/left), size, font, colour, x/y scale,
|
||||
offsets, overhang onto neighbours (none / ½ / 1 ruby character / ½ base / unlimited), automatic
|
||||
scaling, spacing to the base, "ruby adjusts leading".
|
||||
- **Kenten**: position, size, alignment, colour, font (the kinds and a custom character are
|
||||
there).
|
||||
- **Warichu**: an inline run set in 2+ smaller lines inside one line height, with size, line
|
||||
spacing, alignment and breaking rules.
|
||||
- **Tate-chu-yoko**: scale-to-fit-em, automatic TCY for runs of N digits (with or without
|
||||
Latin), rensuji (offsets are there).
|
||||
- **Shatai** (oblique in the CJK sense: angle and magnification, keeping the em box),
|
||||
**character rotation**.
|
||||
- **Character alignment and leading model**: ICF top and bottom from the font's ideographic
|
||||
character face (`BASE` `icft`/`icfb`) rather than its ascender and descender, and centre down
|
||||
as its own model.
|
||||
|
||||
Jidori, tsume, aki before and after, em-box alignment and the other leading models are in place
|
||||
([cjk-typography.md](cjk-typography.md)).
|
||||
|
||||
### Grids
|
||||
|
||||
- **Layout grid** (per page / parent): character size, characters per line, lines per column,
|
||||
columns and gutter, origin; drawn like the baseline grid; snap target.
|
||||
- **Frame grids**: text frames that show a character grid (horizontal or vertical), with a
|
||||
character count, named grids as document resources (font, size, scale, aki between characters
|
||||
and lines, alignment, view options), grid alignment of paragraphs (em-box centre, ICF top/bottom,
|
||||
roman baseline, gyoudori: lines spanning N grid lines), grid tracking.
|
||||
- Frame Grid tools (horizontal and vertical) and Object › Frame Type (text frame ↔ frame grid).
|
||||
|
||||
### Fonts
|
||||
|
||||
- **Composite fonts** (a named font made of a base font plus fonts for character sets, with size,
|
||||
baseline, x/y scale and centre adjustments) are resolved before shaping, read from and written
|
||||
to IDML and listed, replaced and preflighted by their member fonts; PDF embeds the member fonts.
|
||||
Remaining: the Composite Fonts dialog, and the predefined character classes (kanji, kana,
|
||||
full-width symbols, punctuation, Latin, numerals) beside custom sets.
|
||||
- **Language-aware fallback**: Japanese, Simplified Chinese, Traditional Chinese and Korean each
|
||||
have their own fallback chain of system fonts (Japanese: the craft-fonts Japanese faces first);
|
||||
`locl` follows the character language (IDML language names).
|
||||
- **No CJK fonts in the repo.** CJK fonts are large (15–25 MB each even subset) and documents
|
||||
should name the fonts they use. The Japanese UI and document faces come from the optional
|
||||
craft-fonts build input when DesignCraft is built with it (release builds are).
|
||||
- **Document fonts**: fonts in a `Document Fonts` folder beside an opened `.designcraft` or
|
||||
`.idml` file are available to that document only, ahead of system fonts, and unloaded when it
|
||||
closes. File › Package copies the document's fonts into the package's `Document Fonts` folder
|
||||
(font licences permitting: OS/2 `fsType` restricted-licence fonts are listed, not copied), so a
|
||||
packaged CJK document opens with its fonts on another machine. On the web, the fonts of a
|
||||
dropped or opened package folder load the same way.
|
||||
|
||||
### Editing and output
|
||||
|
||||
- Find/Change: kana-sensitive and width-sensitive matching; full/half-width and
|
||||
hiragana/katakana conversion (Type › Change Case).
|
||||
- Numbering styles: kanji, full-width, circled, parenthesised.
|
||||
- Typographer's quotes per language (Simplified/Traditional Chinese, Korean).
|
||||
- Character count (Info panel) by CJK conventions.
|
||||
- EPUB/HTML: `writing-mode: vertical-rl`, `<ruby>`, `text-combine-upright`, `text-emphasis`.
|
||||
- PDF: correct `ActualText` for ruby/warichu; vertical glyph placement as above.
|
||||
- IDML: every attribute below, import and export, round-trip tested.
|
||||
|
||||
## Feature set (UI)
|
||||
|
||||
Preferences › Type gets a *Feature Set* choice (Western, CJK). CJK shows the East Asian
|
||||
controls in the Character and Paragraph panels, styles dialogs and menus (Kinsoku, Mojikumi,
|
||||
Composite Fonts, Named Grids, Layout Grid), makes the Japanese composer the default for new
|
||||
documents and offers the frame grid tools. Western hides them; documents keep their settings
|
||||
either way (opening a CJK document in Western mode loses nothing). Every feature stays reachable
|
||||
as a command regardless of the feature set (agents, MCP).
|
||||
|
||||
## IDML attributes to support
|
||||
|
||||
Already read and written: `KentenKind`, `KentenCustomCharacter`, `RubyFlag`, `RubyString`,
|
||||
`Tatechuyoko`, `TatechuyokoXOffset`, `TatechuyokoYOffset`, `Jidori`, `Tsume`, `LeadingAki`,
|
||||
`TrailingAki`, `CharacterAlignment`, `LeadingModel`, `GlyphForm`, `OTFProportionalMetrics`,
|
||||
`OTFHVKana`, `OTFRomanItalics`; paragraph `KinsokuSet`, `KinsokuType`, `KinsokuHangType`,
|
||||
`BunriKinshi`, `Rensuuji`, `MojikumiTable`, `TreatIdeographicSpaceAsSpace`; resources
|
||||
`KinsokuTable`, `MojikumiTable`, `CompositeFont`, `StoryPreference` direction.
|
||||
|
||||
Character: `KentenFont`, `KentenFontStyle`, `KentenSize`, `KentenPosition`, `KentenAlignment`,
|
||||
`KentenCharacterSet`, `KentenXScale`, `KentenYScale`, `KentenFillColor`/`Tint`, `RubyType`,
|
||||
`RubyAlignment`, `RubyPosition`, `RubyFont`, `RubyFontStyle`, `RubyFontSize`, `RubyXScale`,
|
||||
`RubyYScale`, `RubyXOffset`, `RubyYOffset`, `RubyOverhang`, `RubyParentSpacing`,
|
||||
`RubyParentOverhangAmount`, `RubyAutoAlign`, `RubyAutoScaling`, `RubyAutoTcyDigits`, `Rensuji`,
|
||||
`CharacterRotation`, `ShataiMagnification`, `ShataiDegreeAngle`, `ShataiAdjustRotation`,
|
||||
`ShataiAdjustTsume`, `Warichu`, `WarichuLines`, `WarichuSize`, `WarichuLineSpacing`,
|
||||
`WarichuAlignment`, `WarichuCharsBeforeBreak`, `WarichuCharsAfterBreak`.
|
||||
|
||||
Paragraph: `Composer` (the Japanese composers), `GridAlignment` (all values),
|
||||
`GridAlignFirstLineOnly`, `GridGyoudori`, `ParagraphGyoudori`, `CjkGridTracking`, `AutoTcy`,
|
||||
`AutoTcyIncludeRoman`, `RotateSingleByteCharacters`, `ScaleAffectsLineHeight`.
|
||||
|
||||
Resources: `MojikumiTableSet`, `NamedGrid`, layout grid and frame grid preferences,
|
||||
`StoryPreference` orientation.
|
||||
|
||||
Exact names and enumerations are checked against the IDML specification as each phase lands.
|
||||
|
||||
## Phases
|
||||
|
||||
Each phase lands as reviewable commits with tests (compose tests for layout, engine tests for
|
||||
commands, IDML round trips, a rendered sample page) and `cargo xtask ci`.
|
||||
|
||||
| # | Phase | Contents | Estimate (agent hours) |
|
||||
|---|---|---|---|
|
||||
| 1 | Foundations | language passed to the shaper (`locl`); Simplified/Traditional Chinese and Korean languages; language-aware CJK fallback; document fonts folder; Korean word breaking; `vmtx`/`VORG`/em box and vertical shaping; story-level direction; vertical caret keys | 12–17 |
|
||||
| 2 | Japanese composer | character classes; mojikumi applied (built-in sets and the document's tables) as aki glue; JLREQ kinsoku sets and kinsoku types (push in / push out); Japanese paragraph and single-line composers; CJK justification and last line; Chinese punctuation styles; IDML | 20–30 |
|
||||
| 3 | Inline CJK | full ruby, kenten options, warichu, auto TCY and scale-to-fit, rensuji, shatai, character rotation, ICF alignment from the font; IDML | 20–30 |
|
||||
| 4 | Grids | layout grid, frame grids and tools, named grids, grid alignment and gyoudori, grid tracking, character count; IDML | 20–30 |
|
||||
| 5 | Fonts | Composite Fonts dialog and predefined character classes; CJK OpenType features in the OpenType menu | 5–8 |
|
||||
| 6 | Editing and output | kana/width-sensitive Find/Change, width and kana conversion, CJK numbering, quotes, EPUB/HTML vertical text and ruby | 10–15 |
|
||||
| 7 | Feature set UI | Preferences › Feature Set, panels and dialogs for every attribute, Japanese defaults for new documents, translations | 15–20 |
|
||||
|
||||
Total: roughly 100–150 agent hours.
|
||||
|
||||
### Phase 1 in detail
|
||||
|
||||
1. `CharProps.language` reaches `designcraft_fonts::shape` (harfrust buffer language) and the
|
||||
word cache key; tests: a font with `locl` for JAN vs ZHS draws different glyphs for U+76F4.
|
||||
2. Language list: "Chinese: Simplified", "Chinese: Traditional", "Korean", "Japanese" (IDML
|
||||
names), with quotes and spelling/hyphenation off for them.
|
||||
3. Fallback chains by language in `FontDb::fallback_for` (system fonts per platform; the
|
||||
craft-fonts Japanese faces first for Japanese); tests with stand-in fonts and a fake catalog.
|
||||
4. Document fonts: opening a document loads the fonts in its `Document Fonts` folder for that
|
||||
document; Package copies used fonts there (licence permitting) and reports the ones it
|
||||
couldn't copy; tests with a synthetic package folder.
|
||||
5. Korean: `cjk_break_between` doesn't break between Hangul syllables unless the paragraph asks
|
||||
for character breaking; test with a Korean sentence.
|
||||
6. `FontFace::v_advance/v_origin/em_box` from `vmtx`/`vhea`/`VORG`/`BASE`; upright glyphs in
|
||||
vertical lines use them; tests against synthetic fonts with and without `vmtx` (without, the
|
||||
em box), and Shippori Mincho's tables when built with craft-fonts.
|
||||
7. Upright runs shaped top-to-bottom (`vert`, `vkrn`, `vpal` apply); `vrt2` never on turned
|
||||
runs; test that a full-width comma takes its vertical form and position.
|
||||
8. Story direction at story level (frames of a story agree; IDML maps both ways); arrow keys move
|
||||
along the vertical line.
|
||||
|
||||
## Decisions
|
||||
|
||||
Each follows what an InDesign user expects.
|
||||
|
||||
1. **Feature set gating.** East Asian controls are hidden in the Western feature set and shown in
|
||||
the CJK feature set, as in InDesign. Every feature stays reachable as a command either way.
|
||||
2. **Korean line breaking** is word-based by default (KLREQ), with a per-paragraph option to break
|
||||
between syllables.
|
||||
3. **PDF vertical text** keeps transformed horizontal glyphs (correct-looking, copyable via
|
||||
ActualText) rather than vertical CID fonts (`Identity-V`).
|
||||
4. **No CJK fonts in the repo**; documents carry theirs in `Document Fonts`, scoped to the
|
||||
document as in InDesign. DesignCraft embeds Japanese UI and document faces only when built with
|
||||
the optional craft-fonts input (release builds are).
|
||||
5. **Package copies every font that draws text**, fallback fonts included, so a package
|
||||
reproduces what is on screen.
|
||||
6. **Korean fallback is serif first** (Myungjo), as Chinese and Japanese fall back to Song and
|
||||
Mincho: the serif faces CJK editions of layout software default to.
|
||||
7. **Language names on IDML import** accept the common spellings of Simplified and Traditional
|
||||
Chinese, since the exact names InDesign writes are unconfirmed without a sample file.
|
||||
Loading…
Reference in New Issue