Type: the caret and double-click take Thai word by word

Ctrl/Alt+arrows and double-click treated a run of Thai (Lao, Khmer,
Myanmar) without spaces as one word, and stopped at a word's tone and
vowel marks, which didn't count as part of a word. They now stop at the
word boundaries the line segmenter finds in those scripts (shared with
line breaking, so no more data ships), and combining marks count as
part of a word, also for combining accents in other scripts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
bflatastic 2026-10-11 19:02:19 +07:00
parent d17c07f343
commit a08ca07c69
5 changed files with 65 additions and 17 deletions

File diff suppressed because one or more lines are too long

View File

@ -205,32 +205,49 @@ pub fn next_cluster(s: &str, i: usize) -> usize {
GraphemeCursor::new(i, s.len(), true).next_boundary(s, 0).ok().flatten().unwrap_or(s.len())
}
/// Part of a word: a letter, digit or mark (Thai vowel and tone marks, combining accents), an
/// underscore or an apostrophe.
fn is_word_char(c: char) -> bool {
c.is_alphanumeric() || c == '_' || c == '\'' || c == '’'
use unicode_general_category::{GeneralCategory as G, get_general_category};
c.is_alphanumeric() || matches!(c, '_' | '\'' | '’') || matches!(get_general_category(c), G::NonspacingMark | G::SpacingMark | G::EnclosingMark)
}
/// Start of the previous word (Cmd/Alt+Left).
/// Is byte `b` of `s` (with `breaks`, its [`crate::shape::unspaced_word_breaks`]) a boundary
/// between two words of Thai, Lao, Khmer or Myanmar, which no space separates?
fn inner_word_break(s: &str, breaks: &[usize], b: usize) -> bool {
use crate::shape::is_unspaced_script;
breaks.binary_search(&b).is_ok()
&& s.get(b..).and_then(|t| t.chars().next()).is_some_and(is_unspaced_script)
&& s.get(..b).and_then(|t| t.chars().next_back()).is_some_and(is_unspaced_script)
}
/// Start of the previous word (Cmd/Alt+Left). Thai (Lao, Khmer, Myanmar) words end where their
/// script's word boundaries are, spaces or not.
pub fn prev_word(s: &str, i: usize) -> usize {
let i = floor_char(s, i);
let breaks = crate::shape::unspaced_word_breaks(s);
let chars: Vec<(usize, char)> = s[..i].char_indices().collect();
let mut k = chars.len();
while k > 0 && !is_word_char(chars[k - 1].1) {
k -= 1;
}
while k > 0 && is_word_char(chars[k - 1].1) {
let end = k;
while k > 0 && is_word_char(chars[k - 1].1) && !(k < end && inner_word_break(s, &breaks, chars[k].0)) {
k -= 1;
}
chars.get(k).map(|c| c.0).unwrap_or(i)
}
/// End of the next word (Cmd/Alt+Right).
/// End of the next word (Cmd/Alt+Right): see [`prev_word`].
pub fn next_word(s: &str, i: usize) -> usize {
let i = floor_char(s, i);
let breaks = crate::shape::unspaced_word_breaks(s);
let mut it = s[i..].char_indices().peekable();
while it.peek().is_some_and(|(_, c)| !is_word_char(*c)) {
it.next();
}
while it.peek().is_some_and(|(_, c)| is_word_char(*c)) {
let start = it.peek().map(|(k, _)| i + k);
while it.peek().is_some_and(|(k, c)| is_word_char(*c) && !(Some(i + k) != start && inner_word_break(s, &breaks, i + k))) {
it.next();
}
it.peek().map(|(k, _)| i + k).unwrap_or(s.len())
@ -260,16 +277,18 @@ pub fn word_at(s: &str, i: usize) -> Range<usize> {
if k == 2 {
return at..at + c.len_utf8();
}
// Thai (Lao, Khmer, Myanmar) words also end where their script's word boundaries are.
let breaks = if k == 0 { crate::shape::unspaced_word_breaks(s) } else { vec![] };
let mut a = at;
for (p, d) in s[..at].char_indices().rev() {
if class(d) != k {
if class(d) != k || inner_word_break(s, &breaks, a) {
break;
}
a = p;
}
let mut b = at;
for (p, d) in s[at..].char_indices() {
if class(d) != k {
if class(d) != k || (b > at && inner_word_break(s, &breaks, b)) {
break;
}
b = at + p + d.len_utf8();

View File

@ -241,13 +241,12 @@ pub(crate) fn is_unspaced_script(c: char) -> bool {
matches!(c.script(), Script::Thai | Script::Lao | Script::Khmer | Script::Myanmar)
}
/// Mark the glyphs of `sg` (shaped from `text`, which starts at byte `start` of the text the
/// glyphs index) after which a word of Thai, Lao, Khmer or Myanmar ends: where a line may break in
/// text without spaces between words. The words come from ICU4X's line segmenter (its LSTM models
/// for those scripts); other text keeps its own breaks.
pub(crate) fn mark_word_ends(text: &str, start: usize, sg: &mut [SGlyph]) {
// The dictionaries on the desktop (the most accurate, a few megabytes); the far smaller LSTM
// models on the web.
/// The byte offsets of `text` where a word of Thai, Lao, Khmer or Myanmar starts or ends: where a
/// line may break, and where the caret stops word by word, in text without spaces between words.
/// They come from ICU4X's line segmenter (its dictionaries on the desktop, its far smaller LSTM
/// models on the web); empty when `text` has none of those scripts. Offsets elsewhere (around
/// other text) are UAX #14's breaks.
pub(crate) fn unspaced_word_breaks(text: &str) -> Vec<usize> {
#[cfg(not(target_arch = "wasm32"))]
static SEGMENTER: std::sync::LazyLock<icu_segmenter::LineSegmenterBorrowed<'static>> =
std::sync::LazyLock::new(|| icu_segmenter::LineSegmenter::new_dictionary(Default::default()));
@ -255,9 +254,19 @@ pub(crate) fn mark_word_ends(text: &str, start: usize, sg: &mut [SGlyph]) {
static SEGMENTER: std::sync::LazyLock<icu_segmenter::LineSegmenterBorrowed<'static>> =
std::sync::LazyLock::new(|| icu_segmenter::LineSegmenter::new_auto(Default::default()));
if !text.chars().any(is_unspaced_script) {
return vec![];
}
SEGMENTER.segment_str(text).collect()
}
/// Mark the glyphs of `sg` (shaped from `text`, which starts at byte `start` of the text the
/// glyphs index) after which a word of Thai, Lao, Khmer or Myanmar ends ([`unspaced_word_breaks`]);
/// other text keeps its own breaks.
pub(crate) fn mark_word_ends(text: &str, start: usize, sg: &mut [SGlyph]) {
let breaks: Vec<usize> = unspaced_word_breaks(text).into_iter().map(|b| b + start).collect();
if breaks.is_empty() {
return;
}
let breaks: Vec<usize> = SEGMENTER.segment_str(text).map(|b| b + start).collect();
for g in sg.iter_mut().filter(|g| g.inline.is_none() && is_unspaced_script(g.ch)) {
g.word_end = breaks.binary_search(&(g.byte + g.len)).is_ok();
}

View File

@ -805,3 +805,22 @@ fn caret_moves_over_whole_clusters() {
assert_eq!((edit::prev_cluster(s, 6), edit::next_cluster(s, 6)), (3, 12));
assert_eq!((edit::next_cluster("", 0), edit::prev_cluster("ab", 99)), (0, 1));
}
/// Word steps and double-click stop between Thai words, which no space separates: ภาษา|ไทย|ง่าย.
#[test]
fn thai_words_are_words_to_the_caret() {
let s = "ภาษาไทยง่าย ok";
let (thai, easy) = ("ภาษา".len(), "ภาษาไทย".len());
let end = "ภาษาไทยง่าย".len();
assert_eq!(edit::next_word(s, 0), thai);
assert_eq!(edit::next_word(s, thai), easy);
assert_eq!(edit::next_word(s, easy), end);
assert_eq!(edit::prev_word(s, end), easy);
assert_eq!(edit::prev_word(s, easy), thai);
assert_eq!(edit::prev_word(s, thai), 0);
assert_eq!(edit::word_at(s, thai + 3), thai..easy, "double-click inside ไทย");
assert_eq!(edit::word_at(s, 0), 0..thai);
// Latin keeps its words.
assert_eq!(edit::word_at(s, s.len() - 1), end + 1..s.len());
assert_eq!(edit::next_word("hello world", 0), 5);
}

View File

@ -42,7 +42,7 @@ against Illustrator's public documentation; **estimated** unless noted.
| Touch Type tool | missing | listed in the toolbar, not implemented |
| Retype (identify fonts in outlines/images) | missing | Illustrator's uses a cloud model |
| Snap to Glyph | missing | menu stub |
| Thai, Lao, Khmer, Myanmar: shaping (marks, reordering), lines broken between words | done | words found by ICU4X's line segmenter: dictionaries on the desktop, LSTM models on the web |
| Thai, Lao, Khmer, Myanmar: shaping (marks, reordering), lines broken between words, word-by-word caret and double-click | done | words found by ICU4X's line segmenter: dictionaries on the desktop, LSTM models on the web |
| Middle Eastern: bidirectional layout, Paragraph Direction | done | |
| Middle Eastern: Character Direction, digit types, kashidas, Middle Eastern composers, split caret | missing | |
| CJK: input methods (marked text, candidate window) | done | |
@ -55,6 +55,7 @@ against Illustrator's public documentation; **estimated** unless noted.
| Date | Change | Summary |
|---|---|---|
| 2026-10-11 | minor | Thai, Lao, Khmer and Myanmar words for the caret (Ctrl/Alt+arrows) and double-click; combining marks count as part of a word; score unchanged (estimated) |
| 2026-10-11 | minor | Font menu: a Thai script filter beside Japanese and Latin; score unchanged (estimated) |
| 2026-10-11 | minor | Thai, Lao, Khmer and Myanmar text breaks its lines between words (ICU4X line segmenter) instead of only at spaces; score unchanged (estimated) |
| 2026-10-11 | minor | The Type tool's arrows and Delete move by grapheme cluster, never between a Thai consonant and its vowel or tone marks; score unchanged (estimated) |