Type: the caret and double-click take Thai word by word
Ctrl/Alt+arrows and double-click treated a run of Thai (Lao, Khmer, Myanmar) without spaces as one word, and stopped at a word's tone and vowel marks, which didn't count as part of a word. They now stop at the word boundaries the line segmenter finds in those scripts (shared with line breaking, so no more data ships), and combining marks count as part of a word, also for combining accents in other scripts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
d17c07f343
commit
a08ca07c69
File diff suppressed because one or more lines are too long
|
|
@ -205,32 +205,49 @@ pub fn next_cluster(s: &str, i: usize) -> usize {
|
|||
GraphemeCursor::new(i, s.len(), true).next_boundary(s, 0).ok().flatten().unwrap_or(s.len())
|
||||
}
|
||||
|
||||
/// Part of a word: a letter, digit or mark (Thai vowel and tone marks, combining accents), an
|
||||
/// underscore or an apostrophe.
|
||||
fn is_word_char(c: char) -> bool {
|
||||
c.is_alphanumeric() || c == '_' || c == '\'' || c == '’'
|
||||
use unicode_general_category::{GeneralCategory as G, get_general_category};
|
||||
c.is_alphanumeric() || matches!(c, '_' | '\'' | '’') || matches!(get_general_category(c), G::NonspacingMark | G::SpacingMark | G::EnclosingMark)
|
||||
}
|
||||
|
||||
/// Start of the previous word (Cmd/Alt+Left).
|
||||
/// Is byte `b` of `s` (with `breaks`, its [`crate::shape::unspaced_word_breaks`]) a boundary
|
||||
/// between two words of Thai, Lao, Khmer or Myanmar, which no space separates?
|
||||
fn inner_word_break(s: &str, breaks: &[usize], b: usize) -> bool {
|
||||
use crate::shape::is_unspaced_script;
|
||||
breaks.binary_search(&b).is_ok()
|
||||
&& s.get(b..).and_then(|t| t.chars().next()).is_some_and(is_unspaced_script)
|
||||
&& s.get(..b).and_then(|t| t.chars().next_back()).is_some_and(is_unspaced_script)
|
||||
}
|
||||
|
||||
/// Start of the previous word (Cmd/Alt+Left). Thai (Lao, Khmer, Myanmar) words end where their
|
||||
/// script's word boundaries are, spaces or not.
|
||||
pub fn prev_word(s: &str, i: usize) -> usize {
|
||||
let i = floor_char(s, i);
|
||||
let breaks = crate::shape::unspaced_word_breaks(s);
|
||||
let chars: Vec<(usize, char)> = s[..i].char_indices().collect();
|
||||
let mut k = chars.len();
|
||||
while k > 0 && !is_word_char(chars[k - 1].1) {
|
||||
k -= 1;
|
||||
}
|
||||
while k > 0 && is_word_char(chars[k - 1].1) {
|
||||
let end = k;
|
||||
while k > 0 && is_word_char(chars[k - 1].1) && !(k < end && inner_word_break(s, &breaks, chars[k].0)) {
|
||||
k -= 1;
|
||||
}
|
||||
chars.get(k).map(|c| c.0).unwrap_or(i)
|
||||
}
|
||||
|
||||
/// End of the next word (Cmd/Alt+Right).
|
||||
/// End of the next word (Cmd/Alt+Right): see [`prev_word`].
|
||||
pub fn next_word(s: &str, i: usize) -> usize {
|
||||
let i = floor_char(s, i);
|
||||
let breaks = crate::shape::unspaced_word_breaks(s);
|
||||
let mut it = s[i..].char_indices().peekable();
|
||||
while it.peek().is_some_and(|(_, c)| !is_word_char(*c)) {
|
||||
it.next();
|
||||
}
|
||||
while it.peek().is_some_and(|(_, c)| is_word_char(*c)) {
|
||||
let start = it.peek().map(|(k, _)| i + k);
|
||||
while it.peek().is_some_and(|(k, c)| is_word_char(*c) && !(Some(i + k) != start && inner_word_break(s, &breaks, i + k))) {
|
||||
it.next();
|
||||
}
|
||||
it.peek().map(|(k, _)| i + k).unwrap_or(s.len())
|
||||
|
|
@ -260,16 +277,18 @@ pub fn word_at(s: &str, i: usize) -> Range<usize> {
|
|||
if k == 2 {
|
||||
return at..at + c.len_utf8();
|
||||
}
|
||||
// Thai (Lao, Khmer, Myanmar) words also end where their script's word boundaries are.
|
||||
let breaks = if k == 0 { crate::shape::unspaced_word_breaks(s) } else { vec![] };
|
||||
let mut a = at;
|
||||
for (p, d) in s[..at].char_indices().rev() {
|
||||
if class(d) != k {
|
||||
if class(d) != k || inner_word_break(s, &breaks, a) {
|
||||
break;
|
||||
}
|
||||
a = p;
|
||||
}
|
||||
let mut b = at;
|
||||
for (p, d) in s[at..].char_indices() {
|
||||
if class(d) != k {
|
||||
if class(d) != k || (b > at && inner_word_break(s, &breaks, b)) {
|
||||
break;
|
||||
}
|
||||
b = at + p + d.len_utf8();
|
||||
|
|
|
|||
|
|
@ -241,13 +241,12 @@ pub(crate) fn is_unspaced_script(c: char) -> bool {
|
|||
matches!(c.script(), Script::Thai | Script::Lao | Script::Khmer | Script::Myanmar)
|
||||
}
|
||||
|
||||
/// Mark the glyphs of `sg` (shaped from `text`, which starts at byte `start` of the text the
|
||||
/// glyphs index) after which a word of Thai, Lao, Khmer or Myanmar ends: where a line may break in
|
||||
/// text without spaces between words. The words come from ICU4X's line segmenter (its LSTM models
|
||||
/// for those scripts); other text keeps its own breaks.
|
||||
pub(crate) fn mark_word_ends(text: &str, start: usize, sg: &mut [SGlyph]) {
|
||||
// The dictionaries on the desktop (the most accurate, a few megabytes); the far smaller LSTM
|
||||
// models on the web.
|
||||
/// The byte offsets of `text` where a word of Thai, Lao, Khmer or Myanmar starts or ends: where a
|
||||
/// line may break, and where the caret stops word by word, in text without spaces between words.
|
||||
/// They come from ICU4X's line segmenter (its dictionaries on the desktop, its far smaller LSTM
|
||||
/// models on the web); empty when `text` has none of those scripts. Offsets elsewhere (around
|
||||
/// other text) are UAX #14's breaks.
|
||||
pub(crate) fn unspaced_word_breaks(text: &str) -> Vec<usize> {
|
||||
#[cfg(not(target_arch = "wasm32"))]
|
||||
static SEGMENTER: std::sync::LazyLock<icu_segmenter::LineSegmenterBorrowed<'static>> =
|
||||
std::sync::LazyLock::new(|| icu_segmenter::LineSegmenter::new_dictionary(Default::default()));
|
||||
|
|
@ -255,9 +254,19 @@ pub(crate) fn mark_word_ends(text: &str, start: usize, sg: &mut [SGlyph]) {
|
|||
static SEGMENTER: std::sync::LazyLock<icu_segmenter::LineSegmenterBorrowed<'static>> =
|
||||
std::sync::LazyLock::new(|| icu_segmenter::LineSegmenter::new_auto(Default::default()));
|
||||
if !text.chars().any(is_unspaced_script) {
|
||||
return vec![];
|
||||
}
|
||||
SEGMENTER.segment_str(text).collect()
|
||||
}
|
||||
|
||||
/// Mark the glyphs of `sg` (shaped from `text`, which starts at byte `start` of the text the
|
||||
/// glyphs index) after which a word of Thai, Lao, Khmer or Myanmar ends ([`unspaced_word_breaks`]);
|
||||
/// other text keeps its own breaks.
|
||||
pub(crate) fn mark_word_ends(text: &str, start: usize, sg: &mut [SGlyph]) {
|
||||
let breaks: Vec<usize> = unspaced_word_breaks(text).into_iter().map(|b| b + start).collect();
|
||||
if breaks.is_empty() {
|
||||
return;
|
||||
}
|
||||
let breaks: Vec<usize> = SEGMENTER.segment_str(text).map(|b| b + start).collect();
|
||||
for g in sg.iter_mut().filter(|g| g.inline.is_none() && is_unspaced_script(g.ch)) {
|
||||
g.word_end = breaks.binary_search(&(g.byte + g.len)).is_ok();
|
||||
}
|
||||
|
|
|
|||
|
|
@ -805,3 +805,22 @@ fn caret_moves_over_whole_clusters() {
|
|||
assert_eq!((edit::prev_cluster(s, 6), edit::next_cluster(s, 6)), (3, 12));
|
||||
assert_eq!((edit::next_cluster("", 0), edit::prev_cluster("ab", 99)), (0, 1));
|
||||
}
|
||||
|
||||
/// Word steps and double-click stop between Thai words, which no space separates: ภาษา|ไทย|ง่าย.
|
||||
#[test]
|
||||
fn thai_words_are_words_to_the_caret() {
|
||||
let s = "ภาษาไทยง่าย ok";
|
||||
let (thai, easy) = ("ภาษา".len(), "ภาษาไทย".len());
|
||||
let end = "ภาษาไทยง่าย".len();
|
||||
assert_eq!(edit::next_word(s, 0), thai);
|
||||
assert_eq!(edit::next_word(s, thai), easy);
|
||||
assert_eq!(edit::next_word(s, easy), end);
|
||||
assert_eq!(edit::prev_word(s, end), easy);
|
||||
assert_eq!(edit::prev_word(s, easy), thai);
|
||||
assert_eq!(edit::prev_word(s, thai), 0);
|
||||
assert_eq!(edit::word_at(s, thai + 3), thai..easy, "double-click inside ไทย");
|
||||
assert_eq!(edit::word_at(s, 0), 0..thai);
|
||||
// Latin keeps its words.
|
||||
assert_eq!(edit::word_at(s, s.len() - 1), end + 1..s.len());
|
||||
assert_eq!(edit::next_word("hello world", 0), 5);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -42,7 +42,7 @@ against Illustrator's public documentation; **estimated** unless noted.
|
|||
| Touch Type tool | missing | listed in the toolbar, not implemented |
|
||||
| Retype (identify fonts in outlines/images) | missing | Illustrator's uses a cloud model |
|
||||
| Snap to Glyph | missing | menu stub |
|
||||
| Thai, Lao, Khmer, Myanmar: shaping (marks, reordering), lines broken between words | done | words found by ICU4X's line segmenter: dictionaries on the desktop, LSTM models on the web |
|
||||
| Thai, Lao, Khmer, Myanmar: shaping (marks, reordering), lines broken between words, word-by-word caret and double-click | done | words found by ICU4X's line segmenter: dictionaries on the desktop, LSTM models on the web |
|
||||
| Middle Eastern: bidirectional layout, Paragraph Direction | done | |
|
||||
| Middle Eastern: Character Direction, digit types, kashidas, Middle Eastern composers, split caret | missing | |
|
||||
| CJK: input methods (marked text, candidate window) | done | |
|
||||
|
|
@ -55,6 +55,7 @@ against Illustrator's public documentation; **estimated** unless noted.
|
|||
|
||||
| Date | Change | Summary |
|
||||
|---|---|---|
|
||||
| 2026-10-11 | minor | Thai, Lao, Khmer and Myanmar words for the caret (Ctrl/Alt+arrows) and double-click; combining marks count as part of a word; score unchanged (estimated) |
|
||||
| 2026-10-11 | minor | Font menu: a Thai script filter beside Japanese and Latin; score unchanged (estimated) |
|
||||
| 2026-10-11 | minor | Thai, Lao, Khmer and Myanmar text breaks its lines between words (ICU4X line segmenter) instead of only at spaces; score unchanged (estimated) |
|
||||
| 2026-10-11 | minor | The Type tool's arrows and Delete move by grapheme cluster, never between a Thai consonant and its vowel or tone marks; score unchanged (estimated) |
|
||||
|
|
|
|||
Loading…
Reference in New Issue