Akiba So 0cf58fb67a feat(f8): CJK-safe search text normalization + matching ...

normalizeForIndex (lowercase + collapse whitespace runs incl. hard PDF/OCR
newlines + trim) and matchesNormalized so a query matches across the line breaks
in raw extracted text. Deliberately NO word-tokenization: Chinese has no
inter-word spaces, so substring match over normalized text is correct for both
Latin and CJK (段/word segmentation belongs in the DB FTS tokenizer). Verified
on CJK inputs (你好/笔记应用).

flutter analyze lib/editor clean; 181/181 tests (+9).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

2026-06-23 03:30:42 +08:00

1.7 KiB

Raw Blame History

View Raw