How textsearch_intl performs by language

textsearch_intl splits text with ICU’s word-break rules. For scripts written without spaces (Chinese, Japanese, Thai, Lao, Khmer, Burmese), ICU looks words up in its dictionaries; everywhere else it splits on spaces and punctuation like the default parser does. So it helps most where the default parser fails completely, and it does nothing for problems that come from word forms: particles, clitics, compounds and inflection.

The evidence comes from three places:

  • Benchmark: 500,000 Wikipedia titles per language, with recall and overmatch scored on 250 labeled term/title pairs per language (bench/results.md). Recall is the share of real word matches a method returns; overmatch is the share of its results that aren’t real matches.
  • Production place names: single-word searches against a table of about 550 million business and place names, comparing textsearch_intl with the default parser (simple) and a pg_trgm substring index.
  • Examples: every tokenization below was produced by textsearch_intl on ICU 78.3. Another ICU version can split some dictionary words differently.

Summary

Language Verdict Benchmark recall (default → intl_search → intl_search_ngram) Overmatch (intl_search / ngram) Main weakness
Chinese Good 10% → 93% → 99% 16% / 18% Transliterated names split into single characters; some compounds stay whole
Japanese Good for kanji and katakana 7% → 86% → 100% 20% / 54% Hiragana words split into syllables with intl_search; compounds
Thai Good with intl_search_ngram 5% → 79% → 98% 44% / 69% intl_search alone splits unknown names and loanwords unpredictably
Lao, Khmer, Burmese Likely good (not benchmarked) Same dictionary approach as Thai
Korean Good with intl_search_ngram 47% → 48% → 100% 1% / 22% intl_search alone leaves particles and compound names attached
Vietnamese No gain Words span several space-separated syllables
Arabic Better with intl_search Prepositions other than و stay attached
Hebrew No gain Articles and prepositions are attached to words
English and other spaced languages Same as default No stemming or compound splitting

Advice that applies to every language

  • Query with phraseto_tsquery, not plainto_tsquery. When ICU splits a name into pieces, a plain query requires the pieces to appear anywhere in the title, in any order. 星巴克 (Starbucks) splits into 星|巴|克, so plainto_tsquery matches 克巴星餐厅; phraseto_tsquery requires them adjacent and in order, and doesn’t.
  • Use the intl_search configuration. It applies NFKC, case folding, accent folding and apostrophe removal to each token, so half-width katakana (ｾﾌﾞﾝ), full-width Latin (ＡＢＣ), case (ΑΘΗΝΑΣ, STRASSE), accents (café, ё), apostrophes (McDonald's, Oʻahu), Arabic letter variants (أحمد) and non-ASCII digits (٧) match the way users type them. It also indexes French and Italian words with and without an elided article (L'Occitane and occitane), and Arabic words with and without the article. It keeps the Thai and Lao vowels intact, which a plain normalize(name, NFKC) before tokenizing would split (จำกัด → จํา|กัด).
  • Use intl_search_ngram for Korean, Chinese and Japanese. It adds character unigrams and bigrams for Hangul, Han and kana tokens, so a word matches inside a longer word: particles, compounds and branch names. It costs overmatch, the same as pg_bigm’s, and leaves every other script as intl_search does.
  • Pair it with an n-gram index for names ICU doesn’t know. Outside Hangul, Han and kana, textsearch_intl matches whole words only. For Thai fragments and partial Latin words, add pg_bigm (or pg_trgm for terms of three or more characters) and OR the two conditions.

Chinese: good

The biggest improvement of any language. On Wikipedia titles, recall goes from 10% with the default parser to 93%, at 16% overmatch against 18% for substring search. On place names, single-word searches for common words found 100–1,200 times more rows than the default parser:

Search default parser textsearch_intl pg_trgm
公園 (park) 315 116,892 timed out
銀行 (bank) 45 54,071 timed out
機場 (airport) 44 5,027 timed out
麥當勞 (McDonald’s) 680 722 722
星巴克 (Starbucks) 922 1,169 1,173

pg_trgm can’t use its index for one- and two-character terms, so those searches read the whole index and timed out at 20 seconds.

Where it falls short:

  • Transliterated foreign names aren’t in ICU’s dictionary and come out as single characters: 朱莉娅·加布里埃莱斯基 → 朱|莉|娅|加布里|埃|莱|斯|基, and 星巴克咖啡 → 星|巴|克|咖啡. A phrase query still finds them, but a plain query overmatches.
  • Some compounds stay whole. 國際機場 (international airport, Traditional) is one token, so a search for 機場 misses it, while the Simplified 国际机场 splits into 国际|机场. This is most of the 7% the benchmark missed, and intl_search_ngram finds it (99% recall, 18% overmatch).
  • Traditional and Simplified characters match each other with intl_search and intl_search_ngram, which write traditional characters as simplified: 麥當勞 finds 麦当劳 and the other way round. intl_simple keeps them apart.

Japanese: good for kanji and katakana, weak for hiragana

Recall goes from 7% to 86%, with 20% overmatch against 54% for substring search. Kanji compounds split well (東京国際空港 → 東京|国際|空港, so a search for 空港 finds it). Katakana splits at hyphens and middle dots: セブン-イレブン渋谷店 → セブン|イレブン|渋谷|店, so a search for セブンイレブン finds the hyphenated form. On place names that found 18,268 rows, against 541 for a substring search that expected no hyphen.

Where it falls short:

  • Hiragana-only words split into syllables: ひらがな → ひ|ら|が|な, やきとり → や|き|とり, らーめん → ら|ー|めん. Verb forms fragment the same way: 食べました → 食|べ|ま|した. intl_normalize writes hiragana as katakana, and intl_search_ngram’s parser keeps adjacent kana words together as one run, so らーめん and ラーメン find each other there. With intl_search they still don’t, because the hiragana spelling is split and the katakana one isn’t.
  • Unknown katakana names can come out as fragments (ンズ, ジョ), which then match inside unrelated names. This is most of the 20% overmatch.
  • Half-width katakana (ｾﾌﾞﾝｲﾚﾌﾞﾝ) only matches full-width with intl_search, which applies NFKC.
  • No morphology. ICU isn’t a morphological analyzer like MeCab or Kuromoji, so it won’t link inflected forms or break up every compound.

intl_search_ngram raises recall to 100% and finds words inside compounds (駅 in 駅前), but overmatch rises to 54%, the same as pg_bigm’s, because Japanese titles are full of short kana sequences that also occur inside other words.

Thai: use intl_search_ngram

Recall goes from 5% to 79%, at 44% overmatch against 77% for substring search. Dictionary words split cleanly: ร้านกาแฟสด → ร้าน|กาแฟ|สด, โรงพยาบาลกรุงเทพ → โรง|พยาบาล|กรุงเทพ. On place names, textsearch_intl found 39,019 rows for กาแฟ (coffee) against 3,808 for the default parser and 40,492 for substring search.

Where it falls short:

  • Unknown names pull letters off the next word. กุ๊งกิ๊งกาแฟโบราณ → กุ๊|งกิ๊|งกาแฟ|โบราณ: the name isn’t in the dictionary, so กาแฟ picks up its last letter and a search for it misses.
  • Loanwords split differently depending on what follows them. เซเว่น (Seven) on its own is เซ|เว่น, but in เซเว่นอีเลฟเว่น (7-Eleven) it’s เซ|เว่|นอีเลฟเว่น, so a search for เซเว่น doesn’t find the full brand name. On place names, textsearch_intl found 2,094 rows against 3,794 for substring search.
  • Thai followed directly by Latin letters stays joined: ร้านกาแฟcoffee → ร้าน|กาแฟcoffee.
  • Fragments overmatch. Short fragments like อง and ซ์ come out of unknown names as tokens and match inside unrelated words.

intl_search_ngram avoids ICU’s guesses: its intl_runs parser keeps each unspaced Thai run whole and indexes every pair and every three adjacent grapheme clusters, so เซเว่น finds เซเว่นอีเลฟเว่น and กาแฟ finds กุ๊งกิ๊งกาแฟโบราณ. Recall goes to 98% at 69% overmatch, against 77% for substring search; the overmatch is high on this benchmark because its query terms include ICU fragments like อง. The index is about three times intl_search’s for Thai text.

Lao, Khmer, Burmese: likely good, not benchmarked

ICU has dictionaries for all three, and sentences split into words: ຮ້ານອາຫານລາວ → ຮ້ານ|ອາຫານ|ລາວ, ភោជនីយដ្ឋានខ្មែរ → ភោជនីយដ្ឋាន|ខ្មែរ, မြန်မာစားသောက်ဆိုင် → မြန်မာ|စားသောက်ဆိုင်. Expect the same weaknesses as Thai with unknown names.

Korean: use intl_search_ngram

Korean puts spaces between phrases, but attaches particles, endings and other nouns directly to a word. ICU has no Korean dictionary, so each space-separated run is one token, exactly as with the default parser. Recall was 48% against 47% for the default parser.

  • Particles: 서울의 맛집 → 서울의|맛집, so a search for 서울 doesn’t match.
  • Brand plus branch names: 스타벅스안양평촌점 (Starbucks, Anyang Pyeongchon branch) is a single token. On place names, a search for 편의점 (convenience store) found 2,041 rows with textsearch_intl and 9,096 with substring search, because names like 행복한편의점 are one word.

intl_search_ngram fixes both: it indexes every Hangul syllable and syllable pair, so 서울 matches 서울의, 편의점 matches 행복한편의점, and 스타벅스 matches 스타벅스안양평촌점. Recall goes to 100% with 22% overmatch, the same as pg_bigm, at a similar index size. The overmatch is the substring kind: 스타벅스 also matches 파스타벅스. For true morphology, use an analyzer outside Postgres (such as Elasticsearch’s nori).

Vietnamese: no gain

Vietnamese writes each syllable separately, and most words are two or more syllables. ICU splits at every space, like the default parser: Nhà hàng Phở Hà Nội → Nhà|hàng|Phở|Hà|Nội. Use phraseto_tsquery so Hà Nội matches as a unit. intl_search folds the diacritics. With intl_join’s max_length set it also indexes adjacent short syllables run together, so hanoi finds Hà Nội and saigon finds Sài Gòn; the other direction (sai gon finding Saigon) still doesn’t match.

Arabic: better with intl_search; Hebrew: no gain

Spaces separate words, but the article and some prepositions and conjunctions are written as part of the word. intl_search indexes a word that starts with the article ال or وال both with and without it, so a search for مطعم (restaurant) finds المطعم (the restaurant). It also folds the letter variants people type interchangeably in names (أحمد and احمد, مدرسة and مدرسه, Persian ی and ک) and removes short-vowel marks. Other prefixes stay attached: بالمطعم (in the restaurant) is one token, and a word that merely starts with ال (الله) also gets the shortened form, a small source of overmatch.

Hebrew does the same with המסעדה against מסעדה, and textsearch_intl doesn’t remove Hebrew prefixes.

English and other spaced languages: same as the default parser

For English, Russian, Greek, Hindi, German, Turkish and other languages that separate words with spaces, ICU’s splits match the default parser’s, and tokenizing English is slightly faster (0.63 µs against 0.73 µs per title). Search results were the same within 0.5%: starbucks found 54,395 place names with textsearch_intl and 54,352 with the default parser. Differences in detail:

  • Apostrophes stay inside the word: McDonald's is one token, mcdonald's, so a search for mcdonalds or mcdonald doesn’t match it. The default parser splits it into mcdonald|s instead. intl_search removes the apostrophe, so mcdonalds matches, and indexes French and Italian elisions both ways, so occitane finds L'Occitane.
  • Case folds fully: intl_search uses Unicode case folding rather than the database’s lower(), so Greek final sigma (ΑΘΗΝΑΣ, Αθήνας), German ß (STRASSE) and Turkish İ match.
  • Hyphens and ampersands split: 7-Eleven → 7|eleven, co-op → co|op, AT&T → at|t. Digits and letters split too (7Eleven → 7|eleven).
  • Case changes inside a word split: intl_search indexes CarrefourSA as the whole word and as its parts, so carrefour finds it.
  • Words written together or apart (Mc Donald's and McDonald's) match only with intl_join’s max_length set, which adds the two words run together (see the README).
  • URLs and hostnames stay whole: www.example.com is one token.
  • No stemming or compound splitting. Kaffeehaus doesn’t match haus. For stemming, map word to a Snowball dictionary: ALTER TEXT SEARCH CONFIGURATION my_config ALTER MAPPING FOR word WITH english_stem.