AI Search keyword search now understands Japanese questions!
Hi everyone, it's Shii-chan!
Cloudflare Changelog
developers.cloudflare.com
If you want to search documents written in Japanese, this one is for you!
What was announced?
The Cloudflare Changelog published an update for AI Search. Keyword search with the trigram tokenizer now matches Japanese, Chinese, and Korean queries.
The example in the post is a full natural-language question: 経費精算の締め日はいつですか？ ("When is the expense report deadline?").
The story so far
Before, a query without spaces was treated as a single exact phrase.
Japanese sentences don't use spaces between words, so pasting in a whole question usually returned no keyword results.
The post also notes that the porter tokenizer does not work well for these languages.
What changes
- Space-free questions in Japanese now return keyword matches
- If your content is in Japanese, Chinese, or Korean, you can make real use of keyword search in AI Search
If your internal docs or help pages are in Japanese, you should miss fewer results!
Dive Deep
If your content is in Japanese, Chinese, or Korean, set keyword_tokenizer to trigram.
keyword_tokenizer: trigram
One thing to keep in mind:
- Changing the tokenizer triggers a full reindex
If you have a lot of content, pick a good time to switch. The Keyword search page in the docs has the tokenizer details.
Wrap-up
- Keyword search with the trigram tokenizer matches Japanese, Chinese, and Korean queries
- Before, queries without spaces were one exact phrase and rarely returned results
- For content in these languages, set keyword_tokenizer to trigram
- Changing the tokenizer triggers a full reindex
This is for anyone using AI Search with Japanese content, or planning to!
