babel-memory

---

Why This Exists

Whitespace-tokenized retrieval pipelines commonly lose recall on languages that do not use spaces between words. Research across 8 academic papers (MMTEB, XRAG, MIT 2025) describes a broader 5-layer semantic loss cascade:

Layer	What Breaks	Impact
Token estimation	string.length / 4 underestimates CJK by 4-8x	Context overflow
BM25 tokenization	Whitespace split on Chinese = 0 matches	Hybrid search degrades to vector-only
LLM extraction	English-only KG/summary prompts	-24% factual accuracy on non-English
Cross-lingual retrieval	Query/document language mismatch	-56% recall (XRAG benchmark)
Auto-evaluation	LLM-as-Judge overestimates non-English quality	Problems go systematically unreported

babel-memory is the fix for layers 2-4. Same simple API. Zero required dependencies. Install only the language packs you need.

Before & After

BEFORE babel-memory:
  Store: "机器学习在自然语言处理中的应用"
  BM25 search("机器学习") → [] (zero results)
  KG extract → English prompt struggles with Chinese entities
  
AFTER babel-memory:
  Store: "机器学习在自然语言处理中的应用"
         → fts_text: "机器 学习 机器学习 自然 语言 处理 自然语言 应用"
  BM25 search("机器学习") → [match found!]
  KG extract → Chinese prompt with CJK few-shot examples

BEFORE babel-memory (European):
  Store: "Maschinelles Lernen verbessert die Verarbeitung"
  BM25 search("Verarbeitung") → [match]
  BM25 search("verarbeitet") → [] (different form, zero results)

AFTER babel-memory + snowball-stemmers:
  Store: "Maschinelles Lernen verbessert die Verarbeitung"
         → fts_text: "maschinell lern verbess verarbeit"
  BM25 search("verarbeitet") → stem("verarbeitet") = "verarbeit" → [match found!]

Modular Install

# Core only (zero dependencies — pure TypeScript)
npm install babel-memory

# Add language packs as needed:
npm install jieba-wasm          # Chinese (highest quality)
npm install @sglkc/kuromoji     # Japanese (highest quality)
npm install wordcut             # Thai
npm install snowball-stemmers   # 16 mapped algorithms currently usable by this library
npm install tinyld              # auto-detect Latin-script languages (de/fr/es/...)

You only pay for what you use. The core package has zero dependencies — language packs are loaded lazily at runtime. And thanks to the built-in Intl.Segmenter tier (ICU dictionaries shipped with Node 16+/Bun/Deno/browsers), a zero-dependency install already gets word-level segmentation for Chinese, Japanese and Thai — installing the packs upgrades quality further. It never crashes.

Quick Start

import { detectLanguage, initTokenizer, tokenizeForFts, getKgPrompt } from "babel-memory";

// 1. Initialize once at startup (loads whichever packages are installed)
await initTokenizer();

// 2. Detect language (zero dependencies, pure Unicode analysis)
detectLanguage("这个项目的架构设计非常优秀");  // "zh"
detectLanguage("東京タワーはとても高いです");    // "ja" (not "zh" — hiragana detected first)
detectLanguage("이 프로젝트는 매우 훌륭합니다"); // "ko"
detectLanguage("สวัสดีครับ");                    // "th"
detectLanguage("مرحبا بالعالم");                 // "ar"
detectLanguage("Машинное обучение");              // "ru"

// 3. Pre-tokenize for BM25 (the core fix)
tokenizeForFts("机器学习很有趣", "zh");
// → "机器 学习 很 有趣"  (jieba word segmentation)

tokenizeForFts("東京タワー", "ja");
// → "東京 タワー"  (kuromoji word segmentation)

tokenizeForFts("Maschinelles Lernen", "de");
// → "maschinell lern"  (Snowball stemming)

// 4. ⚠️ CRITICAL: apply the SAME tokenization to your queries.
// Tokenization only matches when index side and query side agree:
const ftsQuery = tokenizeForFts(userQuery, detectLanguage(userQuery));
// store side:  "机器学习很有趣" → "机器 学习 很 有趣"  (indexed)
// query side:  "机器学习"       → "机器 学习"          (matches!)
// forgetting this = querying one giant token "机器学习" against
// segmented text = zero results. This is THE classic FTS pre-tokenization trap.

// 5. Get language-matched prompts for LLM calls
const { system, userTemplate } = getKgPrompt("zh"); // → Chinese prompt
getKgPrompt("ja"); // → native Japanese prompt
getKgPrompt("ko"); // → English prompt (instructions in a third language hurt quality)
// Predicates stay English (normalized keys) in every variant

// 6. Optional power tools
import { detectLanguageDetailed, detectLanguageExtended, getLoadedTokenizers } from "babel-memory";

detectLanguageDetailed("我在用 TypeScript 给 RecallNest 写 tokenizer hook");
// → { language: "en", scripts: { cjk: 0.13, latin: 0.83, ... }, isMixed: true }
// AI conversation logs mix CN/EN constantly — isMixed tells you when.
// Segmentation-needing scripts (CJK/Thai) flag mixed from a ~2% ratio:
// even a tiny Chinese island is unsearchable if left unsegmented.
// (Embedded CJK runs are segmented automatically either way; see below.)

detectLanguageExtended("Das ist ein guter Tag");
// → "de" when tinyld is installed; "en" otherwise (graceful degrade)

tokenizeForFts("机器学习的应用", "zh", { removeStopwords: true });
// → "机器 学习 应用"  (的 removed; opt-in, default off)

getLoadedTokenizers(); // → ["jieba", "kuromoji", "tinyld"] — verify deployment

// Selective init when you don't need everything (kuromoji alone = ~1-2s + ~17MB):
await initTokenizer({ languages: ["zh", "de"] });

How It Works

The key insight: pre-tokenize non-whitespace-delimited text before FTS indexing, and stem inflected languages.

Standard FTS pipeline (broken for Chinese):
  "知识图谱提取" → whitespace split → ["知识图谱提取"] → 1 giant token → no matches

babel-memory pipeline (fixed):
  "知识图谱提取" → jieba segmentation → "知识 图谱 知识图谱 提取" → whitespace split → 4 tokens → matches!

This works with any whitespace-based FTS engine: Tantivy (LanceDB), SQLite FTS5, Elasticsearch, Meilisearch. No engine modifications needed.

Small deterministic smoke benchmark

bun bench/recall-benchmark.ts — SQLite FTS5, 20 Chinese documents, 12 queries:

Tier	Recall	Queries returning nothing
raw text (what most memory systems do)	0.0%	12/12
zero-dependency tier (Intl.Segmenter + bigrams)	100%	0/12
full tier (jieba)	100%	0/12

On this fixed 20-document / 12-query Chinese fixture, raw text returns no results while both preprocessing tiers return all expected documents. This is a regression smoke test, not a general retrieval-quality benchmark.

Mixed-script text (the AI-conversation reality)

Real agent conversations constantly mix languages: "I fixed 机器学习模型 using TensorFlow". Ratio-based detection lands on zh or en depending on the balance — and either route now handles the mix correctly: the zh path's tokenizers keep Latin tokens intact, and the en path detects embedded CJK/Hangul/Thai runs and routes each run to its proper tokenizer. Before v2.1, a Chinese island inside en-classified text passed through unsegmented and unsearchable. Use detectLanguageDetailed() if you want the mixing signal explicitly.

Detection Order Matters

Japanese uses kanji (CJK characters). Naive CJK detection would misclassify Japanese as Chinese. babel-memory checks language-unique scripts first:

1. Hiragana/Katakana present? → Japanese (unique to Japanese)
2. Hangul present? → Korean (unique to Korean)
3. Thai script? → Thai
4. Arabic script? → Arabic
5. Devanagari? → Hindi
6. Cyrillic? → Russian
7. CJK Ideographs without Japanese/Korean markers? → Chinese
8. Default → English

Graceful Degradation

babel-memory never crashes due to a missing optional package. Each language has a three-tier fallback chain — and since v2.1 the middle tier (ICU via Intl.Segmenter, built into Node 16+/Bun/Deno) gives zero-dependency installs word-level quality:

Language	Tier 1: package installed	Tier 2: zero deps (built-in ICU)	Tier 3: last resort
Chinese	jieba search-mode segmentation	ICU word segmentation + CJK bigrams	Character split
Japanese	kuromoji segmentation	ICU word segmentation + CJK bigrams	Character split
Thai	wordcut segmentation	ICU word segmentation	Passthrough
European (de, fr, es...)	Snowball stemming	Passthrough	Passthrough
Korean	—	Syllable-level split (deliberate: see note)	Same
Arabic, Russian	Snowball stemming	Passthrough	Passthrough
Hindi	—	Passthrough (space-delimited)	Same
English	—	Passthrough	Same

Why bigrams? ICU emits whole compounds ("東京タワー" as one token), which breaks partial-match queries like "タワー". CJK tokens of length ≥ 3 get bigram expansion on both index and query side — the same approach lunr-languages uses.

Why is Korean syllable-level? Korean is agglutinative: "프로젝트는" = "프로젝트" + topic particle. Word-level tokens make the query "프로젝트" miss entirely; syllable split keeps BM25 partial matching stable.

A warning is logged once per missing package so you know what to install for better quality. Your application keeps working regardless.

API Reference

Function	Input	Output	Description
detectLanguage(text)	string	Language	Unicode script ratio analysis. Detects zh, ja, ko, th, ar, hi, ru, en. Zero dependencies.
detectLanguageDetailed(text)	string	LanguageDetail	Same + per-script ratios and isMixed flag for mixed-script text.
detectLanguageExtended(text)	string	string	Refines Latin-script text to de/fr/es/... when tinyld is installed; otherwise identical to detectLanguage.
initTokenizer(opts?)	{ languages?: string[] }	Promise<void>	Load available tokenizers in parallel. Pass languages to load selectively. Idempotent, non-fatal.
tokenizeForFts(text, lang, opts?)	string, string, { removeStopwords? }	string	NFKC-normalize, then pre-tokenize for BM25. Apply to both indexed text and queries.
getLoadedTokenizers()	—	string[]	Diagnostic: which optional packs are actually loaded.
segmentWithIntl(text, locale)	string, string	string | null	Raw ICU word segmentation building block. null when unavailable.
intlWithBigrams(text, locale)	string, string	string | null	ICU segmentation + CJK bigram expansion (what the zh/ja fallback uses).
getKgPrompt(lang)	string	{ system, userTemplate }	KG triple extraction prompt. zh → Chinese, ja → Japanese, others → English.
getSessionPrompt(lang)	string	{ system, dimensionLabels }	Session summary prompt, same language routing. 9 structured dimensions.

Type: Language = "zh" | "ja" | "ko" | "th" | "ar" | "hi" | "ru" | "en"

tokenizeForFts also accepts any Snowball language code (e.g., "de", "fr", "es") as a string.

Note on script-based detection limits: detectLanguage cannot distinguish languages that share the Latin alphabet — German, French and Spanish all return "en". That's why detectLanguageExtended + the optional tinyld pack exist: install it and Latin-script languages resolve automatically, completing the auto-detect → Snowball stemming chain. Traditional Chinese note: ICU and jieba both handle zh-Hant text, but jieba's dictionary is Simplified-centric; for mixed Simplified/Traditional corpora consider normalizing externally (OpenCC) before indexing.

Supported Languages

Auto-Detected (via detectLanguage)

Code	Language	Script	FTS Strategy	Package
zh	Chinese	CJK Ideographs	jieba search-mode word segmentation	jieba-wasm
ja	Japanese	Hiragana + Katakana + CJK	kuromoji word segmentation	@sglkc/kuromoji
ko	Korean	Hangul + CJK	Character-level split	(built-in)
th	Thai	Thai script	wordcut segmentation	wordcut
ar	Arabic	Arabic script	Snowball stemming	snowball-stemmers
hi	Hindi	Devanagari	Passthrough	(none)
ru	Russian	Cyrillic	Snowball stemming	snowball-stemmers
en	English	Latin	Passthrough	(none)

Snowball-Stemmed Languages (pass code to tokenizeForFts)

Code	Language	Code	Language
de	German	nl	Dutch
fr	French	sv	Swedish
es	Spanish	no	Norwegian
pt	Portuguese	da	Danish
it	Italian	fi	Finnish
hu	Hungarian	tr	Turkish
ro	Romanian	cs	Czech

Total: 22 distinct supported language codes: 8 auto-detected codes plus 14 additional explicit Snowball codes. Arabic and Russian are already included in the auto-detected eight.

With the optional tinyld pack installed, detectLanguageExtended() can route the supported Latin-script languages into their mapped Snowball algorithms without the caller supplying a code.

Who Is This For

• AI memory system builders — if you're building on LanceDB, ChromaDB, or any vector+BM25 hybrid store
• RAG pipeline developers — if your users speak non-English languages and BM25 returns empty
• MCP server authors — if your memory tools need multilingual support
• Anyone who's noticed their AI agent "forgets" non-English conversations

Scope and Integration

Capability	What babel-memory provides
CJK/Thai segmentation	Optional jieba / kuromoji / wordcut, with ICU-based zero-dependency fallbacks
Mixed-script text	Script-run routing inside otherwise Latin-classified text
Stemming	16 currently usable Snowball mappings; 14 are additional to the 8 auto-detected codes
Language detection	8 script-based codes, with optional tinyld refinement for supported Latin-script mappings
Language-matched prompts	English, Chinese, and Japanese KG/session prompts
FTS integration	Emits whitespace-delimited text for Tantivy, SQLite FTS5, Elasticsearch, Meilisearch, and similar engines
Required dependencies	0

babel-memory is not a memory system — it's a preprocessing layer that makes any memory system work properly across languages.

Used By

• RecallNest — MCP-native shared memory for Claude Code, Codex, and Gemini CLI

Research References

This library is informed by findings from:

• MMTEB: Massive Multilingual Text Embedding Benchmark (arXiv 2502.13595)
• XRAG: Cross-lingual Retrieval-Augmented Generation (arXiv 2505.10089)
• MIT: Tokenization Changes Meaning in LLMs (Computational Linguistics, 2025)

License

MIT