tokenx
Fast and lightweight token count estimation without requiring a full tokenizer.
Estimates are calibrated against OpenAI's o200k_base encoding – the tokenizer of all current GPT models. Counts for other LLM families differ somewhat; the defaultCharsPerToken and languageConfigs options let you tune the heuristics for your model. For precise counts, use a full tokenizer like gpt-tokenizer.
Features
- 95%+ average accuracy, and no single sample below 90%
- Just 2kB minified and gzipped, with zero dependencies
- Bundled CLI – count, slice and split from the shell
- Built-in rules for CJK, Cyrillic, Greek, accented Latin scripts and emoji – extend them with your own
Accuracy
Each row of the chart shows a sample text's o200k_base token count, the tokenx estimate, and the deviation between them:
Bars grow left when tokenx underestimates and right when it overestimates. The axis spans the ±10% per-sample deviation bound.
under ◂·▸ over
Team chat transcript (en) 293 → 285 ███│ -2.73%
Vite releases API response 8,075 → 8,551 │██████ +5.89%
tokenx source code 3,496 → 3,341 ████│ -4.43%
Vite plugin API docs (en) 6,901 → 7,155 │████ +3.68%
Cat article (ja) 12,437 → 11,914 ████│ -4.21%
Cat article (ko) 7,117 → 6,815 ████│ -4.24%
Cat article (zh) 9,057 → 8,953 █│ -1.15%
The Great Gatsby by Fitzgerald (en) 4,391 → 4,479 │██ +2.00%
Die Verwandlung by Kafka (de) 4,437 → 4,384 █│ -1.19%
─────────────────────
mean 3.28%
Accuracy depends on the kind of text: a holdout corpus of French, English, and Russian documents the heuristics were never calibrated on stays within ±15% per document.
These cases are knowingly left outside the bounds:
- High-entropy strings – base64, hashes, digests: ≈-70%.
- Traditional and classical Chinese – the hanzi rate is calibrated on contemporary simplified script: ≈-5% to -20%.
- Scripts without a built-in rule – Arabic ≈-10% to -30%, Hindi ≈-30%, Hebrew ≈-45%, Thai ≈-60%; a custom language config closes the gap.
- Chinese or Japanese mixed with Latin words or code – Latin letters that share a segment with CJK text are priced like hanzi: ≈+30% to +80% for technical prose, up to ≈+200% for a sentence built around code identifiers.
Installation
# npm
npm install tokenx
# pnpm
pnpm add tokenx
# yarn
yarn add tokenx
On Node.js, tokenx requires version 24 or later.
Quick Start
import { estimateTokenCount, isWithinTokenLimit, sliceByTokens, splitByTokens } from 'tokenx'
const text = 'Your text goes here.'
estimateTokenCount(text) // Estimated token count
isWithinTokenLimit(text, 1024) // true
sliceByTokens(text, 0, 5) // Roughly the first five tokens
splitByTokens(text, 100) // Chunks of about 100 tokens each
CLI
The package ships a tokenx binary. Run it with npx tokenx, or install the package to call tokenx directly.
# Count tokens in a file, a pipe, or several files at once
tokenx README.md
cat article.md | tokenx
tokenx count src/*.ts
# Fail a script when a prompt outgrows its budget (exit code 2)
tokenx prompt.txt --limit 8000
# Extract a token range, or chunk a document for RAG
tokenx slice article.md --end 500
tokenx split article.md --size 500 --overlap 50
Only results go to stdout – a bare integer for one input, a table for several, chunks as a JSON array – so $(tokenx count file.txt) and | jq work unchanged. Notices and errors go to stderr. Run tokenx --help for every option.
API
Options
Every function takes an optional options object last:
defaultCharsPerToken– average characters per token for segments no other rule prices (default: 7)languageConfigs–{ pattern, averageCharsPerToken }entries checked before the built-in language configs; the first matching config prices a segment. They take precedence over every built-in rule except whitespace, the CJK script rates included, and the built-in configs still price whatever no custom config matches.
estimateTokenCount(text?, options?)
Returns the estimated token count of text.
estimateTokenCount('Hello, world!')
estimateTokenCount('Bonjour, ça va très bien ?', {
defaultCharsPerToken: 4,
languageConfigs: [
{ pattern: /[éèêëàâîï]/i, averageCharsPerToken: 3 }
],
})
isWithinTokenLimit(text, tokenLimit, options?)
Returns whether the estimated token count stays within tokenLimit, the limit included. Stops counting as soon as the limit is exceeded.
isWithinTokenLimit('Check this text against a limit', 100) // true
sliceByTokens(text, start?, end?, options?)
Extracts a token range like Array.prototype.slice(): start is inclusive, end exclusive, and negative indices count back from the end. A slice can start or end mid-word; adjacent slices still rejoin into the original text.
const text = 'Hello, world! This is a test sentence.'
sliceByTokens(text, 0, 3) // First three tokens
sliceByTokens(text, -2) // Last two tokens
sliceByTokens(text, 1, -1) // All but the first and last token
splitByTokens(text, tokensPerChunk, options?)
Splits text into chunks of about tokensPerChunk tokens – for RAG, batch processing, or staying within a context window.
tokensPerChunk is a target, not a hard maximum: a chunk closes once it reaches the target, so a single long segment can push a chunk beyond it. Chinese and Japanese text has no spaces, so a whole clause up to its punctuation mark forms one segment, and a long clause can push a chunk well past the target.
The overlap option (default: 0, clamped below tokensPerChunk) repeats about overlap tokens from the end of each chunk at the start of the next. The repeat consists of whole segments, so it runs longer when a segment crosses overlap and shorter when one more segment would bring it up to tokensPerChunk. A final chunk consisting only of overlap content is never emitted.
splitByTokens(text, 100)
splitByTokens(text, 100, { overlap: 10 })
License
MIT License 2023-PRESENT Johann Schopplich