npm.io
2.2.1 • Published 4d agoCLI

tokenx

Licence
MIT
Version
2.2.1
Deps
0
Size
52 kB
Vulns
0
Weekly
0
Stars
188

tokenx

Fast and lightweight token count estimation without requiring a full tokenizer.

Estimates are calibrated against OpenAI's o200k_base encoding – the tokenizer of all current GPT models. Counts for other LLM families differ somewhat; the defaultCharsPerToken and languageConfigs options let you tune the heuristics for your model. For precise counts, use a full tokenizer like gpt-tokenizer.

Features

  • 95%+ average accuracy, and no single sample below 90%
  • Just 2kB minified and gzipped, with zero dependencies
  • Bundled CLI – count, slice and split from the shell
  • Built-in rules for CJK, Cyrillic, Greek, accented Latin scripts and emoji – extend them with your own

Accuracy

Each row of the chart shows a sample text's o200k_base token count, the tokenx estimate, and the deviation between them:

Bars grow left when tokenx underestimates and right when it overestimates. The axis spans the ±10% per-sample deviation bound.

                                                          under ◂·▸ over
Team chat transcript (en)               293 →    285          ███│              -2.73%
Vite releases API response            8,075 →  8,551             │██████        +5.89%
tokenx source code                    3,496 →  3,341         ████│              -4.43%
Vite plugin API docs (en)             6,901 →  7,155             │████          +3.68%
Cat article (ja)                     12,437 → 11,914         ████│              -4.21%
Cat article (ko)                      7,117 →  6,815         ████│              -4.24%
Cat article (zh)                      9,057 →  8,953            █│              -1.15%
The Great Gatsby by Fitzgerald (en)   4,391 →  4,479             │██            +2.00%
Die Verwandlung by Kafka (de)         4,437 →  4,384            █│              -1.19%
                                                       ─────────────────────
                                                                        mean     3.28%

Accuracy depends on the kind of text: a holdout corpus of French, English, and Russian documents the heuristics were never calibrated on stays within ±15% per document.

These cases are knowingly left outside the bounds:

  • High-entropy strings – base64, hashes, digests: ≈-70%.
  • Traditional and classical Chinese – the hanzi rate is calibrated on contemporary simplified script: ≈-5% to -20%.
  • Scripts without a built-in rule – Arabic ≈-10% to -30%, Hindi ≈-30%, Hebrew ≈-45%, Thai ≈-60%; a custom language config closes the gap.
  • Chinese or Japanese mixed with Latin words or code – Latin letters that share a segment with CJK text are priced like hanzi: ≈+30% to +80% for technical prose, up to ≈+200% for a sentence built around code identifiers.

Installation

# npm
npm install tokenx

# pnpm
pnpm add tokenx

# yarn
yarn add tokenx

On Node.js, tokenx requires version 24 or later.

Quick Start

import { estimateTokenCount, isWithinTokenLimit, sliceByTokens, splitByTokens } from 'tokenx'

const text = 'Your text goes here.'

estimateTokenCount(text) // Estimated token count
isWithinTokenLimit(text, 1024) // true
sliceByTokens(text, 0, 5) // Roughly the first five tokens
splitByTokens(text, 100) // Chunks of about 100 tokens each

CLI

The package ships a tokenx binary. Run it with npx tokenx, or install the package to call tokenx directly.

# Count tokens in a file, a pipe, or several files at once
tokenx README.md
cat article.md | tokenx
tokenx count src/*.ts

# Fail a script when a prompt outgrows its budget (exit code 2)
tokenx prompt.txt --limit 8000

# Extract a token range, or chunk a document for RAG
tokenx slice article.md --end 500
tokenx split article.md --size 500 --overlap 50

Only results go to stdout – a bare integer for one input, a table for several, chunks as a JSON array – so $(tokenx count file.txt) and | jq work unchanged. Notices and errors go to stderr. Run tokenx --help for every option.

API

Options

Every function takes an optional options object last:

  • defaultCharsPerToken – average characters per token for segments no other rule prices (default: 7)
  • languageConfigs – { pattern, averageCharsPerToken } entries checked before the built-in language configs; the first matching config prices a segment. They take precedence over every built-in rule except whitespace, the CJK script rates included, and the built-in configs still price whatever no custom config matches.
estimateTokenCount(text?, options?)

Returns the estimated token count of text.

estimateTokenCount('Hello, world!')

estimateTokenCount('Bonjour, ça va très bien ?', {
  defaultCharsPerToken: 4,
  languageConfigs: [
    { pattern: /[éèêëàâîï]/i, averageCharsPerToken: 3 }
  ],
})
isWithinTokenLimit(text, tokenLimit, options?)

Returns whether the estimated token count stays within tokenLimit, the limit included. Stops counting as soon as the limit is exceeded.

isWithinTokenLimit('Check this text against a limit', 100) // true
sliceByTokens(text, start?, end?, options?)

Extracts a token range like Array.prototype.slice(): start is inclusive, end exclusive, and negative indices count back from the end. A slice can start or end mid-word; adjacent slices still rejoin into the original text.

const text = 'Hello, world! This is a test sentence.'

sliceByTokens(text, 0, 3) // First three tokens
sliceByTokens(text, -2) // Last two tokens
sliceByTokens(text, 1, -1) // All but the first and last token
splitByTokens(text, tokensPerChunk, options?)

Splits text into chunks of about tokensPerChunk tokens – for RAG, batch processing, or staying within a context window.

tokensPerChunk is a target, not a hard maximum: a chunk closes once it reaches the target, so a single long segment can push a chunk beyond it. Chinese and Japanese text has no spaces, so a whole clause up to its punctuation mark forms one segment, and a long clause can push a chunk well past the target.

The overlap option (default: 0, clamped below tokensPerChunk) repeats about overlap tokens from the end of each chunk at the start of the next. The repeat consists of whole segments, so it runs longer when a segment crosses overlap and shorter when one more segment would bring it up to tokensPerChunk. A final chunk consisting only of overlap content is never emitted.

splitByTokens(text, 100)
splitByTokens(text, 100, { overlap: 10 })

License

MIT License 2023-PRESENT Johann Schopplich

Keywords