# @mlc-ai/web-tokenizers

> | [NPM Package](https://www.npmjs.com/package/@mlc-ai/web-tokenizers) | [WebLLM](https://github.com/mlc-ai/web-llm) |

Latest version **0.1.6** (published 2025-05-03) · Apache-2.0 license · 0 weekly downloads

## Install

```sh
npm install @mlc-ai/web-tokenizers
pnpm add @mlc-ai/web-tokenizers
yarn add @mlc-ai/web-tokenizers
bun add @mlc-ai/web-tokenizers
```

## Health

**Score 45/100 (D)** — status: stable.

Positive: has types; esm support; no vulnerabilities.

Warnings: low downloads; pre 1.0.

Negative: stale.

## Facts

| | |
|---|---|
| Version | 0.1.6 |
| Published | 2025-05-03 |
| First published | 2023-05-22 |
| Weekly downloads | 0 |
| License | Apache-2.0 |
| TypeScript types | bundled |
| Module format | ESM + CommonJS |
| Dependencies | 0 |
| Unpacked size | 9.3 MB |
| Known vulnerabilities | 0 |
| Install scripts | no |
| GitHub stars | 514 |
| Maintainers | tqchen, charliefruan, neetnestor, rickzx, ruihanglai, sudeepag |
| Keywords | machine_learning, llm, nlp |

## Links

- npm: https://www.npmjs.com/package/@mlc-ai/web-tokenizers
- Repository: https://github.com/mlc-ai/tokenizers-cpp
- Homepage: https://github.com/mlc-ai/tokenizers-cpp/tree/main/web
- Issues: https://github.com/mlc-ai/tokenizers-cpp/issues
- npm.io page: https://npm.io/package/@mlc-ai/web-tokenizers

## Recent versions

- 0.1.6 (latest) — 2025-05-03
- 0.1.5 — 2024-08-12
- 0.1.4 — 2024-06-03
- 0.1.3 — 2024-03-30
- 0.1.2 — 2023-11-18
- 0.1.0 — 2023-05-22

## README

# WebTokenizers

| [NPM Package](https://www.npmjs.com/package/@mlc-ai/web-tokenizers) | [WebLLM](https://github.com/mlc-ai/web-llm) |

WebTokenizers is a javascript binding library that can be universally deployed.
It wraps and binds the [HuggingFace tokenizers library](https://github.com/huggingface/tokenizers)
and [sentencepiece](https://github.com/google/sentencepiece) and provides a minimum common interface
in havascript.

## Build from Srouce

```bash
source /path/to/emsdk_env.sh
npm install
npm run build
```

To try out the test webpage
```bash
cd tests
npm start
```

---
_Source: https://npm.io/package/@mlc-ai/web-tokenizers · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._
