# js-tokens

> Tiny JavaScript tokenizer.

Latest version **10.0.0** (published 2025-12-08) · MIT license · 0 weekly downloads

## Install

```sh
npm install js-tokens
pnpm add js-tokens
yarn add js-tokens
bun add js-tokens
```

## Health

**Score 55/100 (C)** — status: stable.

Positive: has types; esm support; no vulnerabilities.

Warnings: low downloads.

## Facts

| | |
|---|---|
| Version | 10.0.0 |
| Published | 2025-12-08 |
| First published | 2014-03-08 |
| Weekly downloads | 0 |
| License | MIT |
| TypeScript types | bundled |
| Module format | ESM |
| Dependencies | 0 |
| Unpacked size | 15.6 KB |
| Known vulnerabilities | 0 |
| Install scripts | no |
| GitHub stars | 556 |
| Author | Simon Lydell |
| Maintainers | lydell |
| Keywords | JavaScript, js, ECMAScript, es, token, tokens, tokenize, tokenizer, regex, regexp |

## Links

- npm: https://www.npmjs.com/package/js-tokens
- Repository: https://github.com/lydell/js-tokens
- Homepage: https://github.com/lydell/js-tokens#readme
- Issues: https://github.com/lydell/js-tokens/issues
- npm.io page: https://npm.io/package/js-tokens

## Alternatives

- [@mapbox/jsonlint-lines-primitives](https://npm.io/package/@mapbox/jsonlint-lines-primitives.md) — 5.3M weekly downloads
- [reftools](https://npm.io/package/reftools.md) — 3.5M weekly downloads
- [@hey-api/openapi-ts](https://npm.io/package/@hey-api/openapi-ts.md) — 3.5M weekly downloads
- [@mapbox/geojson-rewind](https://npm.io/package/@mapbox/geojson-rewind.md) — 2.4M weekly downloads
- [turbo-stream](https://npm.io/package/turbo-stream.md) — 1.7M weekly downloads

## Recent versions

- 10.0.0 (latest) — 2025-12-08
- 9.0.1 — 2024-11-22
- 9.0.0 — 2024-02-08
- 8.0.3 — 2024-02-03
- 8.0.2 — 2023-09-09
- 8.0.1 — 2023-01-16
- 8.0.0 — 2022-05-15
- 7.0.0 — 2021-04-25
- 6.0.0 — 2020-04-13
- 5.0.0 — 2019-06-15
- 4.0.0 — 2018-01-28
- 3.0.2 — 2017-06-28
- 3.0.1 — 2017-01-30
- 3.0.0 — 2017-01-11
- 2.0.0 — 2016-06-19
- … 10 more at https://npm.io/package/js-tokens/versions

## README

# js-tokens

The tiny, regex powered, lenient, _almost_ spec-compliant JavaScript tokenizer that never fails.

```js
import jsTokens from "js-tokens";

const jsString = 'JSON.stringify({k:3.14**2}, null /*replacer*/, "\\t")';

Array.from(jsTokens(jsString), (token) => token.value).join("|");
// JSON|.|stringify|(|{|k|:|3.14|**|2|}|,| |null| |/*replacer*/|,| |"\t"|)
```

**[➡️ Full readme](https://github.com/lydell/js-tokens/)**

---
_Source: https://npm.io/package/js-tokens · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._