# gpt3-tokenizer

> [![Build](https://github.com/botisan-ai/gpt3-tokenizer/actions/workflows/main.yml/badge.svg)](https://github.com/botisan-ai/gpt3-tokenizer/actions/workflows/main.yml) [![NPM Version](https://img.shields.io/npm/v/gpt3-tokenizer.svg)](https://www.npmjs.com/

Latest version **1.1.5** (published 2023-01-27) · MIT license · 0 weekly downloads

## Install

```sh
npm install gpt3-tokenizer
pnpm add gpt3-tokenizer
yarn add gpt3-tokenizer
bun add gpt3-tokenizer
```

## Health

**Score 30/100 (F)** — status: abandoned.

Positive: has types; esm support; no vulnerabilities; high quality score.

Warnings: low downloads; large bundle.

Negative: abandoned; low maintenance score.

## Facts

| | |
|---|---|
| Version | 1.1.5 |
| Published | 2023-01-27 |
| First published | 2021-12-27 |
| Weekly downloads | 0 |
| License | MIT |
| TypeScript types | bundled |
| Module format | ESM + CommonJS |
| Node | >=12 |
| Dependencies | 1 |
| Unpacked size | 11.3 MB |
| Known vulnerabilities | 0 |
| Install scripts | no |
| Author | Simon Liang |
| Maintainers | lhr0909 |

## Links

- npm: https://www.npmjs.com/package/gpt3-tokenizer
- Repository: https://github.com/xanthous-tech/gpt3-tokenizer
- npm.io page: https://npm.io/package/gpt3-tokenizer

## Dependencies (1)

- [array-keyed-map](https://npm.io/package/array-keyed-map.md) ^2.1.3

## Recent versions

- 1.1.5 (latest) — 2023-01-27
- 1.1.4 — 2022-11-23
- 1.1.3 — 2022-08-27
- 1.1.2 — 2022-02-15
- 1.1.1 — 2022-02-15
- 1.1.0 — 2022-02-15
- 1.0.2 — 2021-12-27
- 1.0.1 — 2021-12-27
- 1.0.0 — 2021-12-27

## README

# GPT3 Tokenizer

[![Build](https://github.com/botisan-ai/gpt3-tokenizer/actions/workflows/main.yml/badge.svg)](https://github.com/botisan-ai/gpt3-tokenizer/actions/workflows/main.yml)
[![NPM Version](https://img.shields.io/npm/v/gpt3-tokenizer.svg)](https://www.npmjs.com/package/gpt3-tokenizer)
[![NPM Downloads](https://img.shields.io/npm/dt/gpt3-tokenizer.svg)](https://www.npmjs.com/package/gpt3-tokenizer)

This is a isomorphic TypeScript tokenizer for OpenAI's GPT-3 model. Including support for `gpt3` and `codex` tokenization. It should work in both NodeJS and Browser environments.
## Usage

First, install:

```shell
yarn add gpt3-tokenizer
```

In code:

```typescript
import GPT3Tokenizer from 'gpt3-tokenizer';

const tokenizer = new GPT3Tokenizer({ type: 'gpt3' }); // or 'codex'
const str = "hello 👋 world 🌍";
const encoded: { bpe: number[]; text: string[] } = tokenizer.encode(str);
const decoded = tokenizer.decode(encoded.bpe);
```

## Reference

This library is based on the following:
- [OpenAI Tokenizer Page Source](https://beta.openai.com/tokenizer?view=bpe)
- [gpt-3-encoder](https://github.com/latitudegames/GPT-3-Encoder)

The main difference between this library and gpt-3-encoder is that this library supports both `gpt3` and `codex` tokenization (The dictionary is taken directly from OpenAI so the tokenization result is on par with the OpenAI Playground). Also Map API is used instead of JavaScript objects, especially the `bpeRanks` object, which should see some performance improvement.

## License

[MIT](./LICENSE)

---
_Source: https://npm.io/package/gpt3-tokenizer · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._