# tokenwise

> GPT token estimation and context size utilities without a full tokenizer

Latest version **0.3.6** (published 2024-05-14) · MIT license · 0 weekly downloads

> **Deprecated.** This package is deprecated.

## Install

```sh
npm install tokenwise
pnpm add tokenwise
yarn add tokenwise
bun add tokenwise
```

## Health

**Score 10/100 (F)** — status: deprecated.

Negative: deprecated.

## Facts

| | |
|---|---|
| Version | 0.3.6 |
| Published | 2024-05-14 |
| First published | 2023-11-27 |
| Weekly downloads | 0 |
| License | MIT |
| TypeScript types | bundled |
| Module format | ESM + CommonJS |
| Dependencies | 0 |
| Unpacked size | 19.8 KB |
| Known vulnerabilities | 0 |
| Install scripts | no |
| GitHub stars | 177 |
| Author | Johann Schopplich |
| Maintainers | johannschopplich |
| Keywords | ai, gpt, token, tiktoken |

## Links

- npm: https://www.npmjs.com/package/tokenwise
- Repository: https://github.com/johannschopplich/tokenwise
- Homepage: https://github.com/johannschopplich/tokenwise#readme
- Issues: https://github.com/johannschopplich/tokenwise/issues
- npm.io page: https://npm.io/package/tokenwise

## Alternatives

- [@expo/fingerprint](https://npm.io/package/@expo/fingerprint.md) — 6.2M weekly downloads
- [@azure/monitor-opentelemetry-exporter](https://npm.io/package/@azure/monitor-opentelemetry-exporter.md) — 850.0K weekly downloads
- [@azure/monitor-opentelemetry](https://npm.io/package/@azure/monitor-opentelemetry.md) — 624.0K weekly downloads
- [@posthog/ai](https://npm.io/package/@posthog/ai.md) — 423.3K weekly downloads
- [fakefilter](https://npm.io/package/fakefilter.md) — 63.9K weekly downloads

## Recent versions

- 0.3.6 (latest) — 2024-05-14
- 0.3.5 — 2024-04-24
- 0.3.4 — 2024-01-30
- 0.3.3 — 2024-01-30
- 0.3.2 — 2023-11-30
- 0.3.1 — 2023-11-30
- 0.3.0 — 2023-11-29
- 0.2.0 — 2023-11-28
- 0.1.0 — 2023-11-27

## README

# tokenwise

GPT token count and context size utilities when approximations are good enough. For advanced use cases, please use a full tokenizer like [`gpt-tokenizer`](https://github.com/niieani/gpt-tokenizer). This library is intended to be used for quick estimations and to avoid the overhead of a full tokenizer, e.g. when you want to limit your bundle size.

## Benchmarks

The following table shows the accuracy of the token count approximation for different input texts:

<!-- START GENERATED TOKEN COUNT TABLE -->
| Description | Actual GPT Token Count | Estimated Token Count | Token Count Deviation |
| --- | --- | --- | --- |
| Short English text | 10 | 11 | 10.00% |
| German text with umlauts | 56 | 49 | 12.50% |
| Metamorphosis by Franz Kafka (English) | 31891 | 33928 | 6.39% |
| Die Verwandlung by Franz Kafka (German) | 40620 | 34908 | 14.06% |
| 道德經 by Laozi (Chinese) | 14386 | 11919 | 17.15% |
| TypeScript ES5 Type Declarations (~ 4000 loc) | 47890 | 50464 | 5.37% |
<!-- END GENERATED TOKEN COUNT TABLE -->

## Features

- 🌁 Estimate token count without a full tokenizer
- 📐 Supports multiple model context sizes
- 🗣️ Supports accented characters, like German umlauts or French accents
- 🪽 Zero dependencies

## Installation

Run the following command to add `tokenwise` to your project.

```bash
# npm
npm install tokenwise

# pnpm
pnpm add tokenwise

# yarn
yarn add tokenwise
```

## Usage

```ts
import {
  approximateMaxTokenSize,
  approximateTokenSize,
  isWithinTokenLimit
} from 'tokenwise'

const prompt = 'Your prompt goes here.'
const inputText = 'Your text goes here.'

// Estimate the number of tokens in the input text
const estimatedTokens = approximateTokenSize(inputText)
console.log(`Estimated token count: ${estimatedTokens}`)

// Calculate the maximum number of tokens allowed for a given model
const modelName = 'gpt-3.5-turbo'
const maxResponseTokens = 1000
const availableTokens = approximateMaxTokenSize({
  prompt,
  modelName,
  maxTokensInResponse: maxResponseTokens
})
console.log(`Available tokens for model ${modelName}: ${availableTokens}`)

// Check if the input text is within a specific token limit
const tokenLimit = 1024
const withinLimit = isWithinTokenLimit(inputText, tokenLimit)
console.log(`Is within token limit: ${withinLimit}`)
```

## API

### `approximateTokenSize`

Estimates the number of tokens in a given input string based on common English patterns and tokenization heuristics. Work well for other languages too, like German.

**Usage:**

```ts
const estimatedTokens = approximateTokenSize('Hello, world!')
```

**Type Declaration:**

```ts
function approximateTokenSize(input: string): number
```

### `approximateMaxTokenSize`

Calculates the maximum number of tokens that can be included in a response given the prompt length and model's maximum context size.

**Usage:**

```ts
const maxTokens = approximateMaxTokenSize({
  prompt: 'Sample prompt',
  modelName: 'text-davinci-003',
  maxTokensInResponse: 500
})
```

**Type Declaration:**

```ts
function approximateMaxTokenSize({ prompt, modelName, maxTokensInResponse }: {
  prompt: string
  modelName: ModelName
  /** The maximum number of tokens to generate in the reply. 1000 tokens are roughly 750 English words. */
  maxTokensInResponse?: number
}): number
```

### `isWithinTokenLimit`

Checks if the estimated token count of the input is within a specified token limit.

**Usage:**

```ts
const withinLimit = isWithinTokenLimit('Check this text against a limit', 100)
```

**Type Declaration:**

```ts
function isWithinTokenLimit(input: string, tokenLimit: number): boolean
```

## License

[MIT](./LICENSE) License © 2023-PRESENT [Johann Schopplich](https://github.com/johannschopplich)

---
_Source: https://npm.io/package/tokenwise · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._