# tokensize

> The `tokenizer` function uses the `js-tiktoken` library to encode the input string into tokens using the GPT-2 encoding scheme. It then decodes the tokens back into strings, maps the tokens to their positions in the input string using the `mapTokensToChun

Latest version **1.0.0** (published 2023-12-02) · ISC license · 0 weekly downloads

## Install

```sh
npm install tokensize
pnpm add tokensize
yarn add tokensize
bun add tokensize
```

## Health

**Score 30/100 (F)** — status: abandoned.

Positive: has types; esm support; no vulnerabilities; high quality score.

Warnings: low downloads.

Negative: abandoned; low maintenance score.

## Facts

| | |
|---|---|
| Version | 1.0.0 |
| Published | 2023-12-02 |
| First published | 2023-12-02 |
| Weekly downloads | 0 |
| License | ISC |
| TypeScript types | bundled |
| Module format | ESM + CommonJS |
| Dependencies | 1 |
| Unpacked size | 9.5 KB |
| Known vulnerabilities | 0 |
| Install scripts | no |
| Author | Charl Kruger |
| Maintainers | charlkruger |

## Links

- npm: https://www.npmjs.com/package/tokensize
- npm.io page: https://npm.io/package/tokensize

## Dependencies (1)

- [js-tiktoken](https://npm.io/package/js-tiktoken.md) ^1.0.8

## Recent versions

- 1.0.0 (latest) — 2023-12-02

## README

# NPM Module Documentation

The `tokenizer` function takes a string as input and returns an object with the following properties:
- `count`: the number of tokens in the input string
- `characters`: the number of characters in the input string
- `text`: the original input string
- `tokens`: an array of objects, where each object represents a token and its position in the input string. Each token object has the following properties:
    - `token`: the token string
    - `start`: the starting index of the token in the input string
    - `end`: the ending index of the token in the input string

The `tokenizer` function uses the `js-tiktoken` library to encode the input string into tokens using the GPT-2 encoding scheme. It then decodes the tokens back into strings, maps the tokens to their positions in the input string using the `mapTokensToChunks` function, and returns the resulting object.

## Usage

To use this module, you can import the `tokenizer` function and call it with a string argument. Here's an example:

```typescript
import { tokenizer } from 'your-module-name';

const input = 'This is a sample input string.';
const result = await tokenizer(input);

console.log(result);
/*
{
  count: 7,
  characters: 28,
  text: 'This is a sample input string.',
  tokens: [
    { token: 'This', start: 0, end: 3 },
    { token: 'Ġis', start: 5, end: 7 },
    { token: 'Ġa', start: 8, end: 8 },
    { token: 'Ġsample', start: 10, end: 16 },
    { token: 'Ġinput', start: 18, end: 22 },
    { token: 'Ġstring', start: 24, end: 29 },
    { token: '.', start: 29, end: 29 }
  ]
}
*/

---
_Source: https://npm.io/package/tokensize · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._
