Tokenizer Packages

Isomorphic utilities for GPT-3 tokenization and prompt building.

A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models

GPT Turbo plugin that calculates your conversation size and cost

Lightweight trimmed down encoder/decoder/tokenizer/token counter for gpt3 that is compatible with both node and browser environments

TypeScript library designed to create typesafe, composable parsers and tokenizers for building embedded languages. With powerful parsing combinators and a focus on type safety

A simple typescript tokenizer

JS tokenizer for LLaMA-based LLMs

JS tokenizer for LLaMA 3

A lidera japanese tokenizer wrapper for javascript and typescript

lexer for recursive descent parsers

It is a lexer and parser of C built using typescript

A simple yet flexible lexer (or tokenizer).

A cascade-based JS lexer implementation

Lexer / tokenizer

CLI tool to tokenize codebases for LLM usage

A lightweight tokenizer for OpenAI's GPT model series. Uses OpenAI's tiktoken python package

a wrapper around the LunaSec CLI enabling it to be used as an NPM package

Math expression tokenizer

Light-weight sentence tokenizer for Korean. Supports both full-width and half-width punctuation marks.

JavaScript implementation of Japanese morphological analyzer alternate version