Tokenization Packages

A Bedrock module to provide auto-rotating tokenizers

Basis Theory utility for decrypting Google Pay Tokens

Basis Theory utility for decrypting Apple Pay Tokens

A powerful and flexible text search library for JavaScript that enables you to build a simple text search engine.

RWA Token SDK for the development of permissioned tokens on SVM blockchains.

a wrapper around the LunaSec CLI enabling it to be used as an NPM package

NLP (natural language processing) for server and the browser in TypeScript. All lightweight and super-fast.

Here is a README generated from the code snippet:

This template provides a minimal setup to get React working in Vite with HMR and some ESLint rules.

CLI tool to perform NLP on selected files

OpenVINO™ Tokenizers adds text processing operations to openvino-node package

Library that allows you to make payments and tokenize debit and credit cards with Niubiz.

Very fast Porter Stemmer implementation written in Javascript.

An interface to the pruf network written in javascript

SmartCal is a lightweight and powerful library for evaluating complex mathematical expressions dynamically. It supports tokenization, parsing, and interpreting mathematical formulas seamlessly, offering flexibility to integrate custom data objects for dyn

## How To Use

Isomorphic utilities for GPT-3 tokenization and prompt building.

A high-performance wrapper around `Intl.Segmenter` for efficient text segmentation. This class resolves memory handling issues seen with large strings and can enhance performance by 50-500x. Only ~70 loc (with comments) and no dependencies.

Use LLMs to run map-reduce summarization tasks on large documents until a target token size is met.