Tokenization Packages

Salient is a natural language processing and sentiment analysis library

Salient is a natural language processing and sentiment analysis library

Tools for manipulating sentences

POS tokenization of words int meaningful components usable in POS-Bayes & Elastic Search Indexes

Define a function to execute after completing token-defined tasks.

a lexer in javascript

DropSuit NLP - tok function: A JavaScript and Node.js library for tokenizing text

Taiwanese Hokkien Transliterator and Tokeniser

Well typed, string transmutation via splicing, driven by definitions of regular expressions, priority match and recursive clauses

Textmate token-based language service for Visual Studio Code.

Jawi

Functions for filtering SQL-like json with conjunctive normal form filters

A high-performance wrapper around `Intl.Segmenter` for efficient text segmentation. This class resolves memory handling issues seen with large strings and can enhance performance by 50-500x. Only ~70 loc (with comments) and no dependencies.

Use LLMs to run map-reduce summarization tasks on large documents until a target token size is met.

a wrapper around the LunaSec CLI enabling it to be used as an NPM package

High-performance library for tokenizing text.

A reverse engineered Node.js client for the Blackbox.ai API, supporting chat completions with streaming and aggregated responses.

Adebiet

Taiwanese Hokkien Transliterator and Tokeniser