Tokenize Packages

Lemmize and tokenize string which contains Chinese and English words

hast utility to parse from HTML

A high-performance wrapper around `Intl.Segmenter` for efficient text segmentation. This class resolves memory handling issues seen with large strings and can enhance performance by 50-500x. Only ~70 loc (with comments) and no dependencies.

convert a string to list of words

html tokenizer

Parsing and tokenizing attributes string

Tokenize a Concatenation Expression

get word counts / frequencies on a per-speaker or per-category basis, or as an aggregate

Lexer to tokenize CSS

Parse and manipulate tokenized strings with ease!

Parse and manipulate tokenized strings with ease!

Tokenize strings with double quotes, with embedded escaped, backslashed double quotes

An easy and simple way to encrypt and decrypt information for client and server side components

estree (and esast) utility to parse from JavaScript

General purpose generic matcher

Generate string from a token array by interpolating values.

Tokenize a string into an array of string parts and format identifier objects.