Tokenizer Packages

a lexer in javascript

Not a replacement for REGEX, but an alternative that is far more readable and a bit more flexible

jieba-js chinese word segmentation for node in pure javascript.

Just Another LEXer is a lexical analyzer compatible with Jison.

general purpose regex tokenizer that returns an array of tokens

Simple string tokenizer that returns both tokens and delimiters and preserves their order. Features:

React typeahead with Bootstrap styling

General natural language (tokenizing, stemming (English, Russian, Spanish), part-of-speech tagging, sentiment analysis, classification, inflection, phonetics, tfidf, WordNet, jaro-winkler, Levenshtein distance, Dice's Coefficient) facilities for node.

Logical expression parser and evaluates result, suitable for permissions management

personal tokenize util

tokenize a string that includes ansi code

Replacement for sql-tokenizer which supports Mysql syntax. Also provides an experimental utility for structuring tokens into SQL statements.

js/ts/cs razor-like generator implementation, dedicated to code generation

Esperanto tokenizer

Parse PHP code and returns its AST (with comments indent support)

PostCSS plugin to compile values using Bredon

PostCSS plugin to convert css colours into variables for SCSS

finds a #version field in a set of GLSL tokens

html5 parser for node.js and browsers

Split text into tokens.