Tokenize Packages

A complex string based CSS managment library

split string with quote

Splits text into tokens in as many languages as possible, using libicu

Oh No, Another Markup Language. Based on YAML, but stores considerably more object metadata.

A small library for building toy lexers.

Tokenizer for tokenizing sentences, for BERT or other NLP preprocessing.

Create a snapdragon token. Used by the snapdragon lexer, but can also be used by plugins.

Convert GLSL from WebGL to OpenGL

simple text to word tokenizer. accepts a custom regex. returns an array of tokens

Lexical analyzer built in Javascript

[nodejs] Tokenize data.

Simple utility to get unique / repeated words from strings or phrases

String Tokenizer for Node.js using ICU's BreakIterators

Easily scan a string with an object of regex patterns to produce an array of tokens. ~100 sloc.

transpiles GLSL tokens from v100 to v300 es

Tokenize a graphql schema

a very simple lexer

tokenize c/c++ source code

Generate the Abstract Syntax Tree (AST) of a Bash command.

Stringscanner performs lexical scanning operations on a string.