Tokenizer Packages

Tiny JavaScript tokenizer.

A tool set for CSS: fast detailed parser (CSS → AST), walker (AST traversal), generator (AST → CSS) and lexer (validation and matching) based on specs and browser implementations

Tokenizes a string that represents a regular expression.

A tokenzier for Sass' SCSS syntax

small commonmark compliant markdown parser with positional info and concrete tokens

Simple HTML Tokenizer is a lightweight JavaScript library that can be used to tokenize the kind of HTML normally found in templates.

A promise based streaming tokenizer

TypeScript definition for strtok3 token

r/w stream of glsl tokens

safely inject a block of tokens into a shader

Trim the whitespace within an array of GLSL tokens

General natural language (tokenizing, stemming (English, Russian, Spanish), part-of-speech tagging, sentiment analysis, classification, inflection, phonetics, tfidf, WordNet, jaro-winkler, Levenshtein distance, Dice's Coefficient) facilities for node.

Parse parentheses from a string

React typeahead with Bootstrap styling

lexical analyzer generator used by jison

A parser generator with Bison's API

stream-json is the micro-library of Node.js stream components for creating custom JSON processing pipelines with a minimal memory footprint. It can parse JSON files far exceeding available memory streaming individual primitives using a SAX-inspired API. I

Fork: HTML parsing/serialization toolset for Node.js. WHATWG HTML Living Standard (aka HTML5)-compliant. Fork from the HTMLParseErrorWG branch.

Parse PHP code from JS and returns its AST

Toolkit to make streamable scanners and parsers.