Tokenizer Packages

Is a character on a given index a closing of an HTML attribute?

Is given character suitable to be in an HTML attribute's name?

A simple, Twitter-aware tokenizer.

Source code tokenizer powered by vscode

CSSTree is a tool set to work with CSS, including fast detailed parser (string->AST), walker (AST traversal), generator (AST->string) and lexer (validation and matching) based on knowledge of spec and browser implementations

A super fast html-parser stream that outputs tag, text and closing nodes.

Library to tokenize text to paragraphs, sentences, subsentences and words

Tokenizer for Vietnamese in Nodejs and Javascript

Parser aiming at broken or mixed code, especially HTML & CSS

Markdown parser and lexer. A fork of marked.js maintained for Assemble.

Tokenize javascript expressions.

A simple tool to generate bert tokens and input features

A simple NodeRED module to implement bert-tokenizer

An HTML5 lexer

A Bedrock module to provide auto-rotating tokenizers

A parser generator.

A minimalistic, general purpose tokenizer generator

tf.keras tokenizer implemented in nodejs

Tokenizing strings of text. Extracting arrays of words and optionally number, emojis, tags, usernames and email addresses from strings. For Node.js and the browser. When you need more than just [a-z] regular expressions.

This module covers some basic nlp principles and implementations. Every implementation in this module is written as stream to only hold that data in memory that is currently processed at any step.