Tokenizer Packages

JavaScript implementation of Japanese morphological analyzer

Forked version of kuromoji with better compatibility for browsers

Contains the lezer parser for the GDScript language.

Simple tokenizer for a simple "language"

TypeScript version of PGN Tokenizer, a Byte Pair Encoding (BPE) tokenizer for Chess Portable Game Notiation (PGN).

A simple lexer for TypeScript

A tool set for CSS: fast detailed parser (CSS → AST), walker (AST traversal), generator (AST → CSS) and lexer (validation and matching) based on specs and browser implementations

JavaScript implementation of Japanese morphological analyzer

minimal es6 natural language detection

A collection of tools is designed to aid in working with the Rumantsch (Romansh) language

Node.js binding for huggingface/tokenizers library

finite state machine tokenizer

Tokenizer for Forms Markup Language (FML)

Converts the given strings to universal tokens.

An abstract tokenizer.

Easily assert that values are NonNullable or Nullable.

Regex Based Tokenizer used in parsers developed by Porifa