Tokenizer | npm.io

Evaluate javascript expressions. No eval. No dependencies. Only 6kb.

parser javascript js expressions eval evaluate evaluator tokenizer

0.3.1 • Published 5 years ago

Enhances comprehension for MindScript-powered models by tokenizing text.

tokenizer mindscript ai

0.0.1 • Published 2 years ago

Promise based tokenizer for a ReadableStream

ReadableStream token tokenizer parser decoder bytes

1.3.3 • Published 1 year ago

Have you ever been in need of a tag box that can provide many built in functionalities that suites you needs? TagBox is built with Angular 4+ code and is developed specifically without needing to use ngModel to avoid an existing bug in Angular which makes

angular 6+tags tagger tagging tokenize tokenizer widget autocomplete tagger image tagger audio tagger

3.0.0 • Published 2 years ago

Split text into sentence strings or word arrays with Sentence Boundary Detection (SBD). Based on http://github.com/Tessmore/sbd by Fabiën Tesselaar.

sentence detection boundary lexer tokenizer

2.0.2 • Published 9 years ago

tokenizer for ecmascript

tokenizer compiler ecmascript tool

0.0.1 • Published 8 years ago

Lexer

node javascript typescript lexer scanner token tokenizer compiler parser

1.0.7 • Published 6 years ago

Forked version of kuromoji with better compatibility for browsers