Tokenizer | npm.io

Takes in a text/file/stream and generates random sentences that sound like they could have been in the text

nlg natural language generation ngram tokenizer

0.5.2 • Published 8 years ago

PostCSS plugin to minify values using bredon-minify

css parser css-parser css-value-parser ast lexer tokenizer validation validator cssinjs

1.0.1 • Published 8 years ago

Converts text into tokens, as defined by provided syntax; Supports recursive sub-syntaxes, regular expressions, and more..

lexer tokenizer

1.5.3 • Published 6 years ago

An interpreted dynamic class-based programming language

shield parser tokenizer type-checking interpreter

0.0.1 • Published 5 years ago

React-based typeahead and typeahead-tokenizer

react typeahead tokenizer autocomplete react-component

1.1.5 • Published 9 years ago

React-based typeahead and typeahead-tokenizer

react typeahead tokenizer autocomplete es6 es2015 react-component

1.1.1 • Published 8 years ago

A lexical analyzer for nodejs.

tokenizer token lexer parser analyzer string

1.0.1 • Published 9 years ago

Simplified token definition and lexer creation library for use with Chevrotain.

lexer lexer-generator parser tokenizer chevrotain

2.0.3 • Published 10 months ago

lexer parser compiler tokens tokenizer

0.0.7 • Published 10 years ago

Fast & simple lexical analyzer

lexer tokenizer lexical analysis token scan

0.0.0 • Published 11 years ago

A Streaming Lexer (Tokenizer)

lexer transform stream tokenizer token

3.0.0 • Published 9 years ago

es6 version of aaditmshah/lexer

lex es6 es2015 lexer lexical analysis scan scanner scanning token

1.1.0 • Published 10 years ago

Time a JavaScript tokenizer

complexion tokenizer tokeniser tokenize tokenise profile

1.1.0 • Published 8 years ago

A class-oriented programming-language

clove parser tokenizer type-checking interpreter

2.0.5 • Published 5 years ago

a simplified and optimized fork of oboejs

front-end micro-library oboe clarinet sax ajax xhr fetch stream streaming

1.0.7 • Published 4 years ago

Distill rich entity data matched via typeahead input into simple visual abstractions.

react hoc tokenized input input tokenizer boundless

1.1.0 • Published 8 years ago

Bredon plugin to transform initial values

css parser css-parser css-value-parser ast lexer tokenizer cssinjs bredon

1.0.0 • Published 8 years ago

Bredon plugin to normalize floating-point precision

css parser css-parser css-value-parser ast lexer tokenizer cssinjs bredon

1.0.0 • Published 8 years ago

tokenizer, lexer

tokenizer lexer

0.0.11 • Published 7 years ago

Word and Sentences Tokenizers for Text-Mining and Natural Language Processing - True ITK - Open Source

tokenizer sentence regex regexp regular expression parser document search detection detect

0.5.0 • Published 7 years ago

18 9 1024 25

Tokenizer Packages

ngram-natural-language-generator

postcss-bredon-minify

retokenizer

shield-lang

react-typeahead-enh

react-typeahead-tokenizer

lexan

lexdoc

lexerific

lexi

lexit

lex-es6

complexion-time-map

clove-lang

bassoon

boundless-tokenized-input

bredon-plugin-initial

bredon-plugin-precision

flukejs

tokenizers.io