Tokenizing | npm.io

A little helper for handling strings containing zero width control characters, ANSI styling, whitespaces, newlines, 💩, etc.

string width string length real string width real string length optical string width printed width unicode codepoints code points strlen

1.0.42 • Published 7 years ago

A simple, Twitter-aware tokenizer.

tokenise tokenize tokenising tokenizing tokeniser tokenizer token twitter NLP language

7.1.0 • Published 1 year ago

Tools for manipulating sentences

sentence parsing tokenization tokenizing

4.0.0 • Published 6 years ago

A little helper for handling strings containing zero width control characters, ANSI styling, whitespaces, newlines, etc.

string width string length real string width real string length optical string width printed width unicode codepoints code points strlen

1.0.36 • Published 8 years ago

convert text stream to tokens

tokenizing segmentation

1.5.0 • Published 10 years ago

A JavaScript library for simplification the process of organization/creation of parsers/validators/code-generators of arbitrary complexity.

parsing tokenizing validation generation maps

0.2.1 • Published 1 year ago