1.0.4 • Published 7 years ago
Text Tokenizer 
Split text into tokens
Installation
npm:
npm install @streebs/text-tokenizer`
Usage
const textTokenize = require('@streebs/text-tokenize');
let tokens = textTokenize('Hello, World!');
This would make
tokens = ["Hello", ",", " ", "World", "!"]
License
MIT © Austin Kostreba