@iceylan/tokenizer v0.0.1

Tokenizer

This javascript library converts the given strings to universal tokens. With the help of this tokens, language specific parsers can be created. It can tokenize every kind of text.

Installation

npm install @iceylan/tokenizer

Usage

import tokenizer from "@iceylan/tokenizer";

const tokens = tokenizer( "hello world!" )