1.0.2 • Published 2 years ago
@tsoding/tokenizer.js v1.0.2
⬇️ Installation
To install the latest version of the library, simply run npm install @tsoding/tokenizer.js
, or yarn add @tsoding/tokenizer.js
, depending on your package manager.
📄 Usage
Using tokenizer.js
is very easy. Here's an example of how to use it:
import tokenize from "@tsoding/tokenizer.js";
const str = "const test = 'hello';";
const tokens = tokenize(str);
The above code will return an array of tokens.
📝 Expansion
I plan to expand this library to do more things other than tokenizing variables. If you're willing to help, please contribute! Any help and/or feedback is welcome!