tokenizer-1dv610 v1.1.3

tokenizer

A tokenizer for the course 1dv610 visit the github repo for more info if this page is not up to date.

How to use

npm i tokenizer-1dv610

initilize a new object of the class 'Tokenizer' and pass in 2 arguments, the first one is the string you want to tokenize, the second on is an instance of the Grammar class that is shown below

To get the current token use: tokenizer.getCurrentToken()

To step to the next token use: tokenizer.next()

To step to the previous token use: tokenizer.previous()

To change the string use: tokenizer.setString('new string')

Example:

if you try out with the above code:

output: "Hello"

output: "is"