@kajberg/tokenizer v1.0.2

:memo: Kaj Berg - Tokenizer

User Manual

1. Import startTokenizer from @kajberg/tokenizer.
2. First parameter: Collection of grammar.
3. Second parameter: Document to create tokens from(Type String).
4. startTokenizer(x,y) Returns an Instans of a tokenizer.
5. Use methods below for desired behavior.

How to use it

import { startTokenizer } from '@kajberg/tokenizer.'

const grammar = [
      {
      tokenType:"WORD",
      tokenRegex:/^[\w|åäöÅÄÖ]+/
      },  
      {
      tokenType:"DOT",
      tokenRegex:/^\./
      }
    ]

const tokenizer = startTokenizer(grammar, "Hello World.")
// Returns active Token.
tokenizer.getActiveToken();
// Go to next token and get active token.
tokenizer.nextToken();
tokenizer.getActiveToken();
// Get all created tokens.
tokenizer.allCreatedTokens();

Methods for Tokenizer

getActiveToken()
Returns active token.

nextToken()
Switch to the next token.

previousToken()
Switch to the previous token.

allCreatedTokens()
Returns an array with all created tokens.