@scrapiloo/regex-tokenizer v0.1.2

Regex Tokenizer

Minimal Regex Tokenizer.

Installation

# NPM
npm i @scrapiloo/regex-tokenizer

# Yarn
yarn add @scrapiloo/regex-tokenizer

Usage

To use the tokenizer, call the tokenize() function with a regular expression as input.

import tokenize from "@scrapiloo/regex-tokenizer";

const tokens = tokenize("(.+)@(?!gmail.com)");
console.log(tokens);

The code above will return the following output:

[
  Token [”(”] {
    type: 'capture-group-perimeter', 
    value: '(', 
    scope: 'root capture-group'
  },
  Token [”.”] {
    type: 'character',
    value: '.',
    scope: 'root capture-group'
  },
  Token [”+”] {
    type: 'quantifier',
    value: '+',
    scope: 'root capture-group'
  },
  Token [”)”] {
    type: 'capture-group-perimeter',
    value: ')',
    scope: 'root capture-group'
  },
  Token [”@”] { 
    type: 'character', 
    value: '@', 
    scope: 'root' 
  },
  Token [”(”] {
    type: 'lookaround-perimeter',
    value: '(',
    scope: 'root negative-lookaround'
  },
  Token [”?!”] {
    type: 'lookaround-type',
    value: '?!',
    scope: 'root negative-lookaround'
  },
  Token [”g”] {
    type: 'character',
    value: 'g',
    scope: 'root negative-lookaround'
  },
  Token [”m”] {
    type: 'character',
    value: 'm',
    scope: 'root negative-lookaround'
  },
  Token [”a”] {
    type: 'character',
    value: 'a',
    scope: 'root negative-lookaround'
  },
  Token [”i”] {
    type: 'character',
    value: 'i',
    scope: 'root negative-lookaround'
  },
  Token [”l”] {
    type: 'character',
    value: 'l',
    scope: 'root negative-lookaround'
  },
  Token [”.”] {
    type: 'character',
    value: '.',
    scope: 'root negative-lookaround'
  },
  Token [”c”] {
    type: 'character',
    value: 'c',
    scope: 'root negative-lookaround'
  },
  Token [”o”] {
    type: 'character',
    value: 'o',
    scope: 'root negative-lookaround'
  },
  Token [”m”] {
    type: 'character',
    value: 'm',
    scope: 'root negative-lookaround'
  },
  Token [”)”] {
    type: 'lookaround-perimeter',
    value: ')',
    scope: 'root negative-lookaround'
  }
]