tokenary v2.1.1

tokenary

Build tokenizers for javascript

Basic Usage (CSV tokenizer)

const {
    tokenary,
    reducer: { ifChar, single, makeToken, everythingUntil }
} = require('tokenary');

const TokenType = {
    comma: 'COMMA',
    value: 'VALUE',
    newline: 'NEWLINE',
};

const tokenizeCSV = tokenary([
    ifChar({
        ',': single(makeToken(TokenType.comma)),
        '\n': single(makeToken(TokenType.newline))
    }),
    consume(everythingUntil(',', '\n'))(makeToken(TokenType.value))
]);

const testCSV = 
`1,Up
2,Left
3,Right`;

const tokens = tokenizeCSV(testCSV);
console.log(prettyPrint(tokens));
/* prints:
[
  <Token type='VALUE' lexeme='1' offset=0>,
  <Token type='COMMA' lexeme=',' offset=1>,
  <Token type='VALUE' lexeme='Up' offset=2>,
  <Token type='NEWLINE' lexeme='\n' offset=4>,
  <Token type='VALUE' lexeme='2' offset=5>,
  <Token type='COMMA' lexeme=',' offset=6>,
  <Token type='VALUE' lexeme='Left' offset=7>,
  <Token type='NEWLINE' lexeme='\n' offset=11>,
  <Token type='VALUE' lexeme='3' offset=12>,
  <Token type='COMMA' lexeme=',' offset=13>,
  <Token type='VALUE' lexeme='Right' offset=14>
]
*/

See examples for more.

API

Classes

Objects

Functions

Typedefs

TokenError

Kind: global class

new TokenError(message, lexeme, offset, state)

predicate : object

Functions that return true or false.

Kind: global namespace

• predicate : object
  • .is(truth) ⇒ Predicate
  • .isOneOf(...truths) ⇒ Predicate
  • .matches(regex) ⇒ Predicate
  • .not(predicate) ⇒ Predicate
  • .or(...predicates) ⇒ Predicate
  • .nor(...predicates) ⇒ Predicate
  • .and(...predicates) ⇒ Predicate
  • .nand(...predicates) ⇒ Predicate
  • .xor(predicate1, predicate2) ⇒ Predicate
  • .Predicate ⇒ boolean

predicate.is(truth) ⇒ Predicate

Checks if actual is strict equal (===) to truth

Kind: static method of predicate

predicate.isOneOf(...truths) ⇒ Predicate

Checks if actual is any of the values in truths

Kind: static method of predicate

predicate.matches(regex) ⇒ Predicate

Checks if actual matches the regular expression

Kind: static method of predicate

predicate.not(predicate) ⇒ Predicate

Logical not operator on a predicate

Kind: static method of predicate

predicate.or(...predicates) ⇒ Predicate

Logical or operator on predicates

Kind: static method of predicate

predicate.nor(...predicates) ⇒ Predicate

Logical nor operator on predicates. Short for not(or(...predicates))

Kind: static method of predicate

predicate.and(...predicates) ⇒ Predicate

Logical and operator on predicates

Kind: static method of predicate

predicate.nand(...predicates) ⇒ Predicate

Logical nand operator on predicates. Short for not(and(...predicates))

Kind: static method of predicate

predicate.xor(predicate1, predicate2) ⇒ Predicate

Logical xor operator on 2 predicates

Kind: static method of predicate

predicate.Predicate ⇒ boolean

Kind: static typedef of predicate

reducer : object

Kind: global namespace

• reducer : object
  • .keywords(keywordMap, [settings]) ⇒ Reducer
  • .ifThen(predicate) ⇒ function
  • .ifChar(reducerMap) ⇒ Reducer
  • .single(tokenCreator) ⇒ Reducer
  • .consume(reducer) ⇒ function
  • .sequence(reducers) ⇒ Reducer
  • .everything() ⇒ Reducer
  • .everythingUntil(chars) ⇒ Reducer
  • .char(char) ⇒ Reducer
  • .untilRegexFails(regex) ⇒ Reducer
  • .whitespace() ⇒ Reducer

reducer.keywords(keywordMap, settings) ⇒ Reducer

Extracts keywords from the text

Kind: static method of reducer

reducer.ifThen(predicate) ⇒ function

If the predicate is true, run the reducer

Kind: static method of reducer

reducer.ifChar(reducerMap) ⇒ Reducer

If a matching character is found, runs the given reducer

Kind: static method of reducer

reducer.single(tokenCreator) ⇒ Reducer

Creates a token from the single current character

Kind: static method of reducer

reducer.consume(reducer) ⇒ function

Creates a token from the characters passed over in the given reducer Expects the given reducers to not return any tokens

Kind: static method of reducer

reducer.sequence(reducers) ⇒ Reducer

Pipe output of a sequence of reducers together (left to right)

Kind: static method of reducer

reducer.everything() ⇒ Reducer

Creates a token from every character that follows

Kind: static method of reducer

reducer.everythingUntil(chars) ⇒ Reducer

Creates a token from every character that follows until one given is reached

Kind: static method of reducer

reducer.char(char) ⇒ Reducer

Advances past a single character, throws an error if it's not what is expected

Kind: static method of reducer

reducer.untilRegexFails(regex) ⇒ Reducer

Runs the regex until the regex fails on all characters advanced past

Kind: static method of reducer

reducer.whitespace() ⇒ Reducer

Advances past all contiguous whitespace characters

Kind: static method of reducer

token : object

Kind: global namespace

• token : object
  • .makeToken(type) ⇒ TokenCreator
  • .makeNothing() : TokenCreator
  • .makeError(message) ⇒ TokenCreator
  • .stringifyToken(token) ⇒ string
  • .prettyPrint(tokens) ⇒ string

token.makeToken(type) ⇒ TokenCreator

Creates a token object

Kind: static method of token

token.makeNothing() : TokenCreator

Creates nothing

Kind: static method of token

token.makeError(message) ⇒ TokenCreator

Throws an error

Kind: static method of token
Throws:

• TokenError

token.stringifyToken(token) ⇒ string

Formats a token for printing

Kind: static method of token

token.prettyPrint(tokens) ⇒ string

Formats an array of tokens for printing

Kind: static method of token

tokenary(reducers, settings) ⇒ function

Creates a tokenizing function

Kind: global function

TokState : reducer.Reducer

Kind: global typedef

• TokState : reducer.Reducer
  • .create(text, [current], [tokens]) ⇒ TokState
  • .advance(state) ⇒ TokState

TokState.create(text, current, tokens) ⇒ TokState

Creates a TokState object

Kind: static method of TokState

TokState.advance(state) ⇒ TokState

Increments TokState.current by 1 (creates new object)

Kind: static method of TokState

TokenCreator : tokState.TokState

Kind: global typedef

Reducer ⇒ TokState

Kind: global typedef

TokState : tokState.TokState

Kind: global typedef

• TokState : tokState.TokState
  • .create(text, [current], [tokens]) ⇒ TokState
  • .advance(state) ⇒ TokState

TokState.create(text, current, tokens) ⇒ TokState

Creates a TokState object

Kind: static method of TokState

TokState.advance(state) ⇒ TokState

Increments TokState.current by 1 (creates new object)

Kind: static method of TokState

Token

Kind: global typedef
Properties

TokenCreator ⇒ Token | undefined

Kind: global typedef

Token : token.Token

Kind: global typedef

TokState

Kind: global typedef
Properties

• TokState
  • .create(text, [current], [tokens]) ⇒ TokState
  • .advance(state) ⇒ TokState

TokState.create(text, current, tokens) ⇒ TokState

Creates a TokState object

Kind: static method of TokState

TokState.advance(state) ⇒ TokState

Increments TokState.current by 1 (creates new object)

Kind: static method of TokState

TokState : tokState.TokState

Kind: global typedef

• TokState : tokState.TokState
  • .create(text, [current], [tokens]) ⇒ TokState
  • .advance(state) ⇒ TokState

TokState.create(text, current, tokens) ⇒ TokState

Creates a TokState object

Kind: static method of TokState

TokState.advance(state) ⇒ TokState

Increments TokState.current by 1 (creates new object)

Kind: static method of TokState