tokenize-monster

Tokenize Monster

Yet another powerful tokenizer in JS.

Installation

npm install --save tokenize-monster

Getting Started

const tokenizer = require('tokenize-monster');

const tokens = tokenizer.matchAndMap(
	'i\'d like to <drink> some <milk>',
	/<[^><]+>/,
	text => [[text]],
	text => [text]
);

console.log(tokens);
// => [ 'i\'d like to ', [ '<drink>' ], ' some ', [ '<milk>' ], '' ]

const nTokens = tokenizer.matchAndMap(
	'i\'d like to <drink> some <milk>',
	/<[^><]+>/,
	text => [text],
	text => []
);

console.log(nTokens);
// => [ '<drink>', '<milk>' ]

Example

• Simple XML Parser

Documentation

Our docs are available at Tokenize Monster Wiki.

Changelog

Visit Github Releases page for more information.