tokenizer-next v3.0.1

iterator based tokenizer for writing parsers

API

To capture, RegExps must use a named group.

const tokenize = createTokenizer(
  /(?<ident>[a-z]+)/, // named groups determine token `group`
  /(?<number>[0-9]+)/
)

// using next()
const next = tokenize('hello 123')
console.log(next()) // => {group: 'ident', value: 'hello', index: 0}
console.log(next()) // => {group: 'number', value: '123', index: 6}
console.log(next()) // => undefined

// using for of
for (const token of tokenize('hello 123')) {
  console.log(token)
  // => {group: 'ident', value: 'hello', index: 0}
  // => {group: 'number', value: '123', index: 6}
}

// using spread
const tokens = [...tokenize('hello 123')]
console.log(tokens)
// => [
//   {group: 'ident', value: 'hello', index: 0},
//   {group: 'number', value: '123', index: 6}
// ]

To capture, RegExps must use a named group.

const tokenize = createTokenizer(
  /(?<ident>[a-z]+)/, // named groups determine token `group`
  /(?<number>[0-9]+)/
)

Credits

• match-to-token by stagas – transform a RegExp named group match to a more useful object

Contributing

Fork or edit and submit a PR.

All contributions are welcome!

License

MIT © 2022 stagas