1.0.5 • Published 3 years ago

@bitacode/tokenator v1.0.5

Weekly downloads
13
License
MIT
Repository
github
Last release
3 years ago

tokenator: a Node.js tokenization library

<CircleCI> npm License

tokenator is a simple to use library that rework a string splitting it to meaningful tokens.

Table of Contents

Description

This library transforms a series of characters to tokens identified by input strings.

e.g.
"The itsy bitsy spider crawled up the water spout. | Down came the rain, and washed the spider out. | Out came the sun, and dried up all the rain, | and the itsy bitsy spider went up the spout again"

Splitting strings: spider, rain

Result tokens:
  > "The itsy bitsy "
  > "spider"
  > " crawled up the water spout. | Down came the "
  > "rain"
  > ", and washed the "
  > "spider"
  > " out. | Out came the sun, and dried up all the "
  > "rain"
  > ", | and the itsy bitsy "
  > "spider"
  > " went up the spout again"

Installation

npm install tokenator

Code Examples

The signatures of the available methods are:

function tokenator(source: string, ...separators: string[]): string[]
function tokenatorStream(...separators: string[]): (separatorsEncoding: string = "utf-8") => stream.Transform

Code examples could be consulted within the test folder:

These functions work over multiple separators, like in the example above. The tokenatorStream() function creates a Transform stream. NodeJS Stream usage specification is available on the NodeJS official documentation.

Changelog

Available here

License

MIT

Available here

1.0.5

3 years ago

1.0.4

4 years ago

1.0.3

4 years ago

1.0.2

4 years ago

1.0.1

4 years ago

1.0.0

4 years ago

0.2.1

4 years ago

0.2.2

4 years ago

0.1.2

4 years ago

0.1.3

4 years ago

0.1.1

4 years ago

0.1.0

4 years ago