# tokenspliter

> Simple Token Spliter

Latest version **0.0.4** (published 2016-04-22) · MIT license · 0 weekly downloads

## Install

```sh
npm install tokenspliter
pnpm add tokenspliter
yarn add tokenspliter
bun add tokenspliter
```

## Health

**Score 15/100 (F)** — status: abandoned.

Positive: no vulnerabilities.

Warnings: low downloads; no types; no esm support; pre 1.0.

Negative: abandoned; low maintenance score.

## Facts

| | |
|---|---|
| Version | 0.0.4 |
| Published | 2016-04-22 |
| First published | 2015-10-19 |
| Weekly downloads | 0 |
| License | MIT |
| TypeScript types | none |
| Module format | CommonJS |
| Dependencies | 0 |
| Known vulnerabilities | 0 |
| Install scripts | no |
| Author | ddchen |
| Maintainers | ddchen |
| Keywords | token, stream, spliter, morphology |

## Links

- npm: https://www.npmjs.com/package/tokenspliter
- Repository: https://github.com/LoveKino/tokenspliter
- Homepage: https://github.com/LoveKino/tokenspliter#readme
- Issues: https://github.com/LoveKino/tokenspliter/issues
- npm.io page: https://npm.io/package/tokenspliter

## Alternatives

- [byte-size](https://npm.io/package/byte-size.md) — 2.1M weekly downloads
- [speed-limiter](https://npm.io/package/speed-limiter.md) — 16.0K weekly downloads
- [@powersync/node](https://npm.io/package/@powersync/node.md) — 10.9K weekly downloads
- [@ledgerhq/coin-cardano](https://npm.io/package/@ledgerhq/coin-cardano.md) — 1.0K weekly downloads
- [@jayesol/jayeson.lib.streamfinder](https://npm.io/package/@jayesol/jayeson.lib.streamfinder.md) — 1.0K weekly downloads

## Recent versions

- 0.0.4 (latest) — 2016-04-22
- 0.0.3 — 2016-04-12
- 0.0.2 — 2015-10-31
- 0.0.1 — 2015-10-19

## README

# tokenspliter

Token spliter used to split tokens from text.

## exmaple

```js
let tokenspliter = Tokenspliter([{
    type: 'identify',
    regular: /[a-zA-Z][a-zA-Z0-9]*/
}, {
    type: 'number',
    regular: /[0-9]+/
}, {
    type: 'whitespace',
    regular: /\s+/
}]);

let tokenStream = tokenspliter('a09 bcd 12');
let t1 = tokenStream.next(); // {type: 'identity', 'lexion': 'a09'}
let t2 = tokenStream.next(); // {type: 'whitespace', 'lexion': ' '}
```

## install

`npm i tokenspliter --save`

## usage

### initial tokenspliter

```js
let tokenspliter = Tokenspliter([{
    type: 'identify',
    regular: /[a-zA-Z][a-zA-Z0-9]*/
}, {
    type: 'number',
    regular: /[0-9]+/
}, {
    type: 'whitespace',
    regular: /\s+/
}]);
```

Initial a token spliter, by pass an array which item contains a type and a regular expression.

### accept text

```js
let tokenStream = tokenspliter('a09 bcd 12');
```

### operate token stream

- next

Move to next, and return next token.

```js
let t1 = tokenStream.next(); // {type: 'identity', 'lexion': 'a09'}
```

- lookAhead

Look ahead one step, and return next token, but token stream will hold.

```js
let t1 = tokenStream.lookAhead(); // {type: 'identity', 'lexion': 'a09'}
```

- lookAheads

Look ahead more steps.

First parameter is steps, second paramter is filter (optional). If has filter, only when filter returns true, it will count one step ahead.

```js
let t1 = tokenStream.lookAheads(3, (token) => token.type !== 'whitespace');
```

- isEmpty

If token stream comes to end, will return true, otherwise will return false;

```js
tokenStream.isEmpty()
```

## error token

- default handling

By default, throw an exception.

```js
let tokenspliter = Tokenspliter([{
    type: 'identify',
    regular: /[a-zA-Z][a-zA-Z0-9]*/
}, {
    type: 'number',
    regular: /[0-9]+/
}, {
    type: 'whitespace',
    regular: /\s+/
}]);
let tokenStream = tokenspliter('a09 & bcd 12');
tokenStream.next();
tokenStream.next();
tokenStream.next(); // throw an error
```

- ignore handling

Set tokespliter's second parameter as null, will ignore exception, and when meet unrecoginized token, will return it as error token. The data structure of error token is:

```
{
    type: 'error_type',
    lexcion: 'some characters'
}
```

```js
let tokenspliter = Tokenspliter([{
    type: 'identify',
    regular: /[a-zA-Z][a-zA-Z0-9]*/
}, {
    type: 'number',
    regular: /[0-9]+/
}, {
    type: 'whitespace',
    regular: /\s+/
}]);
let tokenStream = tokenspliter('a09 & bcd 12', null); // null
tokenStream.next();
tokenStream.next();
tokenStream.next(); // {type: 'error_type', lexion: '&'}
```

---
_Source: https://npm.io/package/tokenspliter · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._