# string-tokenizer

> Simple string tokenizer, to make RegExps less painful.

Latest version **0.0.8** (published 2015-04-27) · ISC license · 0 weekly downloads

## Install

```sh
npm install string-tokenizer
pnpm add string-tokenizer
yarn add string-tokenizer
bun add string-tokenizer
```

## Health

**Score 15/100 (F)** — status: abandoned.

Positive: no vulnerabilities.

Warnings: low downloads; no types; no esm support; pre 1.0.

Negative: abandoned; low maintenance score.

## Facts

| | |
|---|---|
| Version | 0.0.8 |
| Published | 2015-04-27 |
| First published | 2015-04-17 |
| Weekly downloads | 0 |
| License | ISC |
| TypeScript types | none |
| Module format | CommonJS |
| Dependencies | 6 |
| Known vulnerabilities | 0 |
| Install scripts | no |
| Author | Damian Baar |
| Maintainers | damianbaar |

## Links

- npm: https://www.npmjs.com/package/string-tokenizer
- npm.io page: https://npm.io/package/string-tokenizer

## Dependencies (6)

- [uniq](https://npm.io/package/uniq.md) ^1.0.1
- [array-last](https://npm.io/package/array-last.md) ^1.0.2
- [object-keys](https://npm.io/package/object-keys.md) ^1.0.3
- [object-assign](https://npm.io/package/object-assign.md) ^2.0.0
- [object-values](https://npm.io/package/object-values.md) ^1.0.0
- [re-define-include-external](https://npm.io/package/re-define-include-external.md) ^0.4.1

## Recent versions

- 0.0.8 (latest) — 2015-04-27
- 0.0.7 — 2015-04-27
- 0.0.6 — 2015-04-21
- 0.0.5 — 2015-04-21
- 0.0.4 — 2015-04-18
- 0.0.3 — 2015-04-18
- 0.0.2 — 2015-04-17
- 0.0.1 — 2015-04-17

## README

## `string-tokenizer`

Break your string into tokens in `node` and `browser` without headache.

### Why
* missing named groups in `RegExp` in js
* to make code cleaner and easier to follow, parsing is not always an elegant thing
* `RegExp.source` concatenation `(/foo/)|(/baz/)|..` is not too flexible
* `JSON` as token provider out of the box
* to keep `RegExps` simple
* to provide easy way to assest whether something should be a token or not based upon external data

### Usage

* in `node` or any other `common-js-browser-bundler`

```
npm install --save string-tokenizer
```

```js
var tokenizer = require('string-tokenizer')
```

* In browser 

```html 
<script src="./string-tokenizer/dist/string-tokenizer.js"></script>
```

```js
  var tokenizer = db.tokenizer //as a global
                               //wrapped by UMD, available via AMD as string-tokenizer
  var tokens = 
    tokenizer()
      .input('#aa test')
      .token('tag', /#(\w{2})\ /)
      .token('input', /.+/)
      .resolve()
  
  //result { tag: 'aa', input: 'test' }
```

### API

* basic functionallity

```js
    //Defining INPUT
    tokenizer(input?)                       //instance
      .input('#aa tes')                     //input

    //Making DEFINITIONs
      .token('tag', /#(\w{2})\ /, helper?)    //define pattern as 'tag' with optional helper method
      .token('input', /.+/)                   //define pattern as 'input'
      .token('tokenName', 'string or RegExp')
      
      //from object
      .tokens({ tag: 'pattern', query: 'pattern'}
    
    //Adding HELPERs to prettify matched result value (RegExp.exec)
      .helper('url', function(values){ return values[0] + values[1] })
      .helper('tokenName', function(values){ return values })

      //from object
      .helpers({ tag: function(val) {}, query: function(val) {} })

    //Invocation MODEs
    .walk(function(type, value, match) { })   //manual mode, invoked on each token
    //or
    .resolve(includeTokensPositionInfo?)      //auto mode, exporting tokens to object with optional info about token position

    //Debugging
    .debug()                                  //...
```

* skipping unwanted phrase during `walk`

TODO

* defining and using `helpers`

TODO

### Examples

```js
  tokenizer()
    .input('Test with #tag and a http://google.com/ #url')
      .token('tag', /#[\w\d-_]+/)
      .token('url', /(https?:\/\/)?([\da-z\.-]+)\.([a-z\.]{2,6})([\/\w \.-]*)*\/?/, function(values){ return values[0] })
      .token('space', /[\s]+/)
      .token('symbol', /[\w]+/)
    .resolve()

  //result .resolve()
  { symbol: [ 'Test', 'with', 'and', 'a' ],
    space: [ ' ', ' ', ' ', ' ', ' ' ],
    tag: [ '#tag', '#url' ],
    url: 'http://google.com/ ',
    _source: 'Test with #tag and a http://google.com/ #url' }
  
  //with .resolve(true)
  { symbol:
     [ { value: 'Test', position: 0 },
       { value: 'with', position: 5 },
       { value: 'and', position: 15 },
       { value: 'a', position: 19 } ],
    space:
     [ { value: ' ', position: 4 },
       { value: ' ', position: 9 },
       { value: ' ', position: 14 },
       { value: ' ', position: 18 },
       { value: ' ', position: 20 } ],
    tag:
     [ { value: '#tag', position: 10 },
       { value: '#url', position: 40 } ],
    url: { value: 'http://google.com/ ', position: 21 },
    _source: 'Test with #tag and a http://google.com/ #url' }
```

run `npm run example` or open `example.html` file to play with and get better overview.

---
_Source: https://npm.io/package/string-tokenizer · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._