# regexp-stream-tokenizer

> A regular expression (RexExp) stream tokenizer.

Latest version **0.2.2** (published 2016-06-12) · MIT license · 0 weekly downloads

## Install

```sh
npm install regexp-stream-tokenizer
pnpm add regexp-stream-tokenizer
yarn add regexp-stream-tokenizer
bun add regexp-stream-tokenizer
```

## Health

**Score 15/100 (F)** — status: abandoned.

Positive: no vulnerabilities.

Warnings: low downloads; no types; no esm support; pre 1.0.

Negative: abandoned; low maintenance score.

## Facts

| | |
|---|---|
| Version | 0.2.2 |
| Published | 2016-06-12 |
| First published | 2016-01-03 |
| Weekly downloads | 0 |
| License | MIT |
| TypeScript types | none |
| Module format | CommonJS |
| Dependencies | 5 |
| Known vulnerabilities | 0 |
| Install scripts | no |
| GitHub stars | 2 |
| Author | James Ramsay |
| Maintainers | jramsay |
| Keywords | streams, through, through2, tokenizer, tokeniser, regexp, regex |

## Links

- npm: https://www.npmjs.com/package/regexp-stream-tokenizer
- Repository: https://github.com/jamesramsay/regexp-stream-tokenizer
- Issues: https://github.com/jamesramsay/regexp-stream-tokenizer/issues
- npm.io page: https://npm.io/package/regexp-stream-tokenizer

## Dependencies (5)

- [xtend](https://npm.io/package/xtend.md) ^4.0.1
- [lodash](https://npm.io/package/lodash.md) ^4.0.0
- [through2](https://npm.io/package/through2.md) ^2.0.0
- [left-split](https://npm.io/package/left-split.md) ^1.0.0
- [clone-regexp](https://npm.io/package/clone-regexp.md) ^1.0.0

## Alternatives

- [byte-size](https://npm.io/package/byte-size.md) — 2.1M weekly downloads
- [speed-limiter](https://npm.io/package/speed-limiter.md) — 16.0K weekly downloads
- [@powersync/node](https://npm.io/package/@powersync/node.md) — 10.9K weekly downloads
- [@ledgerhq/coin-cardano](https://npm.io/package/@ledgerhq/coin-cardano.md) — 1.0K weekly downloads
- [@jayesol/jayeson.lib.streamfinder](https://npm.io/package/@jayesol/jayeson.lib.streamfinder.md) — 1.0K weekly downloads

## Recent versions

- 0.2.2 (latest) — 2016-06-12
- 0.2.0 — 2016-05-28
- 0.1.1 — 2016-03-29
- 0.1.0 — 2016-03-29
- 0.0.1 — 2016-01-03

## README

# regexp-stream-tokenizer

[![Version](https://img.shields.io/npm/v/regexp-stream-tokenizer.svg)](https://npmjs.com/package/regexp-stream-tokenizer)
[![License](https://img.shields.io/npm/l/regexp-stream-tokenizer.svg)](https://npmjs.com/package/regexp-stream-tokenizer)
[![Build Status](https://img.shields.io/travis/jamesramsay/regexp-stream-tokenizer.svg)](https://travis-ci.org/jamesramsay/regexp-stream-tokenizer)
[![Coverage Status](https://img.shields.io/codecov/c/github/jamesramsay/regexp-stream-tokenizer.svg)](https://codecov.io/github/jamesramsay/regexp-stream-tokenizer)
[![Dependency Status](https://img.shields.io/david/jamesramsay/regexp-stream-tokenizer.svg)](https://david-dm.org/jamesramsay/regexp-stream-tokenizer)

[![NPM](https://nodei.co/npm/regexp-stream-tokenizer.png)](https://nodei.co/npm/regexp-stream-tokenizer/)

This is a simple regular expression based tokenizer for streams.

**IMPORTANT:** If you return `null` from your function, the stream will end there.

**IMPORTANT:** Only supports object mode streams.

```javascript

var tokenizer = require("regexp-stream-tokenizer");

var words = tokenizer(/w+/g);

// Sink receives tokens: 'The', 'quick', 'brown', 'fox', 'jumps', 'over', 'the', 'lazy', 'dog'
words.write('The quick brown fox jumps over the lazy dog');
words.pipe(sink)

// Separators are excluded by default, but can be included
var wordsAndSeparators = tokenizer({ separator: true }, /w+/g);

// Sink receives tokens: 'The', ' ', 'quick', ' ', 'brown', ' ', 'fox', ' ', 'jumps', ' ', 'over', ...
words.write('The quick brown fox jumps over the lazy dog');
words.pipe(sink)

```

## API

```javascript
require("regexp-stream-tokenizer")([options,] regexp)
```

Create a `stream.Transform` instance with `objectMode: true` that will tokenize the input stream using the regexp.

```javascript
var Tx = require("regexp-stream-tokenizer").ctor([options,] regexp)
```

Create a reusable `stream.Transform` TYPE that can be called via `new Tx` or `Tx()` to create an instance.

__Arguments__

- `options`
  - `excludeZBS` (boolean): defaults `true`.
  - `token` (boolean|string|function): defaults `true`.
  - `separator` (boolean|string|function): defaults `false`.
  - `leaveBehind` (string|Array): optionally provides pseudo-lookbehind support.
  - all other through2 options.
- `regexp` (RegExp): The regular expression using which the stream will be tokenized.

---
_Source: https://npm.io/package/regexp-stream-tokenizer · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._