# bert-tokenizer

> A simple tool to generate bert tokens and input features

Latest version **1.1.8** (published 2020-04-05) · apache-2.0 license · 0 weekly downloads

## Install

```sh
npm install bert-tokenizer
pnpm add bert-tokenizer
yarn add bert-tokenizer
bun add bert-tokenizer
```

## Health

**Score 15/100 (F)** — status: abandoned.

Positive: no vulnerabilities.

Warnings: low downloads; no types; no esm support.

Negative: abandoned; low maintenance score.

## Facts

| | |
|---|---|
| Version | 1.1.8 |
| Published | 2020-04-05 |
| First published | 2020-03-02 |
| Weekly downloads | 0 |
| License | apache-2.0 |
| TypeScript types | none |
| Module format | CommonJS |
| Dependencies | 0 |
| Unpacked size | 421 KB |
| Known vulnerabilities | 0 |
| Install scripts | no |
| GitHub stars | 10 |
| Maintainers | htchang |
| Keywords | tokenizer, bert, wordpiece |

## Links

- npm: https://www.npmjs.com/package/bert-tokenizer
- Repository: https://github.com/tedhtchang/bert-tokenizer
- Homepage: https://github.com/tedhtchang/bert-tokenizer#readme
- Issues: https://github.com/tedhtchang/bert-tokenizer/issues
- npm.io page: https://npm.io/package/bert-tokenizer

## Recent versions

- 1.1.8 (latest) — 2020-04-05
- 1.1.7 — 2020-03-10
- 1.1.6 — 2020-03-10
- 1.1.5 — 2020-03-09
- 1.1.4 — 2020-03-05
- 1.1.3 — 2020-03-05
- 1.1.2 — 2020-03-05
- 1.1.1 — 2020-03-05
- 1.1.0 — 2020-03-02
- 1.0.0 — 2020-03-02

## README

# bert-tokenizer
A simple tool to generate bert tokens and input features

## Installation
```npm install bert-tokenizer```

## Usage

```
const {BertTokenizer} = require('bert-tokenizer');

const text = "i like strawberries";
const tokenIds = [ 1045, 2066, 13137, 20968 ];
const vocabUrl = 'node_modules/bert-tokenizer/assets/vocab.json'

const bertTokenizer = new BertTokenizer(vocabUrl, true);
console.log(bertTokenizer.tokenize(text));
//[ 1045, 2066, 13137, 20968 ]
console.log(bertTokenizer.convertIdsToTokens(tokenIds));
//[ '▁i', '▁like', '▁straw', 'berries' ]
console.log(bertTokenizer.convertSingleExample(text));
//[ '[CLS]', '▁i', '▁like', '▁straw', 'berries', '[SEP]' ]
{
  inputIds: [
    101, 1045, 2066, 13137, 20968, 102, 0, 0, 0, 0, 0,
      0,    0,    0,     0,     0,   0, 0, 0, 0, 0, 0,
      0,    0,    0,     0,     0,   0, 0, 0, 0, 0, 0,
      0,    0,    0,     0,     0,   0, 0, 0, 0, 0, 0,
      0,    0,    0,     0,     0,   0, 0, 0, 0, 0, 0,
      0,    0,    0,     0,     0,   0, 0, 0, 0, 0, 0,
      0,    0,    0,     0,     0,   0, 0, 0, 0, 0, 0,
      0,    0,    0,     0,     0,   0, 0, 0, 0, 0, 0,
      0,    0,    0,     0,     0,   0, 0, 0, 0, 0, 0,
      0,
    ... 28 more items
  ],
  segmentIds: [
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
    0, 0, 0, 0,
    ... 28 more items
  ],
  inputMask: [
    1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
    0, 0, 0, 0,
    ... 28 more items
  ]
}
```
## Development

To modified the code, first clone the repo and
install the dependencies.
```
git clone https://github.com/tedhtchang/bert-tokenizer.git
cd bert-tokenizer
npm install
```
To make a change, edit the source code under `src` folder and save.
Next compile the source code.
```
npm run build
```

Run test cases to make sure you did not break any thing:
```
npm run test
```

To test your package, run npm install with the full path to
your package directory:
```
npm install local-path-to-bert-tokenizer-folder
```

---
_Source: https://npm.io/package/bert-tokenizer · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._