1.0.7 • Published 2 years ago

@beskar-labs/gpt-encoder v1.0.7

Weekly downloads
-
License
MIT
Repository
github
Last release
2 years ago

gpt-encoder

GPT-2 and GPT-3 use byte pair encoding to turn text into a series of integers to feed into the model. latitudegames implemented this in Node.js. This is an implementation of their encoder for use in the browser, re-written in Typescript.

Installation

npm install @beskar-labs/gpt-encoder

Usage

import { encode } from '@beskar-labs/gpt-encoder';

const tokens = encode(
  'This is an example sentence to try encoding out on!'
).length;
1.0.7

2 years ago

1.0.6

2 years ago

1.0.5

2 years ago

1.0.4

2 years ago

1.0.3

2 years ago

1.0.2

2 years ago

1.0.1

2 years ago