0.1.27 • Published 3 months ago

@anysphere/tiktoken-node v0.1.27

Weekly downloads
-
License
-
Repository
-
Last release
3 months ago

@anysphere/tiktoken-node

We use our own fork for now because we are making changes that may not be useful to everyone. For example, we add support for special tokens, and we also add support for running tokenization asynchronously with the computation happening on a different thread.