0.1.5 • Published 11 months ago

@mlc-ai/web-tokenizers v0.1.5

Weekly downloads
-
License
Apache-2.0
Repository
github
Last release
11 months ago

WebTokenizers

| NPM Package | WebLLM |

WebTokenizers is a javascript binding library that can be universally deployed. It wraps and binds the HuggingFace tokenizers library and sentencepiece and provides a minimum common interface in havascript.

Build from Srouce

source /path/to/emsdk_env.sh
npm install
npm run build

To try out the test webpage

cd tests
npm start
0.1.4

1 year ago

0.1.5

11 months ago

0.1.3

1 year ago

0.1.2

2 years ago

0.1.0

2 years ago