npm.io
0.1.6 • Published 1 year ago

@mlc-ai/web-tokenizers

Licence
Apache-2.0
Version
0.1.6
Deps
0
Size
9.3 MB
Vulns
0
Weekly
0
Stars
514

WebTokenizers

| NPM Package | WebLLM |

WebTokenizers is a javascript binding library that can be universally deployed. It wraps and binds the HuggingFace tokenizers library and sentencepiece and provides a minimum common interface in havascript.

Build from Srouce

source /path/to/emsdk_env.sh
npm install
npm run build

To try out the test webpage

cd tests
npm start

Keywords