@mlc-ai/web-tokenizers v0.1.5

WebTokenizers

| NPM Package | WebLLM |

WebTokenizers is a javascript binding library that can be universally deployed. It wraps and binds the HuggingFace tokenizers library and sentencepiece and provides a minimum common interface in havascript.

Build from Srouce

source /path/to/emsdk_env.sh
npm install
npm run build

To try out the test webpage

cd tests
npm start