@lenml/tokenizers
a lightweight no-dependency fork of transformers.js (only tokenizers)
a lightweight no-dependency fork of transformers.js (only tokenizers)
baichuan2 tokenizer for NodeJS/Browser
chatglm3 tokenizer for NodeJS/Browser
claude tokenizer for NodeJS/Browser
claude1 tokenizer for NodeJS/Browser
command_r_plus tokenizer for NodeJS/Browser
gemma tokenizer for NodeJS/Browser
gpt2 tokenizer for NodeJS/Browser
gpt3 tokenizer for NodeJS/Browser
gpt35turbo tokenizer for NodeJS/Browser
gpt35turbo16k tokenizer for NodeJS/Browser
gpt4 tokenizer for NodeJS/Browser
gpt4o tokenizer for NodeJS/Browser
Prompt utilities for llama-guard. Use MLCommons taxonomies or build your own safety categories.