bert-tokenizer
A simple tool to generate bert tokens and input features
1.1.8 • Published 5 years ago
Wordpiece Packages
kitoken
Fast and versatile tokenizer for language models, supporting BPE, Unigram and WordPiece tokenization
0.10.1 • Published 7 months ago