bert-tokenizer
A simple tool to generate bert tokens and input features
1.1.8 • Published 6 years ago
WordPiece Packages
kitoken
Fast and versatile tokenizer for language models, supporting BPE, Unigram and WordPiece tokenization
0.10.1 • Published 1 year ago