1.1.2 • Published 9 days ago
@lenml/tokenizer-yi v1.1.2
@lenml/tokenizer-yi
a tokenizer.
based on
@lenml/tokenizers
Usage
import { fromPreTrained } from "@lenml/tokenizer-yi";
const tokenizer = fromPreTrained();
console.log(
"encode()",
tokenizer.encode("Hello, my dog is cute", null, {
add_special_tokens: true,
})
);
console.log(
"_encode_text",
tokenizer._encode_text("Hello, my dog is cute")
);
Full Tokenizer API
Complete api parameters and usage can be found in transformer.js tokenizers document
License
Apache-2.0