Tokenizer | npm.io

Split text into sentence strings or word arrays with Sentence Boundary Detection (SBD). Based on http://github.com/Tessmore/sbd by Fabiën Tesselaar.

sentence detection boundary lexer tokenizer

2.0.2 • Published 9 years ago

a wrapper around the LunaSec CLI enabling it to be used as an NPM package

lunasec lunasec-cli tokenization tokenizer

1.0.8-alpha.0 • Published 4 years ago

一个功能强大且易于使用的命令行解析库，专为现代 TypeScript/JavaScript 应用程序设计

command parser cli command-line typescript javascript shell ast tokenizer pipe

0.1.1 • Published 12 months ago

Configurable SQL formatter

sql formatter tokenizer

1.0.1 • Published 7 years ago

JavaScript implementation of Japanese morphological analyzer

japanese morphological analyzer nlp pos pos tagger tokenizer

0.3.0 • Published 6 years ago

Forked version of kuromoji with better compatibility for browsers

japanese morphological analyzer nlp pos pos tagger tokenizer

1.1.0 • Published 3 years ago

The path pattern parser, that supports named variables, variable constraints, bash-like alternation, regular expressions, and wildcards.

url route path pattern parser tokenizer

1.0.1 • Published 5 years ago

Enhances comprehension for MindScript-powered models by tokenizing text.

tokenizer mindscript ai

0.0.1 • Published 3 years ago

Evaluate javascript expressions. No eval. No dependencies. Only 6kb.

parser javascript js expressions eval evaluate evaluator tokenizer

0.3.1 • Published 5 years ago

A lua parser

lua parser ast tokenizer

1.0.9 • Published 2 years ago

Flexible and asynchronous string tokenizer

lexer tokenizer tool util string regex

1.0.3 • Published 6 years ago

A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 / Claude Instant / Claude 2

BPE encoder decoder tokenizer GPT GPT-2 GPT-3 GPT-3.5 GPT-4 NLP

0.3.2 • Published 3 years ago

125 26

Tokenizer Packages

@knod/sbd

@lunasec/cli

@istock-shell/command-parser

@simonwardjones/sql-formatter

@silvestre/kuromoji

@sglkc/kuromoji

@smikhalevski/route-pattern

@mindlearn/tokenizer

@rosell/js-expression

@meepen/luaparse

@rustworkshop/lexer

@meistrari/gpt-tokenizer