Tokenizer | npm.io

n8n node for working with BPE Tokens with OpenAI's GPT models in mind.

n8n-community-node-package n8n nodemation n8n-node gpt bpe tokenizer

0.1.1 • Published 2 years ago

Parser grammar builder

grammars parsing parser-library tokenizer lexer

0.5.4 • Published 5 months ago

BPE tokenizer used for the NovelAI frontend.

gpt gpt2 pile sentencepiece nerdstash tokenizer novelai

1.1.0 • Published 12 months ago

Node based tokenizers for open source models hosted on HuggingFace.

tokenizer mistral mixtral gemma

2.0.1 • Published 1 year ago

React typeahead with Bootstrap styling

auto complete auto suggest auto-complete auto-suggest autocomplete autosuggest bootstrap bootstrap tokenizer bootstrap typeahead bootstrap-tokenizer

1.0.0 • Published 2 years ago

React typeahead with Bootstrap styling

auto complete auto suggest auto-complete auto-suggest autocomplete autosuggest bootstrap bootstrap tokenizer bootstrap typeahead bootstrap-tokenizer

5.2.2-beta4 • Published 3 years ago

Render HTML-like markup language from strings using React components

typescript npm lib boilerplate library react html hypertext markup headless

1.1.0 • Published 2 years ago

Node.js binding for nlpO3 Thai language processing library

thai tokenizer nlp rust word-segmentation

0.2.1 • Published 4 years ago

POS Tagger and lemmatizer for javascript

nlp tokenizer tokenize pos tagger Part-Of-Speech Tagger lemma lemmatizer javascript

1.0.2 • Published 12 months ago

String ngram splitter.

ngram split search tokenize tokenizer convert converter parse parser

0.1.3 • Published 4 months ago

Tokenizes a text using regex rules and returns the tokens with their positions in the text given.

tokenizer tokenize parser regex regexp token tokens

1.0.6 • Published 2 years ago

A tokenizer for AI repositories

ai repository tokenizer

0.0.1 • Published 5 months ago

A lightly-typescriptified version of jison

rerum repellat molestias parser generator voluptate flex tokenizer id typescript

1.0.2 • Published 1 year ago

Token generator/validator without external dependencies

random token generator javascript keygen string tokenizer number token number generator random string

2.2.1 • Published 3 years ago

A CLI tool to concatenate all text files in your CWD with headers for GPT prompt engineering.

text concatenate CLI Current Working Directory GPT tokens tokenizer ChatGPT prompt engineering token-count

1.0.2 • Published 1 year ago

Rich text and markdown tokenization made easy.

markdown rich text tokenizer

1.0.13 • Published 2 years ago

A simple (linear) lexer for converting a language string into a sequence of user-defined TOKEN_TYPES.

lexer parser tokenizer javascript

1.0.8 • Published 4 years ago

React component to convert a text string into React.ReactNodes

tokenizer parser text

1.0.4 • Published 9 months ago

r/w stream of glsl tokens

glsl tokenizer stream

3.0.3 • Published 11 months ago

Library to help you build your own type-safe parsers for any language.

abstract concrete syntax tree ast cst parser language tokenizer lexer

0.0.0 • Published 6 months ago

120 21 2224 25

Tokenizer Packages

n8n-nodes-gpt-tokenizer

nanolex

nai-js-tokenizer

omni-tokenizer

react-bootstrap-typeahead-with-restart

react-bootstrap-typeahead-portal

react-markup-formatter

nlpo3

nlp-js-tools-english

ngramjs

positional-tokenizer

repotok

repellenduscum

random-web-token

promptfusion

rich-text-tokenizer

simplelexer

react-text-tokenizer

glsl-tokenizer-fork-for-gm

grammars