Tokenize | npm.io

printf format string tokenize

printf tokenize token lexer

0.0.6 • Published 2 years ago

personal tokenize util

token tokenize tokenizer

2.0.0 • Published 6 years ago

Pacote desenvolvido para ajudar a criar tokens rapidamente.

nubin nubinbrasil tokenizer tokenize token

1.0.1 • Published 5 years ago

Convenient lexical scanner inspired by Ruby StringScanner

stringscanner scanner string text textscanner lex lexer lexical analysis token

0.0.8 • Published 4 years ago

JavaScript library for tokenizing strings with regular expressions

lexer tokenizer syntax lex tokenize token parsing

2.2.0 • Published 4 years ago

get word counts / frequencies on a per-speaker or per-category basis, or as an aggregate

wordcount frequency speakers tokenize stem category nlp bayes predict

0.0.165 • Published 5 years ago

Convenient lexical scanner inspired by Ruby's StringScanner

x-scanner xscanner stringscanner scanner string text textscanner lex lexer lexical

0.4.9 • Published 4 years ago

Tokenizes an HTML string, extracting plain text while ignoring HTML tags

html plaintext text token tokenize strip nlp

1.0.3 • Published 2 years ago

Extracts plain text from Markdown strings

markdown plaintext text token tokenize strip nlp

1.0.3 • Published 2 years ago

json-operations allows you to work with JSON, parse in a standardized way, get tokenize JSON, etc...

json util utilities operations token tokenize tokenizer parser JSON.parse JSON

0.0.6 • Published 3 years ago

A CLI tool for converting JSX files into standard JS.

jsx transform jsx-transform convert parse tokenize tokenizer

1.1.7 • Published 3 years ago

Tokenizer for js, jsx, (ts, tsx)

js ts jsx tsx tokenize

0.2.9 • Published 3 years ago

A regex that tokenizes JavaScript.

JavaScript js token tokenize regex

5.0.0 • Published 2 years ago

A CLI command for converting JSX files into standard JS.

cli jsx transform jsx-transform h hyperscript htm convert parse tokenize

2.2.1 • Published 3 years ago

Lemmize and tokenize string which contains Chinese and English words

imago pni tokenize

1.0.0 • Published 7 years ago

hast utility to parse from HTML

unist hast hast-util util utility html parse tokenize

2.0.3 • Published 2 years ago

A high-performance wrapper around `Intl.Segmenter` for efficient text segmentation. This class resolves memory handling issues seen with large strings and can enhance performance by 50-500x. Only ~70 loc (with comments) and no dependencies.

ansi ascii character characters chunk chunks emoji emojis graphemes intl

1.0.0 • Published 1 year ago

convert a string to list of words

lexer string-to-words tokenize token

1.0.1 • Published 4 years ago

html tokenizer

html token tokenize parse sanitize

1.2.0 • Published 4 years ago

Parsing and tokenizing attributes string

attr attrs attribute attributes prop props property properties object tokenize

2.2.3 • Published 2 years ago