# mysql-tokenizer

> Replacement for sql-tokenizer which supports Mysql syntax. Also provides an experimental utility for structuring tokens into SQL statements.

Latest version **1.0.7** (published 2019-06-17) · MIT license · 0 weekly downloads

## Install

```sh
npm install mysql-tokenizer
pnpm add mysql-tokenizer
yarn add mysql-tokenizer
bun add mysql-tokenizer
```

## Health

**Score 15/100 (F)** — status: abandoned.

Positive: no vulnerabilities.

Warnings: low downloads; no types; no esm support.

Negative: abandoned; low maintenance score.

## Facts

| | |
|---|---|
| Version | 1.0.7 |
| Published | 2019-06-17 |
| First published | 2019-05-14 |
| Weekly downloads | 0 |
| License | MIT |
| TypeScript types | none |
| Module format | CommonJS |
| Dependencies | 0 |
| Unpacked size | 98.6 KB |
| Known vulnerabilities | 0 |
| Install scripts | no |
| GitHub stars | 4 |
| Author | Andy Holland |
| Maintainers | mashthekeys |
| Keywords | mysql, tokenizer, sql, parser |

## Links

- npm: https://www.npmjs.com/package/mysql-tokenizer
- Repository: https://github.com/mashthekeys/mysql-tokenizer
- Homepage: https://github.com/mashthekeys/mysql-tokenizer#readme
- Issues: https://github.com/mashthekeys/mysql-tokenizer/issues
- npm.io page: https://npm.io/package/mysql-tokenizer

## Alternatives

- [babylon](https://npm.io/package/babylon.md) — 5.1M weekly downloads
- [csscolorparser](https://npm.io/package/csscolorparser.md) — 3.7M weekly downloads
- [expr-eval-fork](https://npm.io/package/expr-eval-fork.md) — 1.5M weekly downloads
- [@leeoniya/ufuzzy](https://npm.io/package/@leeoniya/ufuzzy.md) — 247.7K weekly downloads
- [xml-parser](https://npm.io/package/xml-parser.md) — 78.4K weekly downloads

## Recent versions

- 1.0.7 (latest) — 2019-06-17
- 1.0.6 — 2019-06-17
- 1.0.5 — 2019-05-19
- 1.0.4 — 2019-05-18
- 1.0.3 — 2019-05-14

## README

# Mysql Tokenizer

Splits strings into tokens using the syntax of Mysql 8.

```javascript
const tokenize = require('mysql-tokenizer')();

const tokens = tokenize("SELECT 0, 0x1, '2', X'33';")

console.log(JSON.stringify(tokens));
/* Output:
["SELECT"," ","0",",","0x1",","," ","'2'",","," ","X'33'",";"]
*/
```

## Basic Usage
```javascript
const tokenize = require('mysql-tokenizer')();

const tokens = tokenize("SELECT tbl.x->'$', 1--2=<=>-1 FROM tbl;");

console.log(`Mysql: ${JSON.stringify(tokens)}`);

/* Output:
mysql: ["SELECT"," ","tbl",".","x","->","'$'",","," ","1","-","-","2","=","<=>","-","1"," ","FROM"," ","tbl",";"]
*/
````


## Comment Syntax
Comments are supported using ``--``, ``#`` and ``/*...*/`` syntaxes.

By default, comments using ```--``` must start with a space, or come at the end of a line, as in Mysql.

```javascript
const tokenize = require('mysql-tokenizer')();

const tokens = tokenize("SELECT 1--2=3 -- One less minus two equals three")

console.log(JSON.stringify(tokens));
/* Output:
["SELECT"," ","1","-","-","2","=","3"," ","-- One less minus two equals three"]
*/
```

To implement the SQL standard, which does not require following whitespace,
include ``{ regExp: require('mysql-tokenizer/lib/regexp-sql92') }`` in the options.


## Options
The following options are supported:

###``options.operators``: Array

Specify the list of valid operator tokens.  If not specified,
all operators from Mysql 8 are supported.

If this is the only option required, options can be specified as an Array.

Alternatively, specify ``options.regExp.Token.Operator``.

###``options.regExp.Initial``: Object

Override the parser's built-in regular expressions for character parsing.  
Initial RegExps must start with ```^``` and match at most one character.

```typescript
declare namespace options.regExp.Initial {
    let BinaryOrHex: RegExp;
    let Identifier: RegExp;
    let PossibleComment: RegExp;
    let Punctuation: RegExp;
    let Quote: RegExp;
    let Whitespace: RegExp;
}
```

###``options.regExp.Token``: Object

Token RegExps must have the 'g' and 'y' flags set.

``Token.Operator`` is optional.  Without it, the parser treats punctuation as invalid characters.
```typescript
declare namespace options.regExp.Token {
    let StartComment: RegExp;
    let HexNumber: RegExp;
    let Identifier: RegExp;
    let IdentifierAfterAt: RegExp;
    let Number: RegExp;
    let Operator: RegExp|null;
    let Whitespace: RegExp;
}
```

###``options.commentHandlers``: Object

Override the parser's built-in parsing of comments.  

After a successful match on ``Token.StartComment``, the 
matching string is used to look up a parsing function.

The default comment object is
```javascript
comment = {
    '/*': Parser.prototype.starComment,
    '--': Parser.prototype.singleLineComment,
    '#': Parser.prototype.singleLineComment
}
```

## Example: Comparing Mysql and SQL-92 Operator and Comment Parsing
```javascript
const tokenizerFactory = require('mysql-tokenizer');

const tokenizeMysql = tokenizerFactory();

const tokensMysql = tokenizeMysql("SELECT tbl.x->'$', -1<=>1--2 FROM tbl;");

console.log(`Mysql: ${JSON.stringify(tokensMysql)}`);

/* Output:
Mysql: ["SELECT"," ","tbl",".","x","->","'$'",","," ","-","1","<=>","1","-","-","2"," ","FROM"," ","tbl",";"]
 */



const sql92Operators = require('sql92-operators')
const sql92RegExp = require('mysql-tokenizer/lib/regexp-sql92');
const tokenizeSQL92 = tokenizerFactory({
    operators: sql92Operators,
    regExp: sql92RegExp
});
const tokensSQL92 = tokenizeSQL92("SELECT tbl.x->'$', -1<=>1--2 FROM tbl;");
console.log(`SQL92: ${JSON.stringify(tokensSQL92)}`);
/* Output:
SQL92: ["SELECT"," ","tbl",".","x","-",">","'$'",","," ","-","1","<=",">","1","--2 FROM tbl;"]
 */
```

---
_Source: https://npm.io/package/mysql-tokenizer · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._