# textlex

> Lexical analyser using TextMate-style grammars

Latest version **0.6.3** (published 2017-01-23) · MIT license · 17 weekly downloads

## Install

```sh
npm install textlex
pnpm add textlex
yarn add textlex
bun add textlex
```

Provides the command `textlex`.

## Health

**Score 15/100 (F)** — status: abandoned.

Positive: no vulnerabilities.

Warnings: low downloads; no types; no esm support; pre 1.0.

Negative: abandoned; low maintenance score.

## Facts

| | |
|---|---|
| Version | 0.6.3 |
| Published | 2017-01-23 |
| First published | 2015-04-16 |
| Weekly downloads | 17 |
| License | MIT |
| TypeScript types | none |
| Module format | CommonJS |
| Dependencies | 6 |
| Known vulnerabilities | 0 |
| Install scripts | no |
| GitHub stars | 2 |
| Maintainers | jonruttan |

## Links

- npm: https://www.npmjs.com/package/textlex
- Repository: https://github.com/jonruttan/textlex
- Issues: https://github.com/jonruttan/textlex/issues
- npm.io page: https://npm.io/package/textlex

## Dependencies (6)

- [yargs](https://npm.io/package/yargs.md) ^6.6.0
- [season](https://npm.io/package/season.md) ^5.4.1
- [fs-plus](https://npm.io/package/fs-plus.md) ^2.9.3
- [first-mate](https://npm.io/package/first-mate.md) ^6.2.1
- [underscore-plus](https://npm.io/package/underscore-plus.md) ^1.6.6
- [first-mate-select-grammar](https://npm.io/package/first-mate-select-grammar.md) ^1.0.1

## Recent versions

- 0.6.3 (latest) — 2017-01-23
- 0.6.2 — 2017-01-20
- 0.6.1 — 2017-01-19
- 0.6.0 — 2017-01-19
- 0.5.1 — 2015-05-04
- 0.5.0 — 2015-05-04
- 0.4.1 — 2015-05-03
- 0.4.0 — 2015-04-19
- 0.3.0 — 2015-04-16
- 0.2.0 — 2015-04-16

## README

A lexical analyser using TextMate-style grammars: reads in code, returns an array containing an array of tokens for each line.

Based on code converted from the [Highlights](https://github.com/atom/highlights) project.

[![Build Status](https://travis-ci.org/jonruttan/textlex.svg)](https://travis-ci.org/jonruttan/textlex)

### Installing

```sh
npm install textlex
```

### Using

Run `textlex -h` for full details about the supported options.

To convert a source file to tokenized JSON run the following:

```sh
textlex source.js -o tokens.json
```

Now you have a `file.json` file with an array of line token arrays.

#### Using in code

To convert a source string to tokens use the following code:

```js
var textlex = require('textlex');
var textlexer = new textlex();
var linetokens = textlexer.lexSync({
  fileContents: "var hello = 'world';",
  scopeName: 'source.js',
});

console.log(JSON.stringify(linetokens, null, 2));
```

Outputs:

```js
[
  [
    {
      "value": "var",
      "scopes": [
        "source.js",
        "storage.modifier.js"
      ]
    },
    {
      "value": " hello ",
      "scopes": [
        "source.js"
      ]
    },
    {
      "value": "=",
      "scopes": [
        "source.js",
        "keyword.operator.js"
      ]
    },
    {
      "value": " ",
      "scopes": [
        "source.js"
      ]
    },
    {
      "value": "'",
      "scopes": [
        "source.js",
        "string.quoted.single.js",
        "punctuation.definition.string.begin.js"
      ]
    },
    {
      "value": "world",
      "scopes": [
        "source.js",
        "string.quoted.single.js"
      ]
    },
    {
      "value": "'",
      "scopes": [
        "source.js",
        "string.quoted.single.js",
        "punctuation.definition.string.end.js"
      ]
    },
    {
      "value": ";",
      "scopes": [
        "source.js",
        "punctuation.terminator.statement.js"
      ]
    }
  ]
]
```

### Loading Grammars From Modules

textlex exposes the method `requireGrammarsSync`, for loading grammars from
npm modules. The usage is as follows:

```bash
npm install atom-language-clojure
```

```coffee
textlex = require 'textlex'
textlexer = new textlex()
textlexer.requireGrammarsSync
  modulePath: require.resolve('atom-language-clojure/package.json')
```

### Developing

* Clone this repository `git clone https://github.com/jonruttan/textlex`
* Update the submodules by running `git submodule update --init --recursive`
* Run `npm install` to install the dependencies, compile the CoffeeScript, and
  build the grammars
* Run `npm test` to run the specs

:green_heart: Pull requests are greatly appreciated and welcomed.

---
_Source: https://npm.io/package/textlex · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._
