# robots-txt-parse

> Streaming parser for robots.txt files

Latest version **2.0.1** (published 2021-02-23) · MIT license · 0 weekly downloads

## Install

```sh
npm install robots-txt-parse
pnpm add robots-txt-parse
yarn add robots-txt-parse
bun add robots-txt-parse
```

## Health

**Score 15/100 (F)** — status: abandoned.

Positive: no vulnerabilities.

Warnings: low downloads; no types; no esm support.

Negative: abandoned; low maintenance score.

## Facts

| | |
|---|---|
| Version | 2.0.1 |
| Published | 2021-02-23 |
| First published | 2014-10-14 |
| Weekly downloads | 0 |
| License | MIT |
| TypeScript types | none |
| Module format | CommonJS |
| Dependencies | 1 |
| Unpacked size | 11.9 KB |
| Known vulnerabilities | 0 |
| Install scripts | no |
| GitHub stars | 4 |
| Author | Jan Potoms |
| Maintainers | woorank-admin, janpotoms, ndemoor, woorank-ci |

## Links

- npm: https://www.npmjs.com/package/robots-txt-parse
- Repository: https://github.com/Woorank/robots-txt-parse
- Homepage: https://github.com/Woorank/robots-txt-parse#readme
- Issues: https://github.com/Woorank/robots-txt-parse/issues
- npm.io page: https://npm.io/package/robots-txt-parse

## Dependencies (1)

- [split2](https://npm.io/package/split2.md) ^3.2.2

## Recent versions

- 2.0.1 (latest) — 2021-02-23
- 2.0.0 — 2021-02-23
- 1.0.1 — 2017-12-04
- 1.0.0 — 2017-11-29
- 0.0.4 — 2015-12-04
- 0.0.3 — 2015-03-31
- 0.0.2 — 2014-10-17
- 0.0.1 — 2014-10-14

## README

# robots-txt-parse [![Build Status](https://travis-ci.org/Woorank/robots-txt-parse.svg)](https://travis-ci.org/Woorank/robots-txt-parse)

Streaming robots.txt parser

## usage

```js
const parse = require('robots-txt-parse');
const fs = require('fs');

const input = fs.createReadStream(__dirname + '/robots.txt');
const result = await parse(input);
```
assuming this file
```
user-agent: *
user-agent: googlebot
disallow: /

user-agent: twitterbot
disallow: /
allow: /twitter

user-agent: mozilla
disallow: /path
noindex: /path

Sitemap: http://www.example.com/sitemap.xml
```
produces following output
```json
{
  "groups": [{
    "agents": [ "*", "googlebot" ],
    "rules": [
      { "rule": "disallow", "path": "/" }
    ]
  }, {
    "agents": [ "twitterbot" ],
    "rules": [
      { "rule": "disallow", "path": "/" },
      { "rule": "allow", "path": "/twitter" }
    ]
  }, {
    "agents": [ "mozilla" ],
    "rules": [
      { "rule": "disallow", "path": "/path" },
      { "rule": "noindex", "path": "/path" }
    ]
  }],
  "extensions": [
    { "extension": "sitemap", "value": "http://www.example.com/sitemap.xml" }
  ]
}
```

---
_Source: https://npm.io/package/robots-txt-parse · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._