# exclusion

> A simple, RFC-compliant robots.txt parser

Latest version **0.2.0** (published 2023-10-12) · MIT license · 0 weekly downloads

## Install

```sh
npm install exclusion
pnpm add exclusion
yarn add exclusion
bun add exclusion
```

## Health

**Score 30/100 (F)** — status: abandoned.

Positive: has types; esm support; no vulnerabilities; high quality score.

Warnings: low downloads; pre 1.0.

Negative: abandoned; low maintenance score.

## Facts

| | |
|---|---|
| Version | 0.2.0 |
| Published | 2023-10-12 |
| First published | 2022-01-23 |
| Weekly downloads | 0 |
| License | MIT |
| TypeScript types | bundled |
| Module format | ESM + CommonJS |
| Node | >=18.18 |
| Dependencies | 1 |
| Unpacked size | 54.7 KB |
| Known vulnerabilities | 0 |
| Install scripts | no |
| GitHub stars | 0 |
| Author | Russell Steadman |
| Maintainers | russellsteadman |
| Keywords | robotstxt, parser, bot, scrape, spider, crawl, robot, robots, exclusion |

## Links

- npm: https://www.npmjs.com/package/exclusion
- Repository: https://github.com/russellsteadman/netscrape
- Homepage: https://github.com/russellsteadman/netscrape/tree/main/packages/exclusion#readme
- Issues: https://github.com/russellsteadman/netscrape/issues
- npm.io page: https://npm.io/package/exclusion

## Dependencies (1)

- [@types/node](https://npm.io/package/@types/node.md) ^20.8.4

## Alternatives

- [babylon](https://npm.io/package/babylon.md) — 5.1M weekly downloads
- [csscolorparser](https://npm.io/package/csscolorparser.md) — 3.7M weekly downloads
- [expr-eval-fork](https://npm.io/package/expr-eval-fork.md) — 1.5M weekly downloads
- [@leeoniya/ufuzzy](https://npm.io/package/@leeoniya/ufuzzy.md) — 247.7K weekly downloads
- [xml-parser](https://npm.io/package/xml-parser.md) — 78.4K weekly downloads

## Recent versions

- 0.2.0 (latest) — 2023-10-12
- 0.1.1 — 2022-01-30
- 0.1.0 — 2022-01-29
- 0.1.0-alpha0 — 2022-01-23

## README

# Exclusion

A simple and RFC 9309 compliant `robots.txt` parser for Node.js.

![](https://img.shields.io/librariesio/release/npm/exclusion?style=flat-square)
![](https://img.shields.io/npm/l/exclusion?style=flat-square) ![](https://img.shields.io/snyk/vulnerabilities/npm/exclusion?style=flat-square)

## Get Started

```sh
npm install --save exclusion
```

```sh
yarn add exclusion
```

This is the `robots.txt` parser used by the
[`netscrape`](https://www.npmjs.com/packages/netscrape) package. You may want to
check out
[NetScrape](https://github.com/russellsteadman/netscrape/tree/main/packages/bot#readme)
for most simple bot use-cases.

## Usage

```js
const { RobotsTxt } = require('exclusion');

// Fetch a robots.txt file...

// Pass a robots.txt string to initialize the parser
const robotsTxt = new RobotsTxt(myRobotsTxtString);

// Check a path
robotsTxt.isPathAllowed('/certain/path', 'MyUserAgent');
// returns boolean

// Check a delay
robotsTxt.getDelay('MyUserAgent');
// returns delay in milliseconds or undefined
```

## License

MIT (C) 2023 [Russell Steadman](https://github.com/russellsteadman). See LICENSE file. Visit [Google
deps.dev](https://deps.dev/npm/exclusion) for dependency license information.

---
_Source: https://npm.io/package/exclusion · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._