# robots-generator

> Robots.txt generator for Node.js

Latest version **2.3.6** (published 2016-07-26) · ISC license · 0 weekly downloads

## Install

```sh
npm install robots-generator
pnpm add robots-generator
yarn add robots-generator
bun add robots-generator
```

## Health

**Score 15/100 (F)** — status: abandoned.

Positive: no vulnerabilities.

Warnings: low downloads; no types; no esm support.

Negative: abandoned; low maintenance score.

## Facts

| | |
|---|---|
| Version | 2.3.6 |
| Published | 2016-07-26 |
| First published | 2014-05-15 |
| Weekly downloads | 0 |
| License | ISC |
| TypeScript types | none |
| Module format | CommonJS |
| Dependencies | 4 |
| Known vulnerabilities | 0 |
| Install scripts | no |
| Author | Hayden Bleasel |
| Maintainers | haydenbleasel |
| Keywords | robots, txt, generator, node, gulpfriendly, gulpplugin |

## Links

- npm: https://www.npmjs.com/package/robots-generator
- Repository: https://github.com/haydenbleasel/robots-generator
- Issues: https://github.com/haydenbleasel/robots-generator/issues
- npm.io page: https://npm.io/package/robots-generator

## Dependencies (4)

- [vinyl](https://npm.io/package/vinyl.md) *
- [cheerio](https://npm.io/package/cheerio.md) *
- [through2](https://npm.io/package/through2.md) *
- [underscore](https://npm.io/package/underscore.md) *

## Alternatives

- [@expo/fingerprint](https://npm.io/package/@expo/fingerprint.md) — 6.2M weekly downloads
- [@azure/monitor-opentelemetry-exporter](https://npm.io/package/@azure/monitor-opentelemetry-exporter.md) — 850.0K weekly downloads
- [@azure/monitor-opentelemetry](https://npm.io/package/@azure/monitor-opentelemetry.md) — 624.0K weekly downloads
- [@posthog/ai](https://npm.io/package/@posthog/ai.md) — 423.3K weekly downloads
- [fakefilter](https://npm.io/package/fakefilter.md) — 63.9K weekly downloads

## Recent versions

- 2.3.6 (latest) — 2016-07-26
- 2.3.5 — 2015-11-28
- 2.3.4 — 2015-11-24
- 2.3.3 — 2015-11-24
- 2.3.2 — 2015-11-22
- 2.3.1 — 2015-11-22
- 2.3.0 — 2015-11-22
- 2.2.1 — 2015-11-22
- 2.2.0 — 2015-11-22
- 2.1.1 — 2015-11-21
- 2.1.0 — 2015-11-21
- 2.0.3 — 2015-11-21
- 2.0.2 — 2015-11-21
- 2.0.1 — 2015-11-21
- 2.0.0 — 2015-11-21
- … 18 more at https://npm.io/package/robots-generator/versions

## README

# Robots Generator [![Build Status](https://travis-ci.org/haydenbleasel/robots-generator.svg?branch=master)](https://travis-ci.org/haydenbleasel/robots-generator)

Produces a simple, valid robots.txt to be parsed by web crawlers. Adheres to the [specification](https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt) provided by Google, however currently only supports one User-Agent rule. Requires Node 4+. Installed through NPM with:

```shell
npm install robots-generator --save-dev
```

Simply require the module and execute it with an optional array of configuration.

- User-Agent: A means of identifying a specific crawler or set of crawlers.
- Allow: An array of directories that a crawler is allowed to access.
- Disallow: An array of directories that a crawler is not allowed to access.
- Sitemap: Your website's sitemap URL.

```js
var robots = require('robots-generator');

robots({
    useragent: '*',
    allow: ['folder1/', 'folder2/'],
    disallow: ['cgi-bin/'],
    sitemap: 'http://haydenbleasel.com/sitemap.xml'
}, function (error, robots) {
    // Join ('\n') and write this to a file
    console.log(error, robots);
});
```

If you need an ES5 build for legacy purposes, just require the ES5 file:

```js
var robots = require('robots-generator/es5');
```

Outputs the following file:

```
User-agent: *
Allow: /folder1/
Allow: /folder2/
Disallow: cgi-bin/
Sitemap: http://haydenbleasel.com/sitemap.xml
```

To build the ES5 version:

```sh
npm install -g babel-cli
babel --presets es2015 index.js --out-file es5.js
```

---
_Source: https://npm.io/package/robots-generator · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._