# generate-robotstxt

> Awesome generator robots.txt

Latest version **8.0.3** (published 2020-05-12) · MIT license · 0 weekly downloads

## Install

```sh
npm install generate-robotstxt
pnpm add generate-robotstxt
yarn add generate-robotstxt
bun add generate-robotstxt
```

Provides the command `generate-robotstxt`.

## Health

**Score 25/100 (F)** — status: abandoned.

Positive: no vulnerabilities.

Warnings: low downloads; no types; no esm support.

Negative: abandoned.

## Facts

| | |
|---|---|
| Version | 8.0.3 |
| Published | 2020-05-12 |
| First published | 2014-11-07 |
| Weekly downloads | 0 |
| License | MIT |
| TypeScript types | none |
| Module format | CommonJS |
| Node | >= 10.13.0 |
| Dependencies | 6 |
| Unpacked size | 18.1 KB |
| Known vulnerabilities | 0 |
| Install scripts | no |
| GitHub stars | 65 |
| Author | itgalaxy |
| Maintainers | cap-bernardito, evilebottnawi, itgalaxy-owner |
| Keywords | robotstxt, robots.txt, generate, robots-txt, user-agent, police, allow, disallow, crawl-delay, robot, robots |

## Links

- npm: https://www.npmjs.com/package/generate-robotstxt
- Repository: https://github.com/itgalaxy/generate-robotstxt
- Homepage: https://github.com/itgalaxy/generate-robotstxt#readme
- Issues: https://github.com/itgalaxy/generate-robotstxt/issues
- npm.io page: https://npm.io/package/generate-robotstxt

## Dependencies (6)

- [meow](https://npm.io/package/meow.md) ^7.0.1
- [fs-extra](https://npm.io/package/fs-extra.md) ^9.0.0
- [ip-regex](https://npm.io/package/ip-regex.md) ^4.1.0
- [cosmiconfig](https://npm.io/package/cosmiconfig.md) ^6.0.0
- [resolve-from](https://npm.io/package/resolve-from.md) ^5.0.0
- [is-absolute-url](https://npm.io/package/is-absolute-url.md) ^3.0.3

## Recent versions

- 8.0.3 (latest) — 2020-05-12
- 8.0.2 — 2020-03-24
- 8.0.1 — 2020-02-19
- 8.0.0 — 2019-11-05
- 7.1.0 — 2019-07-03
- 7.0.0 — 2019-07-03
- 6.0.1 — 2019-03-26
- 6.0.0 — 2019-01-09
- 5.0.7 — 2018-05-22
- 5.0.6 — 2018-05-05
- 5.0.5 — 2018-03-28
- 5.0.4 — 2018-02-19
- 5.0.3 — 2018-01-16
- 5.0.2 — 2017-12-12
- 5.0.1 — 2017-11-28
- … 13 more at https://npm.io/package/generate-robotstxt/versions

## README

# generate-robotstxt

[![NPM version](https://img.shields.io/npm/v/generate-robotstxt.svg)](https://www.npmjs.org/package/generate-robotstxt)
[![Travis Build Status](https://img.shields.io/travis/itgalaxy/generate-robotstxt/master.svg?label=build)](https://travis-ci.org/itgalaxy/generate-robotstxt)
[![dependencies Status](https://david-dm.org/itgalaxy/generate-robotstxt/status.svg)](https://david-dm.org/itgalaxy/generate-robotstxt)
[![devDependencies Status](https://david-dm.org/itgalaxy/generate-robotstxt/dev-status.svg)](https://david-dm.org/itgalaxy/generate-robotstxt?type=dev)

Awesome generator robots.txt.

## Installation

```shell
npm install --save-dev generate-robotstxt
```

## Usage

```js
import robotstxt from "generate-robotstxt";

robotstxt({
  policy: [
    {
      userAgent: "Googlebot",
      allow: "/",
      disallow: "/search",
      crawlDelay: 2,
    },
    {
      userAgent: "OtherBot",
      allow: ["/allow-for-all-bots", "/allow-only-for-other-bot"],
      disallow: ["/admin", "/login"],
      crawlDelay: 2,
    },
    {
      userAgent: "*",
      allow: "/",
      disallow: "/search",
      crawlDelay: 10,
      cleanParam: "ref /articles/",
    },
  ],
  sitemap: "http://example.com/sitemap.xml",
  host: "http://example.com",
})
  .then((content) => {
    console.log(content);

    return content;
  })
  .catch((error) => {
    throw error;
  });
```

## File based configuration

**robots-txt.config.js**

```js
module.exports = {
  policy: [
    {
      userAgent: "Googlebot",
      allow: "/",
      disallow: ["/search"],
      crawlDelay: 2,
    },
    {
      userAgent: "OtherBot",
      allow: ["/allow-for-all-bots", "/allow-only-for-other-bot"],
      disallow: ["/admin", "/login"],
      crawlDelay: 2,
    },
    {
      userAgent: "*",
      allow: "/",
      disallow: "/search",
      crawlDelay: 10,
      cleanParam: "ref /articles/",
    },
  ],
  sitemap: "http://example.com/sitemap.xml",
  host: "http://example.com",
};
```

## CLI

```shell
Awesome generator robots.txt

  Usage generate-robotstxt [options] <dest>

  Options:
     --config  Path to a specific configuration file.
```

## Contribution

Feel free to push your code if you agree with publishing under the MIT license.

## [Changelog](CHANGELOG.md)

## [License](LICENSE)

---
_Source: https://npm.io/package/generate-robotstxt · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._