# express-robots-txt

> Express middleware to serve and generate robots.txt

Latest version **1.0.0** (published 2021-08-21) · MIT license · 0 weekly downloads

## Install

```sh
npm install express-robots-txt
pnpm add express-robots-txt
yarn add express-robots-txt
bun add express-robots-txt
```

## Health

**Score 20/100 (F)** — status: abandoned.

Positive: esm support; no vulnerabilities.

Warnings: low downloads; no types.

Negative: abandoned; low maintenance score.

## Facts

| | |
|---|---|
| Version | 1.0.0 |
| Published | 2021-08-21 |
| First published | 2017-04-17 |
| Weekly downloads | 0 |
| License | MIT |
| TypeScript types | none |
| Module format | ESM + CommonJS |
| Dependencies | 0 |
| Unpacked size | 14.4 KB |
| Known vulnerabilities | 0 |
| Install scripts | no |
| GitHub stars | 16 |
| Author | modosc |
| Maintainers | modosc |
| Keywords | express, robots, robots.txt, robots |

## Links

- npm: https://www.npmjs.com/package/express-robots-txt
- Repository: https://github.com/modosc/express-robots-txt
- Issues: https://github.com/modosc/express-robots-txt/issues
- npm.io page: https://npm.io/package/express-robots-txt

## Recent versions

- 1.0.0 (latest) — 2021-08-21
- 0.5.0 — 2020-08-17
- 0.4.1 — 2019-07-14
- 0.4.0 — 2018-07-08
- 0.3.0 — 2018-06-04
- 0.2.0 — 2017-04-17

## README

# express-robots-txt [![npm version](https://badge.fury.io/js/express-robots-txt.svg)](https://badge.fury.io/js/express-robots-txt) ![Node.js CI](https://github.com/modosc/express-robots-txt/workflows/Node.js%20CI/badge.svg)

Express middleware for generating a robots.txt or responding with an existing file.
# Usage
```javascript
// es6 usage requires a version of node which supports esm_conditional_exports, see
// https://nodejs.org/api/esm.html#esm_conditional_exports
import express from 'express'
import robots from 'express-robots-txt'

// commonjs
const express = require('express')
const robots = require('express-robots-txt')

const app = express()
const port = 3000

app.use(robots({
  UserAgent: '*',
  Disallow: '/',
  CrawlDelay: '5',
  Sitemap: 'https://nowhere.com/sitemap.xml',
}))

app.listen(port)
```
## Using a file

```javascript
app.use(robots(__dirname + '/robots.txt'));
```

## Using an object

### Basic object

```javascript
app.use(robots({
  UserAgent: '*',
  Disallow: '/'
}))
```

#### Will produce:
```
User-agent: *
Disallow: /
```

### Crawl Delay
You can optionally pass a CrawlDelay in just like passing in Disallow

```javascript
app.use(robots({
  UserAgent: '*',
  Disallow: '/',
  CrawlDelay: '5'
}))
```

#### Will produce:
```
User-agent: *
Disallow: /
Crawl-delay: 5
```
### Sitemap
You can optionally pass a Sitemap in just like passing in CrawlDelay:


```javascript
app.use(robots({
  UserAgent: '*',
  Disallow: '/',
  CrawlDelay: '5',
  Sitemap: 'https://nowhere.com/sitemap.xml'
}))
```

#### Will produce:
```
User-agent: *
Disallow: /
Crawl-delay: 5
Sitemap: https://nowhere.com/sitemap.xml
```

You can also pass in an array of Sitemaps:

```javascript
app.use(robots({
  UserAgent: '*',
  Disallow: '/',
  CrawlDelay: '5',
  Sitemap: [
    'https://nowhere.com/sitemap.xml',
    'https://nowhere.com/sitemap2.xml'
  ]
}))
```

#### Will produce:
```
User-agent: *
Disallow: /
Crawl-delay: 5
Sitemap: https://nowhere.com/sitemap.xml
Sitemap: https://nowhere.com/sitemap2.xml
```

### Or an array of objects

```javascript
app.use(robots([
  {
    UserAgent: 'Googlebot',
    Disallow: '/no-google'
  },
  {
    UserAgent: 'Bingbot',
    Disallow: '/no-bing'
  }
]));
```

#### Will produce:
```
User-agent: Googlebot
Disallow: /no-google
User-agent: Bingbot
Disallow: /no-bing
```

### Or either property (UserAgent | Disallow) as an array

```javascript
app.use(robots([
  {
    UserAgent: [ 'Googlebot', 'Slurp' ],
    Disallow: [ '/no-google', '/no-yahoo' ]
  },
  {
    UserAgent: '*',
    Disallow: [ '/no-bots', '/still-no-bots' ]
  }
]));
```

#### Will produce:
```
User-agent: Googlebot
User-agent: Slurp
Disallow: /no-google
Disallow: /no-yahoo
User-agent: *
Disallow: /no-bots
Disallow: /still-no-bots
```

### Host
You can optionally pass a Host in just like passing in Disallow. Note that [some
crawlers](https://en.wikipedia.org/wiki/Robots_exclusion_standard#Host) may not
support this drective.

```javascript
app.use(robots({
  UserAgent: '*',
  Disallow: '/',
  CrawlDelay: '5',
  Host: 'foobar.com'
}))
```

#### Will produce:
```
User-agent: *
Disallow: /
Crawl-delay: 5
Host: foobar.com
```

## Thanks
Originally forked from [weo-edu/express-robots](https://github.com/weo-edu/express-robots).

---
_Source: https://npm.io/package/express-robots-txt · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._