express-robots-middleware v1.0.1

express-robots-middleware

Middleware for generating robots.txt in Express

Installation

npm i express-robots-middleware --save

Features

• Multiple User Agents
• Multiple Allow/Disallow
• Crawl-delay argument

Usage

Simple initialize and assign the middleware to a route and pass in your robots.txt options see mock.js for a working example.

const robotsMiddleware = expressRobotsMiddleware({
  UserAgent: '*',
  Disallow: ['/private'],
  Allow: '/',
  CrawlDelay: '5'
}]);

app.get('/robots.txt', robotsMiddleware);

License

MIT