Robots.txt Packages

A specification compliant robots.txt parser with wildcard (*) matching support.

Awesome generator robots.txt

A webpack plugin to output a robots.txt file

Parse robot directives within HTML meta and/or HTTP headers.

Express middleware to serve and generate robots.txt

Parser for XML Sitemaps to be used with Robots.txt and web crawlers

Express middleware for generating your robots.txt

Express middleware for generating your robots.txt

Robots.txt bounce middleware for koa

Parser for robots.txt

Robots.txt file generator plugin for vuepress

A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.

Promise based parser for robots.txt files.

Middleware for Express to add a robots.txt

Higher order Next.js configuration for generating sitemap.xml and robots.txt

koa middleware to handle different robots.txt responses on different domains

Simple robots.txt file generation for express

A selector-based html snapshot tool using Puppeteer or PhantomJS that sources sitemap.xml, robots.txt, or arbitrary input

The grunt task for html-snapshots

Robots.txt generator done right.