Robots.txt Packages

It uses the user-agents.org xml file for detecting bots.

A plugin for Robots Exclusion Standard for webcheck

Lightweight robots.txt parsing component without any external dependencies for Node.js.

highlight.js syntax definition for Google's robot.txt format

robots.txt utilities

Express middleware to handle different robots.txt responses on different domains

XML sitemap and robots.txt generator plugin for Node Reactor CMS.

Parses robots.txt files to provide meaningful, useful output as well as reporting syntax errors.

Parser for XML Sitemaps to be used with Robots.txt and web crawlers

Creates robots.txt page

robots.txt agent with cache.

A proper robots.txt parser and combinator that works with eulalie

Install robots.txt files on every site on a staging server, taking into account both PHP and node/stagecoach sites

A sitemapping tool for crawlers

robots.txt middleware for Kerouac.

Deney all robots express middleware

Rules related to checking for any SEO issues on the page given

Check user agents against robots.txt files

robots.txt generation for webpack 3

Lightweight robots.txt parsing component without any external dependencies for Node.js.