bot-detector
It uses the user-agents.org xml file for detecting bots.
It uses the user-agents.org xml file for detecting bots.
Lightweight robots.txt parsing component without any external dependencies for Node.js.
A plugin for Robots Exclusion Standard for webcheck
robots.txt utilities
Express middleware to handle different robots.txt responses on different domains
highlight.js syntax definition for Google's robot.txt format
XML sitemap and robots.txt generator plugin for Node Reactor CMS.
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
Parses robots.txt files to provide meaningful, useful output as well as reporting syntax errors.
Creates robots.txt page
Install robots.txt files on every site on a staging server, taking into account both PHP and node/stagecoach sites
robots.txt agent with cache.
A proper robots.txt parser and combinator that works with eulalie
A sitemapping tool for crawlers
robots.txt generation for webpack 3
Check user agents against robots.txt files
Deney all robots express middleware
Rules related to checking for any SEO issues on the page given
robots.txt middleware for Kerouac.
Lightweight robots.txt parsing component without any external dependencies for Node.js.