robots-generator v2.3.6
Robots Generator
Produces a simple, valid robots.txt to be parsed by web crawlers. Adheres to the specification provided by Google, however currently only supports one User-Agent rule. Requires Node 4+. Installed through NPM with:
npm install robots-generator --save-dev
Simply require the module and execute it with an optional array of configuration.
- User-Agent: A means of identifying a specific crawler or set of crawlers.
- Allow: An array of directories that a crawler is allowed to access.
- Disallow: An array of directories that a crawler is not allowed to access.
- Sitemap: Your website's sitemap URL.
var robots = require('robots-generator');
robots({
useragent: '*',
allow: ['folder1/', 'folder2/'],
disallow: ['cgi-bin/'],
sitemap: 'http://haydenbleasel.com/sitemap.xml'
}, function (error, robots) {
// Join ('\n') and write this to a file
console.log(error, robots);
});
If you need an ES5 build for legacy purposes, just require the ES5 file:
var robots = require('robots-generator/es5');
Outputs the following file:
User-agent: *
Allow: /folder1/
Allow: /folder2/
Disallow: cgi-bin/
Sitemap: http://haydenbleasel.com/sitemap.xml
To build the ES5 version:
npm install -g babel-cli
babel --presets es2015 index.js --out-file es5.js
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
10 years ago
10 years ago
10 years ago
10 years ago
10 years ago
10 years ago
10 years ago
10 years ago
10 years ago
10 years ago
10 years ago
10 years ago