Robots-txt | npm.io

Awesome generator robots.txt

robotstxt robots.txt generate robots-txt user-agent police allow disallow crawl-delay robot

8.0.3 • Published 5 years ago

A webpack plugin to output a robots.txt file

robots.txt robots robotstxt generate robots-txt user-agent webpack webpack-plugin plugin

8.0.0 • Published 3 years ago

Promise based parser for robots.txt files.

crawl web robots.txt seo robotstxt robots robots-txt sitemap

3.2.0 • Published 5 years ago

Lightweight robots.txt parsing component without any external dependencies for Node.js.

robots.txt robots robots-parser robots-exclusion-standard javascript js node-js robots-txt crawler crawlers

0.0.5-dev • Published 3 years ago

Lightweight robots.txt parsing component without any external dependencies for Node.js.

robots.txt robots robots-parser robots-exclusion-standard javascript js node-js robots-txt crawler crawlers

0.0.2 • Published 5 years ago

Generate a robots.txt file for your Astro site

astro-integration astro-component withastro astro robots robots.txt robots-txt

0.1.2 • Published 1 year ago

A really lightweight and simple robots-txt parser.

www robots-txt security parser world-wide-web web website

1.0.0 • Published 9 months ago

Generate a robots.txt for Astro

astro-integration astro astro-component seo robots.txt robotstxt robots-txt

1.0.0 • Published 2 years ago

Static Site Generator

static-site-generator seo seo-friendly html-generator nodejs nodejs-tool pwa progressive-web-app json-ld sitemap-generator

1.0.12 • Published 7 months ago