Robots-txt | npm.io

Awesome generator robots.txt

robotstxt robots.txt generate robots-txt user-agent police allow disallow crawl-delay robot

8.0.3 • Published 4 years ago

A webpack plugin to output a robots.txt file

robots.txt robots robotstxt generate robots-txt user-agent webpack webpack-plugin plugin

8.0.0 • Published 2 years ago

Promise based parser for robots.txt files.

crawl web robots.txt seo robotstxt robots robots-txt sitemap

3.2.0 • Published 4 years ago

Lightweight robots.txt parsing component without any external dependencies for Node.js.

robots.txt robots robots-parser robots-exclusion-standard javascript js node-js robots-txt crawler crawlers

0.0.5-dev • Published 2 years ago

Lightweight robots.txt parsing component without any external dependencies for Node.js.

robots.txt robots robots-parser robots-exclusion-standard javascript js node-js robots-txt crawler crawlers

0.0.2 • Published 3 years ago

Generate a robots.txt file for your Astro site

astro-integration astro-component withastro astro robots robots.txt robots-txt

0.1.1 • Published 6 days ago

The lightweight robots.txt generator makes it simple to create compliant robots.txt files specifically for Astro integrations, with zero-config support, an intuitive JSDoc API, and always up-to-date verified bots user-agents.

withastro astro-integration robots.txt generator seo assistant verified bots verified robots robots robots-txt astro-robots

2.0.1 • Published 2 months ago

Generate a robots.txt for Astro

astro-integration astro astro-component seo robots.txt robotstxt robots-txt

1.0.0 • Published 8 months ago

Robots-txt Packages

generate-robotstxt

robotstxt-webpack-plugin

parse-robots

robots-txt-component

@ibragim64/robots-txt-component

@itsmatteomanf/astro-robots-txt

astro-robots

astro-robots-txt