Robots.txt | npm.io

Parser for XML Sitemaps to be used with Robots.txt and web crawlers

3.2.9 • Published 3 years ago

Parser for XML Sitemaps to be used with Robots.txt and web crawlers

5.0.0 • Published 1 year ago

Parser for XML Sitemaps to be used with Robots.txt and web crawlers

2.1.19 • Published 4 years ago

A reliable robots.txt generator for Astro projects, offering zero-config setup and Verified Bots support.

2.3.1 • Published 7 months ago

Generate a robots.txt for Astro

1.0.0 • Published 2 years ago

Block AI scrapers in the `robots.txt` for your Astro site

0.1.1 • Published 4 months ago

An Eleventy plugin to generate a robots.txt file for your static site

1.0.0 • Published 12 months ago

generate robots.txt based on env content for static websites

1.0.4 • Published 3 years ago

Robots.txt Packages