Robots-exclusion-standard | npm.io

A specification compliant robots.txt parser with wildcard (*) matching support.

robots.txt parser user-agent scraper spider bot robots-exclusion-standard

3.0.1 • Published 1 year ago

Lightweight robots.txt parsing component without any external dependencies for Node.js.

robots.txt robots robots-parser robots-exclusion-standard javascript js node-js robots-txt crawler crawlers

0.0.5-dev • Published 2 years ago

Lightweight robots.txt parsing component without any external dependencies for Node.js.

robots.txt robots robots-parser robots-exclusion-standard javascript js node-js robots-txt crawler crawlers

0.0.2 • Published 3 years ago