Robots-parser | npm.io

Lightweight robots.txt parsing component without any external dependencies for Node.js.

robots.txt robots robots-parser robots-exclusion-standard javascript js node-js robots-txt crawler crawlers

0.0.5-dev • Published 2 years ago

Lightweight robots.txt parsing component without any external dependencies for Node.js.

robots.txt robots robots-parser robots-exclusion-standard javascript js node-js robots-txt crawler crawlers

0.0.2 • Published 3 years ago