World-wide-web | npm.io

A really lightweight and simple robots-txt parser.

www robots-txt security parser world-wide-web web website

1.0.0 • Published 10 months ago

A simple crawler made in JavaScript for Node.

crawl crawler crawled crawling node web browser both ky got

1.0.1 • Published 11 months ago