A library for efficiently walking a directory recursively
Stealth mode: Applies various techniques to make detection of headless puppeteer harder.
Yet another node torrent scraper based on x-ray. (Support iptorrents, torrentleech, torrent9, Yyggtorrent, ThePiratebay, torrentz2, 1337x, KickassTorrent, Rarbg, TorrentProject, Yts, Limetorrents, Eztv)
Node.js agent for Sqreen, please see https://www.sqreen.io/
Very straightforward, event driven web crawler. Features a flexible queue interface and a basic cache mechanism with extensible backend.
Crawler is a web spider written with Nodejs. It gives you the full power of jQuery on the server to parse a big number of pages as they are downloaded, asynchronously
Analyzes license information for multiple node.js modules (package.json files) as part of your software project.
Easily create XML sitemaps for your website.
http client module with cheerio & iconv(-lite) & promise
The fastest directory crawler & globbing alternative to glob, fast-glob, & tiny-glob. Crawls 1m files in < 1s
Automatically extracts structured information from webpages
A tiny node module to detect spiders/crawlers quickly and comes with optional middleware for ExpressJS
Create xml sitemaps from the command line.
Detect SEO Bot Crawler
Distributed web crawler powered by Headless Chrome
This is an ES6 adaptation of the original PHP library CrawlerDetect, this library will help you detect bots/crawlers/spiders vie the useragent.