Crawler Packages

A library for efficiently walking a directory recursively

Stealth mode: Applies various techniques to make detection of headless puppeteer harder.

Yet another node torrent scraper based on x-ray. (Support iptorrents, torrentleech, torrent9, Yyggtorrent, ThePiratebay, torrentz2, 1337x, KickassTorrent, Rarbg, TorrentProject, Yts, Limetorrents, Eztv)

Node.js agent for Sqreen, please see https://www.sqreen.io/

Very straightforward, event driven web crawler. Features a flexible queue interface and a basic cache mechanism with extensible backend.

Crawler is a ready-to-use web spider that works with proxies, asynchrony, rate limit, configurable request pools, jQuery, and HTTP/2 support.

Analyzes license information for multiple node.js modules (package.json files) as part of your software project.

Easily create XML sitemaps for your website.

Apify API client for JavaScript

The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.

http client module with cheerio & iconv(-lite) & promise

The fastest directory crawler & globbing alternative to glob, fast-glob, & tiny-glob. Crawls 1m files in < 1s

Automatically extracts structured information from webpages

x-ray's crawler

JavaScript module detecting bots/crawlers/spiders via user-agent

A tiny node module to detect spiders/crawlers quickly and comes with optional middleware for ExpressJS

Create xml sitemaps from the command line.

Detect SEO Bot Crawler

Distributed web crawler powered by Headless Chrome

This is an ES6 adaptation of the original PHP library CrawlerDetect, this library will help you detect bots/crawlers/spiders vie the useragent.