Crawler Packages

xvideos.com api implementation.

Website schema based crawler

Recursive and multi-threaded broken link checker

A small package to crawl a site and return a redirect template. This is helpful for migration from one to another website with different url schemes.

crawler parser

Yet another js-crawler, a highly customized js-crawler (https://github.com/antivanov/js-crawler) for advanced usage, featuring: 1. priority request queue; 1. opt-in retry when error happens; 2. submit requests manually; 3. oblivious (no trail) and timeout

A demo package for crawing the basic details of a website

The crawler on top of puppeteer library

Recursively read a directory, blazing fast.

linkedin scraper for 2020 website

Very straightforward, event driven web crawler. Features a flexible queue interface and a basic cache mechanism with extensible backend.

A webcrawler with optional screenshotting websites.

The scalable web crawling and scraping library for JavaScript. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.

Modular web scraper for Node.JS

Microcrawler Worker

A really small crawler.

A proxy crawler.

Simple and powerful crawler. It scraps content and collects links from websites using request or phantomjs. The whole magic and simplicity is behind configuration.

## Dom selection tool based on cheerio.js

JSON Driven Site Scraper (Scavater). It is useful when you want to extract some information from a site when the extraction process may not require any complex logic, only xpath patterns of the elements that contain the information are sufficient.