Crawler Packages

A crawler

An easiest crawling and scraping module for NestJS

Webpage crawler for qualweb

A crawler

A crawler based on Phantom. Allows discovery of dynamic content and supports custom scrapers.

Scrape data from any webpage.

A simple node spider

Crawl a site to generate a backstopjs config

Provides a production server which help bots to index a react page

crawler for single page applications

Retrieve data from different websites using html elements to gather the information you need.

Web service and CLI tool for SEO site audit: crawl site, lighthouse all pages, view public reports in browser. Also output to console, json, csv.

DCrawler is a distribited web spider written in Nodejs and queued with Mongodb. It gives you the full power of jQuery to parse big pages as they are downloaded, asynchronously. Simplifying distributed crawler!

[](https://www.npmjs.com/package/recrawl-sync) [](https://github.com/aleclarson/recrawl/actions/workflows/release.yml)

An extremely lightweight HTTP request client for the command-line. Supports: http, https, proxy, redirects, cookies, content-encoding, multipart/form-data, multi-threading, recursive website crawling and mirroring.

Movie crawler for y80s.com

A page scraping DSL for extracting structured information from unstructured XHTML, built on Node.js and jQuery.

Enhanced request library with some specific cases like win1251 encoding and encoding support.

Get Aliexpress product details as a json reponse including feedbacks, variants, description, images, etc.,

crawler parser