Crawler Packages

crawl and archive a website and all its pages as plain HTML files organized in directories to match the URL structure of the website

A simple Node crawling application (CLI and lib) that collects links URL from a root one

Tool to crawl events, leagues and statistics from WBSC based websites.

Parser for robots.txt

A library for efficiently walking a directory recursively

A Simple ontology database built from a list of ontologies, with auto-download abilities

Playwright crawler core

IT Events Crawler of China

Recursive and multi-threaded broken link checker

Access Google Play by logging in and making requests as an Android device!

A library for crawling a filesystem tree, based on Effect-ts

A library for efficiently walking a directory recursively

User Agent

基于Node.js的网络爬虫

基于Node.js的网络爬虫(轻量版)，该版本不支持动态页面爬取。

linkedin scraper support fetching company posts

Crawler is a web spider written with Nodejs. It gives you the full power of jQuery on the server to parse a big number of pages as they are downloaded, asynchronously

A Bing command line dictionary, which obtains the query results of bing dictionary by crawler.

Get Aliexpress product details as a json reponse including feedbacks, variants, description, images, etc.,