Crawler Packages

A command-line tool to crawl websites using puppeteer.

Crawls/scrapes a website for og data, so Facebook will show current images and descriptions when sharing.

A Node.js Web Spider

Parkour the web like a yamakazi

A spider framework has a Koa like APIs, written by Typescript.

web crawler

A robots.txt reader, parser and matcher.

A fast crawler cli with pyppteer, this crawler can crawl SPA(single page application)

node-readability的cheerio版。支持GBK、GB2312等编码的网页抓取和解析。Turning any web page into a clean view.

node.js crawler

The spider to grab the future of the internet using nodejs, defind patterns, get data.

Web Download CLI

Simple web-crawler for nodejs

Wrapper around npm-license-crawler to download and export text of licenses in JSON format

基于nodejs平台下使用的Scrapy模型框架组件

Abstract layer for managing big sets of files

A qingcloud docs broken links checker

An intelligent proxy and request manager.


command line interface to crawl a rippled network