Spider Packages

This is an API that parses www.cc98.org into useful informations in json.

DCrawler is a distribited web spider written in Nodejs and queued with Mongodb. It gives you the full power of jQuery to parse big pages as they are downloaded, asynchronously. Simplifying distributed crawler!

An extremely lightweight HTTP request client for the command-line. Supports: http, https, proxy, redirects, cookies, content-encoding, multipart/form-data, multi-threading, recursive website crawling and mirroring.

1. 爬虫单机版本，redis作为队列存储爬虫请求模板，支持mysql和mongodb存储 2. 用法及例子：[spider-test](https://github.com/jiang19210/spider-test)

all Marvel comic book characters

Get Aliexpress product details as a json reponse including feedbacks, variants, description, images, etc.,

Cylon module for RollingSpider

xvideos.com api implementation.

Environment for Goose Parser which allows to run it using JsDOM

The Next-Gen Programming Language for the Web

> Leetcode solutions spider.

Grabbing metadata from webpages

A utility library to make downloading & extracting specific content from a URL easy

Generic web crawler powered by Node.js

Am I bot or not?

Comic Spider

Search for anything on web.

Ядро веб-краулера для обхода сайта

A node.js wrapper for the PageMunch link unfurling API

Scrap the web asynchronously in live, reusing Node.js, all in one file, with a few lines!