Spider Packages

skyeye,天眼计划,nodejs爬虫

Multi environment web page parser

Follows and collects breadcrumbs accross the web

spider framework

Aimer for single page application.

The spider's sharp ax.

Aragog web scraping framework client

Crawl the web breadth-first from a seed url, statefully

Checks for mixed content a.k.a. HTTP references on HTTPS, very alpha.

Package for Play40 backend

A powerful crawler support strategy to different url, the crawler can traverse all web page in a site recursively with certain deep. of course, you alse can do not use recursive crawel.

nodejs web crawl tool

A crawler framework, just my experience.

百度图片爬虫小工具 By Node.js.

dht spider

A user agent parser that determines whether a request is from a robot, crawler, spider, etc.

Web crawler for Node and browsers

Web scraper with a jQuery like wrapper

Web crawler wrapper around puppeteer module to simply the crawling on ajax/java script enabled pages.

A Node.js based library for controlling a Parrot Rolling Spider drone