Crawler Packages

simple crawler

Magnetic is a tool that makes it easy to fetch web pages that may have dynamic content and render them to static HTML.

爬虫

Crawl websites for accessibility issues from the command line.

A JavaScript library that allows for the quick transformation of DOM documents into useful formats.

crawls mysql database and creates insert queries or returns data from multiple table depending on the relationship information of the tables provided

Bittorrent dht network infohash spider, for engiy.com[a bittorrent resource search engine]

A tool to allow for quick running of JSON-based scrapers using request-promise and jsonframe-cheerio.

bot to download pictures from theplace.ru

Crawl content of all pages for a wikipedia-site to use in stopword analysis or other natural language processing.

WebCreeper easy web crawler

Web crawling and scraping engine.

A simple evented web scraping framework using node.js

A lightweight web crawler.

🕷 Express middleware to detect crawler requests: request.isbot

Automatically extracts structured information from webpages

Very straightforward web crawler. Uses EventEmitter. Based on Simplecrawler but using a distributed queue system.

Spiderman makes it trivial work to write a crawler. Just define some components, use segues to connet them, finally write some script to specify data that needs to be transfer from one component to the next, that's all!

Robust web spider for Node.js

A simple utility for crawling a file system.