simplecrawler
Very straightforward, event driven web crawler. Features a flexible queue interface and a basic cache mechanism with extensible backend.
1.1.9 • Published 4 years ago
Simplecrawler Packages
semantic-crawler
Priority based Semantic Web Crawler.
simplecrawler-referrer-filter
Very straigntforward web crawler. Uses EventEmitter. Generates queue statistics and has a basic cache mechanism with extensible backend. This version is forked in order to add the ability to filter by referrer url
spiderhunt
Very straightforward web crawler. Uses EventEmitter. Based on Simplecrawler but using a distributed queue system.
simplecrawler-fork
Forked version of https://github.com/ciffard/node-simplecrawler.
simplecrawler-zic
Very straightforward, event driven web crawler. Features a flexible queue interface and a basic cache mechanism with extensible backend.
simplecrawler-mongo-queue
MongoDB queue implementation for simplecrawler
ispider
iSpider is a simple and fast web spider written with Nodejs!
web-spider
web-spider is a simple and fast web spider written with Nodejs!
sqlite-simplecrawler-queue
SQLite FetchQueue Implementation for Simplecrawler
jespider
A web crawler forked from simplecrawler.
jes-spider
A web crawler forked from simplecrawler.
@gombosg/simplecrawler
Very straightforward, event driven web crawler. Features a flexible queue interface and a basic cache mechanism with extensible backend.
mongodb-simplecrawler-queue
MongoDB FetchQueue Implementation for Simplecrawler