1clickcrawl v1.0.1

1ClickCrawler

Simple quick webcrawler for nodejs. Provide your own visit function and the crawler will do the rest.

Installation

$ npm install 1clickcrawler

Usage

var Crawler = require('1clickcrawler');
var crawler = new Crawler(delay, seeds, visit);
crawler.start()

Parameters

delay: milliseconds to wait between requests (ex: 200) seeds: array of urls to start the crawl (ex: 'https://example.com') visit: a function that will process the data with params url, res, html (ex: function(url,res,html) {})