1.0.1 • Published 4 years ago

1clickcrawl v1.0.1

Weekly downloads
-
License
ISC
Repository
-
Last release
4 years ago

1ClickCrawler

Simple quick webcrawler for nodejs. Provide your own visit function and the crawler will do the rest.

Installation

$ npm install 1clickcrawler

Usage

var Crawler = require('1clickcrawler');
var crawler = new Crawler(delay, seeds, visit);
crawler.start()

Parameters

delay: milliseconds to wait between requests (ex: 200) seeds: array of urls to start the crawl (ex: 'https://example.com') visit: a function that will process the data with params url, res, html (ex: function(url,res,html) {})

1.0.1

4 years ago

1.0.0

4 years ago