Smart Crawler
What is this
This module is propose to scrapy website pages and extract information from doms which selected by jQuery-like selectors
Features
- Batch page scrapy support
- jQuery-like selector supported for extracting dom infomation
- Promises/A style support
How to use
- Scrapy website:
new Crawler([domain1, domain2], callback)
var Crawler = require("crawler");
var domain = "http://example.com";
new Crawler(domain, function (err, result, mergedResult) {
var $body = result[domain];
console.log($body.html());
});- Using selector:
new Crawler([domain1, domain2], queryString, callback)
Note: not all jQuery query style is supported, details on cheerio
var Crawler = require("crawler");
var domain = "http://example.com";
new Crawler(domain, "p", function (err, result, mergedResult) {
if (err) return;
var paragraphs = result[domain];
paragraphs.forEach(function ($p) {
console.log($p.text());
});
});- Custom request options:
new Crawler([domain1, domain2], requestOptions,queryString, callback)
var Crawler = require("crawler");
var domain = "http://example.com";
new Crawler(domain, {
'timeout': 2000,
'headers': {
'User-Agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36'
}
} "p", function (err, result, mergedResult) {
if (err) return;
});Request Options available on Request: Custom HTTP Headers
More examples please visit: examples
External API
.refetch(): to refetch the same sites with same parameters;