1.0.21 • Published 4 years ago
epic-crawler v1.0.21
Epic Crawler
A simple crawler for scraping important data from web pages.
Installation
$ npm i epic-crawler --save
Usage
const crawler = new epicCrawler;
crawler.init("https://google.com", {
depth: 5,
}).then(() => {
crawler.crawl().then((data) => {
console.log(data);
});
}).catch((data) => {
console.log(data);
});
Options
Just three options are supported for now.
- depth - 1 to 5 (Default 1) | Crawling Depth.
- strict - boolean (Default True) | Set to False if you also want to collect links related to other websites.
- cache - boolean (Default True) | Speeds up the crawl by saving data in the cache.
Methods
- init: (url: string, { depth, strict, cache }?: options) => Promise - Initialize crawler.
- blackList: (fingerPrintList: (string | RegExp)[]) => this - Black List Links.
- clearCache: () => this - Clear previous crawled cache.
- crawl: () => Promise - Start Crawling.
1.0.19
4 years ago
1.0.18
4 years ago
1.0.21
4 years ago
1.0.20
4 years ago
1.0.17
4 years ago
1.0.16
4 years ago
1.0.15
4 years ago
1.0.14
4 years ago
1.0.13
4 years ago
1.0.12
4 years ago
1.0.11
4 years ago
1.0.9
4 years ago
1.0.8
4 years ago
1.0.10
4 years ago
1.0.7
4 years ago
1.0.6
4 years ago
1.0.5
4 years ago
1.0.4
4 years ago
1.0.3
4 years ago
1.0.2
4 years ago
1.0.1
4 years ago
1.0.0
4 years ago