# @abilashinamdar/node-crawler

> Fast asynchronous NodeJS module for crawling/scraping a web through worker_threads.

Latest version **1.0.3** (published 2023-09-25) · ISC license · 0 weekly downloads

## Install

```sh
npm install @abilashinamdar/node-crawler
pnpm add @abilashinamdar/node-crawler
yarn add @abilashinamdar/node-crawler
bun add @abilashinamdar/node-crawler
```

## Health

**Score 20/100 (F)** — status: abandoned.

Positive: esm support; no vulnerabilities.

Warnings: low downloads; no types.

Negative: abandoned; low maintenance score.

## Facts

| | |
|---|---|
| Version | 1.0.3 |
| Published | 2023-09-25 |
| First published | 2023-09-21 |
| Weekly downloads | 0 |
| License | ISC |
| TypeScript types | none |
| Module format | ESM + CommonJS |
| Dependencies | 3 |
| Unpacked size | 20.7 KB |
| Known vulnerabilities | 0 |
| Install scripts | no |
| GitHub stars | 0 |
| Author | Abilash inamdar |
| Maintainers | abilash_inamdar |
| Keywords | crawler, scraper, crawling, spider, node-js-crawler, node-spider, dom, scraping, nodejs |

## Links

- npm: https://www.npmjs.com/package/@abilashinamdar/node-crawler
- Repository: https://github.com/Abilashinamdar/node-crawler
- Homepage: https://node-crawler-server.glitch.me
- Issues: https://github.com/Abilashinamdar/node-crawler/issues
- npm.io page: https://npm.io/package/@abilashinamdar/node-crawler

## Dependencies (3)

- [axios](https://npm.io/package/axios.md) ^1.4.0
- [cheerio](https://npm.io/package/cheerio.md) ^1.0.0-rc.12
- [robots-parser](https://npm.io/package/robots-parser.md) ^3.0.1

## Alternatives

- [babylon](https://npm.io/package/babylon.md) — 5.1M weekly downloads
- [csscolorparser](https://npm.io/package/csscolorparser.md) — 3.7M weekly downloads
- [expr-eval-fork](https://npm.io/package/expr-eval-fork.md) — 1.5M weekly downloads
- [@leeoniya/ufuzzy](https://npm.io/package/@leeoniya/ufuzzy.md) — 247.7K weekly downloads
- [xml-parser](https://npm.io/package/xml-parser.md) — 78.4K weekly downloads

## Recent versions

- 1.0.3 (latest) — 2023-09-25
- 1.0.2 — 2023-09-21
- 1.0.1 — 2023-09-21
- 1.0.0 — 2023-09-21

## README

# node-crawler

> Fast asynchronous NodeJS module for crawling/scraping a web through worker_threads.

[![npm package](https://nodei.co/npm/@abilashinamdar/node-crawler.png)](https://www.npmjs.com/package/@abilashinamdar/node-crawler)


[![Version][version-image]][download-url]
[![License][license-image]][download-url]

[version-image]: https://img.shields.io/npm/v/@abilashinamdar/node-crawler.svg
[license-image]: https://img.shields.io/npm/l/@abilashinamdar/node-crawler.svg
[download-url]: https://npmjs.com/package/@abilashinamdar/node-crawler

### Features
- Crawling on threads(CPU cores)
- Crawl images and links
- Configurable max threshold crawl
- Configurable retries
- Configurable request and retry delays
- Rotate proxies
- Bypass robots.txt

### Demo: https://node-crawler-server.glitch.me

# Get started

## Install
```sh
npm i @abilashinamdar/node-crawler

    OR

yarn add @abilashinamdar/node-crawler
```


# Basic usage

```js
import crawl from "@abilashinamdar/node-crawler";

crawl(config, onEveryCrawl, onCrawlError, onCrawlComplete)
```


### config
Key | Type | Default | Value
--- | --- | --- | ---
url | String | - | 
crawl | Array<String> | ["links"] | ["links", "images"]
maxCrawl | Number | 0 | Acts as threshold value. Stops crawling once it reaches the maxCrawl. 0 represents no threshold.
maxRetries | Number | 0 | No. of times the request should be retried when fails.
allowExternalImages| Boolean | false | true/false. true allows crawler to pick image if it points to outside the origin/host
retryDelay| Number | 0 | in milliseconds. Delay between the retry requests.
useProxy| Boolean | false | true/false. true allows crawler to rotate some free proxies from [https://www.free-proxy-list.com](https://www.free-proxy-list.com)
proxies| Array<String> | [] | Crawler will rotate the proxies from the list rather than fetching free proxies. useProxy: true is mandatory.
skipRobotsFile| Boolean | false | true/false. true allows crawler to bypass the robots.txt check.
requestDelay| Number | 0 | in milliseconds. Delay between the concurrent requests.


### onEveryCrawl(value)
```js
onEveryCrawl will get triggered when crawler will crawl Link or Image.
It will be triggered with the value as object which is as follows:
{
    type: string,
    link: string,
    reason: string
}

type - It can be 'LINK' / 'IMAGE' / 'IGNORE'
reason - Description for the IGNORE type.
```


### onCrawlError(error)
```js
onCrawlError will get triggered when crawler get error while crawling.
It will be triggered with the error as sample object as below or JS Error object.

{
    type: string,
    link: string,
    reason: string
}

type - It will be 'DISALLOWED'
reason - Description for the DISALLOWED type. Basically occurs when robots.txt disallows the url.
```


### onCrawlComplete(value)
```js
onCrawlComplete will get triggered when crawler done with the crawling.
It will be triggered with the value as object which is as follows:

{
    links: Set<String>,
    images: Set<String>,
    ignored: Set<String>
}

links - List of links from the crawled url.
images - List of image links from the crawled url.
ignored - List of ignored links from the crawled url.

Note: After this trigger the thread will get exit and will be available for others.
```


# License
ISC

---
_Source: https://npm.io/package/@abilashinamdar/node-crawler · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._
