# bauer-crawler

> Multi-thread crawler engine.

Latest version **0.2.9** (published 2016-04-16) · MIT license · 0 weekly downloads

## Install

```sh
npm install bauer-crawler
pnpm add bauer-crawler
yarn add bauer-crawler
bun add bauer-crawler
```

## Health

**Score 15/100 (F)** — status: abandoned.

Positive: no vulnerabilities.

Warnings: low downloads; no types; no esm support; pre 1.0.

Negative: abandoned; low maintenance score.

## Facts

| | |
|---|---|
| Version | 0.2.9 |
| Published | 2016-04-16 |
| First published | 2015-05-31 |
| Weekly downloads | 0 |
| License | MIT |
| TypeScript types | none |
| Module format | CommonJS |
| Dependencies | 5 |
| Known vulnerabilities | 0 |
| Install scripts | no |
| GitHub stars | 1 |
| Author | Yuri Neves Silveira |
| Maintainers | yneves |
| Keywords | multi-thread, multi-core, process, fork, multi-process, cluster, crawler, scraper |

## Links

- npm: https://www.npmjs.com/package/bauer-crawler
- Repository: https://github.com/yneves/node-bauer-crawler
- Issues: https://github.com/yneves/node-bauer-crawler/issues
- npm.io page: https://npm.io/package/bauer-crawler

## Dependencies (5)

- [bauer-cluster](https://npm.io/package/bauer-cluster.md) ^1.3.8
- [bauer-factory](https://npm.io/package/bauer-factory.md) ^2.0.5
- [bauer-promise](https://npm.io/package/bauer-promise.md) ^1.1.3
- [bauer-cluster-queue](https://npm.io/package/bauer-cluster-queue.md) ^0.3.10
- [bauer-cluster-super](https://npm.io/package/bauer-cluster-super.md) ^0.3.7

## Alternatives

- [@tsparticles/shape-image](https://npm.io/package/@tsparticles/shape-image.md) — 303.7K weekly downloads
- [@tsparticles/shape-line](https://npm.io/package/@tsparticles/shape-line.md) — 233.7K weekly downloads
- [stringify-attributes](https://npm.io/package/stringify-attributes.md) — 58.6K weekly downloads
- [mobile-drag-drop](https://npm.io/package/mobile-drag-drop.md) — 46.3K weekly downloads
- [@comunica/actor-rdf-parse-html](https://npm.io/package/@comunica/actor-rdf-parse-html.md) — 29.2K weekly downloads

## Recent versions

- 0.2.9 (latest) — 2016-04-16
- 0.2.8 — 2015-11-26
- 0.2.7 — 2015-11-26
- 0.2.6 — 2015-10-04
- 0.2.5 — 2015-09-20
- 0.2.4 — 2015-09-12
- 0.2.2 — 2015-09-04
- 0.2.0 — 2015-09-03
- 0.1.13 — 2015-08-30
- 0.1.12 — 2015-08-29
- 0.1.11 — 2015-08-27
- 0.1.10 — 2015-08-08
- 0.1.9 — 2015-08-08
- 0.1.8 — 2015-07-31
- 0.1.7 — 2015-07-25
- … 7 more at https://npm.io/package/bauer-crawler/versions

## README

# bauer-crawler

Multi-process task runner.

## Installation

```
npm install bauer-crawler
```

## Usage

```js
var Crawler = require("bauer-crawler");

var myCrawler = new Crawler({
  
  // plugins to be loaded
  plugins: [
    "bauer-plugin-fetch",
    "bauer-plugin-scrape"
  ],
  
  // configurations for each plugin
  config: {
    fetch: {
      workers: 2,
      cache: {
        dir: "./http-cache",
        expires: "1d"
      }
    },
    scrape: {
      workers: 1,
      cache: {
        dir: "./scraper-output",
        expires: "1d"
      }
    }
  }
});

// starts the cluster application
myCrawler.start(function(promise) {
  
  // when returned promise is resolved process exits
  return promise
    .fetch("http://http-bin.org")
    .scrape({
      "a[href]": {
        urls: "attr:href"
      }
    });
});
```

## API Summary

  * `Crawler`
    * `new Crawler(options Object) :Crawler`
    * `.require(module String) :Object`
    * `.setupConfig(config Object) :void`
    * `.setupConfig(file String) :void`
    * `.setupCluster() :void`
    * `.setupCluster(cluster Cluster) :void`
    * `.setupCrawler(options Object) :void`
    * `.loadPlugin(plugins Array) :void`
    * `.loadPlugin(plugin String) :void`
    * `.loadPlugin(plugin Object) :void`
    * `.wrapCallback(module String) :Function`
    * `.wrapCallback(callback Function) :Function`
    * `.getWorkerRoleMap() :Object`
    * `.createWorkers() :void`
    * `.requestWorker(role String, request Object) :Promise`
    * `.wrapPromise(module String) :void`
    * `.wrapPromise(callback Function) :void`
    * `.wrapPromise(promise Promise, callback Function) :void`
    * `.start(main Function) :void`
    * `.setRepeat(repeat Boolean) :void`
    * `.getRepeat() :Boolean`
    

  * `Response`
    * `.sendOk() :void`
    * `.sendOk(data Object) :void`
    * `.sendError(error Error) :void`


  * `Worker`
    * `.sendReady() :void`


  * `Promise`
    * `.requestWorker(role String, request Object) :Promise`
    * `.repeat() :Promise`
    * `.repeat(repeat Boolean) :Promise`
    * `.time(callback Function) :Promise`
    * `.print() :Promise`
    * `.print(text String) :Promise`
    * `.print(array Array) :Promise`
    * `.print(object Object) :Promise`
    * `.print(anything) :Promise`
    * `.exit() :Promise`


## License

[MIT](./LICENSE)

---
_Source: https://npm.io/package/bauer-crawler · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._