# node-simple-crawler

> Simple web-crawler for nodejs

Latest version **0.1.6** (published 2015-06-07) · WTFPL license · 0 weekly downloads

## Install

```sh
npm install node-simple-crawler
pnpm add node-simple-crawler
yarn add node-simple-crawler
bun add node-simple-crawler
```

## Health

**Score 15/100 (F)** — status: abandoned.

Positive: no vulnerabilities.

Warnings: low downloads; no types; no esm support; pre 1.0.

Negative: abandoned; low maintenance score.

## Facts

| | |
|---|---|
| Version | 0.1.6 |
| Published | 2015-06-07 |
| First published | 2015-06-06 |
| Weekly downloads | 0 |
| License | WTFPL |
| TypeScript types | none |
| Module format | CommonJS |
| Dependencies | 8 |
| Known vulnerabilities | 0 (+8 in 2 direct dependencies) |
| Install scripts | no |
| Author | antixrist |
| Maintainers | antixrist |
| Keywords | spider, crawler, grabber, parser |

## Links

- npm: https://www.npmjs.com/package/node-simple-crawler
- Repository: https://github.com/antixrist/node-simple-crawler
- Issues: https://github.com/antixrist/node-simple-crawler/issues
- npm.io page: https://npm.io/package/node-simple-crawler

## Dependencies (8)

- [url](https://npm.io/package/url.md) ~0.10.3
- [extend](https://npm.io/package/extend.md) ~2.0.1
- [lodash](https://npm.io/package/lodash.md) ~3.9.3
- [charset](https://npm.io/package/charset.md) ~1.0.0
- [cheerio](https://npm.io/package/cheerio.md) ~0.19.0
- [request](https://npm.io/package/request.md) ~2.57.0
- [encoding](https://npm.io/package/encoding.md) ~0.1.11
- [jschardet](https://npm.io/package/jschardet.md) ~1.1.1

## Alternatives

- [babylon](https://npm.io/package/babylon.md) — 5.1M weekly downloads
- [csscolorparser](https://npm.io/package/csscolorparser.md) — 3.7M weekly downloads
- [expr-eval-fork](https://npm.io/package/expr-eval-fork.md) — 1.5M weekly downloads
- [@leeoniya/ufuzzy](https://npm.io/package/@leeoniya/ufuzzy.md) — 247.7K weekly downloads
- [xml-parser](https://npm.io/package/xml-parser.md) — 78.4K weekly downloads

## Recent versions

- 0.1.6 (latest) — 2015-06-07
- 0.1.5 — 2015-06-07
- 0.1.4 — 2015-06-07
- 0.1.3 — 2015-06-07
- 0.1.2 — 2015-06-06
- 0.1.1 — 2015-06-06
- 0.1.0 — 2015-06-06

## README

# node-simple-crawler
Simple web-crawler for node.js

# Installation
Using npm:
```
$ npm install node-simple-crawler --save
```

# Usage
```js
var Crawler = require('node-simple-crawler');

var crawler = new Crawler({
  concurrent: 5,
  logs: true,
  request: { // config for 'request' package
    headers: { 'user-agent': 'node-crawler' },
  },
  // If defined 'decodeTo' and not 'decodeFrom' then charset of 'response.body' will be detected automatically
  decodeTo: 'utf8',
  decodeFrom: '', // e.g. 'win-1251'
  callback: function (err, visited) {
    if (err) { throw err; }

    // all requests are done
    console.log('Done! Visited links:');
    console.log(visited);
  }
});

var handleRequest = function (err, url, response, $) {
  if (err) {
    console.log('Error url: '+ url);
    throw err;
  }

  // request done
  var that = this;

  console.log(url);
  console.log(response); // response object

  $('a').each(function(index, node) { // cheerio
    var $node = $(node);
    var href = $node.attr('href').split('#')[0];

    crawler.queue(that.resolve(url, href), handleRequest);
  });
};

crawler.queue('http://google.com/', handleRequest);
crawler.queue('http://yahoo.com/', {/* custom options for this link */}, handleRequest);
```

---
_Source: https://npm.io/package/node-simple-crawler · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._