# mini-crawler

> A really small crawler.

Latest version **4.0.2** (published 2020-07-23) · MIT license · 0 weekly downloads

## Install

```sh
npm install mini-crawler
pnpm add mini-crawler
yarn add mini-crawler
bun add mini-crawler
```

## Health

**Score 25/100 (F)** — status: abandoned.

Positive: has types; no vulnerabilities; high quality score.

Warnings: low downloads; no esm support.

Negative: abandoned; low maintenance score.

## Facts

| | |
|---|---|
| Version | 4.0.2 |
| Published | 2020-07-23 |
| First published | 2019-07-30 |
| Weekly downloads | 0 |
| License | MIT |
| TypeScript types | bundled |
| Module format | CommonJS |
| Dependencies | 3 |
| Unpacked size | 13 KB |
| Known vulnerabilities | 0 |
| Install scripts | no |
| GitHub stars | 1 |
| Author | Tamas Halasi |
| Maintainers | trusted.tomato |
| Keywords | crawler, request, spider, web |

## Links

- npm: https://www.npmjs.com/package/mini-crawler
- Repository: https://github.com/trustedtomato/mini-crawler
- Homepage: https://github.com/trustedtomato/mini-crawler#readme
- Issues: https://github.com/trustedtomato/mini-crawler/issues
- npm.io page: https://npm.io/package/mini-crawler

## Dependencies (3)

- [got](https://npm.io/package/got.md) ^11.5.1
- [p-queue](https://npm.io/package/p-queue.md) ^6.5.0
- [@types/jest](https://npm.io/package/@types/jest.md) ^26.0.5

## Alternatives

- [@opentelemetry/exporter-zipkin](https://npm.io/package/@opentelemetry/exporter-zipkin.md) — 14.8M weekly downloads
- [pusher-js](https://npm.io/package/pusher-js.md) — 2.0M weekly downloads
- [browserify](https://npm.io/package/browserify.md) — 1.7M weekly downloads
- [sqs-consumer](https://npm.io/package/sqs-consumer.md) — 1.7M weekly downloads
- [@sanity/eventsource](https://npm.io/package/@sanity/eventsource.md) — 930.8K weekly downloads

## Recent versions

- 4.0.2 (latest) — 2020-07-23
- 4.0.1 — 2020-07-23
- 4.0.0 — 2020-07-23
- 3.0.0 — 2019-08-11
- 2.0.0 — 2019-07-31
- 1.1.0 — 2019-07-30
- 1.0.0 — 2019-07-30

## README

# mini-crawler
A really small crawler based on [p-queue](https://www.npmjs.com/package/p-queue)
and [got](https://www.npmjs.com/package/got).

## Installation
```
npm install mini-crawler
```

## Usage
```javascript
const { Crawler } = require('..')
const crawler = new Crawler({
  // All of p-queue options are accepted,
  // see https://www.npmjs.com/package/p-queue#options.
  concurrency: 10,
  interval: 1000,
  intervalCap: 500
})

crawler.crawl({
  // The crawling will begin at this URL.
  url: 'https://www.google.com/search?q=Tame+Impala',
  // All of got's options are accepted,
  // except isStream and resolveBodyOnly.
  // See https://www.npmjs.com/package/got#options.
  gotOptions: {
    headers: {
      'Accept-Language': 'en-US'
    }
  },
  callback: (err, { body, options: { url } }) => {
    if (err) {
      console.error(`Error occured at ${url}!`)
      return
    }

    console.log(`Fetched ${url}!`)

    const urls = [...body.matchAll(/href="(.*?)"/g)]
      .map(match => match[1]
        .replace(/&/g, '&')
      )
    console.log(`Found ${urls.length} new URLs!`)

    // The returned value will be used for further crawls.
    // It will be transformed into a CrawlOptions array,
    // then crawl() will be called for all of the array's items.
    // To understand the transforming mechanism,
    // see https://github.com/trustedtomato/mini-crawler/blob/master/src/result-to-crawloptions-array.ts
    return urls
  }
})
```

Output:
```text
Fetched https://www.google.com/search?q=Tame+Impala!
Found 145 new URLs!
Fetched https://www.google.com/?sa=X&ved=0ahUKEwiM39zjjuHqAhUUdCsKHf7xAX8QOwgC!
Found 19 new URLs!
Fetched https://www.google.com/?output=search&ie=UTF-8&sa=X&ved=0ahUKEwiM39zjjuHqAhUUdCsKHf7xAX8QPAgE!
Found 19 new URLs!
Fetched https://www.google.com/advanced_search!
Found 22 new URLs!
Fetched https://www.google.com/search?q=Tame+Impala&ie=UTF-8&source=lnms&tbm=isch&sa=X&ved=0ahUKEwiM39zjjuHqAhUUdCsKHf7xAX8Q_AUICSgC!
Found 59 new URLs!
…
```

## API
For the auto-generated typedoc, see the [project's GitHub Page](https://trustedtomato.github.io/mini-crawler/).

## Examples
See the [examples](https://github.com/trustedtomato/mini-crawler/tree/master/examples) folder.
There lies a crawler which tries to find Hitler's Wikipedia article
by navigating from article to article, using the links in them.
You know, the classic [clicks to Hitler](https://en.wikipedia.org/wiki/Wikipedia:Wiki_Game) game.

To try out the example,
clone this respository,
cd into it,
run `npm install && npm run build` then `node examples/clicks-to-hitler.js Barack_Obama`.
(At the time of writing, you can get there with two clicks.)

---
_Source: https://npm.io/package/mini-crawler · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._