# @opd/crawler

> web crawler based on Puppeteer

Latest version **1.7.0** (published 2023-11-13) · MIT license · 0 weekly downloads

## Install

```sh
npm install @opd/crawler
pnpm add @opd/crawler
yarn add @opd/crawler
bun add @opd/crawler
```

## Health

**Score 15/100 (F)** — status: abandoned.

Positive: no vulnerabilities.

Warnings: low downloads; no types; no esm support.

Negative: abandoned; low maintenance score.

## Facts

| | |
|---|---|
| Version | 1.7.0 |
| Published | 2023-11-13 |
| First published | 2019-07-17 |
| Weekly downloads | 0 |
| License | MIT |
| TypeScript types | none |
| Module format | CommonJS |
| Node | >=10.0.0 |
| Dependencies | 3 |
| Unpacked size | 10.4 KB |
| Known vulnerabilities | 0 |
| Install scripts | no |
| GitHub stars | 13 |
| Author | kagawagao |
| Maintainers | kagawa |
| Keywords | crawler, puppeteer, headless-chrome |

## Links

- npm: https://www.npmjs.com/package/@opd/crawler
- Repository: https://github.com/open-data-plan/crawler
- Homepage: https://github.com/open-data-plan/crawler#readme
- Issues: https://github.com/open-data-plan/crawler/issues
- npm.io page: https://npm.io/package/@opd/crawler

## Dependencies (3)

- [core-js](https://npm.io/package/core-js.md) ^3.6.5
- [puppeteer](https://npm.io/package/puppeteer.md) ^21.5.1
- [@babel/runtime](https://npm.io/package/@babel/runtime.md) ^7.12.1

## Alternatives

- [@snazzah/davey](https://npm.io/package/@snazzah/davey.md) — 1.5M weekly downloads
- [@vendure/testing](https://npm.io/package/@vendure/testing.md) — 8.3K weekly downloads
- [vue-simple-context-menu](https://npm.io/package/vue-simple-context-menu.md) — 6.6K weekly downloads
- [cypress-webpack-preprocessor-v5](https://npm.io/package/cypress-webpack-preprocessor-v5.md) — 2.1K weekly downloads
- [@backstage/plugin-catalog-backend-module-puppetdb](https://npm.io/package/@backstage/plugin-catalog-backend-module-puppetdb.md) — 1.3K weekly downloads

## Recent versions

- 1.7.0 (latest) — 2023-11-13
- 1.6.2 — 2022-08-04
- 1.6.1 — 2022-02-08
- 1.6.0 — 2021-10-09
- 1.5.1 — 2021-07-22
- 1.5.0 — 2021-05-14
- 1.4.0 — 2021-03-03
- 1.3.2 — 2020-10-12
- 1.3.1 — 2020-09-14
- 1.3.0 — 2020-07-20
- 1.2.2 — 2020-07-19
- 1.2.1 — 2020-05-21
- 1.2.0 — 2020-05-06
- 1.1.4 — 2020-03-24
- 1.1.3 — 2020-03-24
- … 9 more at https://npm.io/package/@opd/crawler/versions

## README

# crawler

> Web crawler based on [`Puppeteer`](https://github.com/GoogleChrome/puppeteer)

[![node (scoped)](https://img.shields.io/node/v/@opd/crawler.svg)](https://www.npmjs.com/package/@opd/crawler)
[![npm (scoped)](https://img.shields.io/npm/v/@opd/crawler.svg)](https://www.npmjs.com/package/@opd/crawler)
![build](https://github.com/open-data-plan/crawler/workflows/build/badge.svg)
[![Build Status](https://dev.azure.com/kagawagao/OPD/_apis/build/status/open-data-plan.crawler?branchName=master)](https://dev.azure.com/kagawagao/OPD/_build/latest?definitionId=3&branchName=master)
[![Coverage Status](https://coveralls.io/repos/github/open-data-plan/crawler/badge.svg?branch=master)](https://coveralls.io/github/open-data-plan/crawler?branch=master)

## Install

```bash
npm install @opd/crawler
```

## Use

```js
import Crawler from '@opd/crawler'
// or commonjs
const Crawler = require('@opd/crawler').default

const crawler = new Crawler(options)
```

## API

### `new Crawler(options)`

create crawler instance

`options`: crawler instance config

- `parallel`: maximum number of crawlers, default is `5`
- `pageEvaluate`: evaluate function on current page, see [`Puppeteer`](https://pptr.dev/#?product=Puppeteer&version=v1.18.1&show=api-pageevaluatepagefunction-args), **cannot support extra args now**

### `crawler.launch([options])`

launch browser use [`puppeteer.launch`](https://pptr.dev/#?product=Puppeteer&version=v1.18.1&show=api-puppeteerlaunchoptions)

### `crawler.queue(urls)`

add urls to crawler queue

> Note: check url **strictly**, means url must start with `https?`

### `crawler.start([urls]): PageResult[]`

start crawl page, if `urls` is presented, will call `crawler.queue` firstly.

```js
const result = await crawler.start()
console.log(result)

// [
//   {
//     url, // page url
//     result // crawled result
//   }
// ]
```

> Note: if you call `start` before `launch`, `browser` will also be launched, but with no extra launch options

---
_Source: https://npm.io/package/@opd/crawler · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._