# hunt-crawler

> a web crawler based on crawlee, use file to cache result. Easy to maintain as Singleton Service.

Latest version **1.1.0** (published 2023-08-10) · ISC license · 0 weekly downloads

## Install

```sh
npm install hunt-crawler
pnpm add hunt-crawler
yarn add hunt-crawler
bun add hunt-crawler
```

## Health

**Score 15/100 (F)** — status: abandoned.

Positive: no vulnerabilities.

Warnings: low downloads; no types; no esm support.

Negative: abandoned; low maintenance score.

## Facts

| | |
|---|---|
| Version | 1.1.0 |
| Published | 2023-08-10 |
| First published | 2023-08-09 |
| Weekly downloads | 0 |
| License | ISC |
| TypeScript types | none |
| Module format | CommonJS |
| Dependencies | 5 |
| Unpacked size | 8.8 KB |
| Known vulnerabilities | 0 |
| Install scripts | no |
| Author | halfong |
| Maintainers | sunkey |
| Keywords | crawl, web, crawler, nodejs |

## Links

- npm: https://www.npmjs.com/package/hunt-crawler
- npm.io page: https://npm.io/package/hunt-crawler

## Dependencies (5)

- [dotenv](https://npm.io/package/dotenv.md) ^16.3.1
- [moment](https://npm.io/package/moment.md) ^2.29.4
- [crawlee](https://npm.io/package/crawlee.md) ^3.5.0
- [crypto-js](https://npm.io/package/crypto-js.md) ^4.1.1
- [puppeteer](https://npm.io/package/puppeteer.md) ^20.9.0

## Alternatives

- [@opentelemetry/exporter-zipkin](https://npm.io/package/@opentelemetry/exporter-zipkin.md) — 14.8M weekly downloads
- [pusher-js](https://npm.io/package/pusher-js.md) — 2.0M weekly downloads
- [browserify](https://npm.io/package/browserify.md) — 1.7M weekly downloads
- [sqs-consumer](https://npm.io/package/sqs-consumer.md) — 1.7M weekly downloads
- [@sanity/eventsource](https://npm.io/package/@sanity/eventsource.md) — 930.8K weekly downloads

## Recent versions

- 1.1.0 (latest) — 2023-08-10
- 1.0.2 — 2023-08-10
- 1.0.0 — 2023-08-09

## README

# What's this

A **http crawler** for nodejs.
Based on **crawlee**, **puppeteer**.
Auto handle **cache** using filesystem
Easy to planning your **request in queue**

# How to use

### Install
```
npm i hunt-crawler
```

### Sample

```javascript
const { Hunter } = require('hunt-crawler')

const $hunter = new Hunter({
  // options for Hunter
  cacheFolder : __dirname + '/../caches'   // where to save cache files
  log : false // show logs
},{
  // options for PuppeteerCrawler
  keepAlive: false, // should the crawler keep alive after request cleaned
})

/**
 * Crawl the url
 * - Start to crawl the url and cache its content to file
 * - Launch browser if not launched
 */
await $hunter.run(
  'https://111.com/somepage', // url to scrawl

  /**
   * Callback
   * handle the content( already cached to file )
   * or start new requests ( $hunter.run )
   */
  async( url, content ){
    console.log( content )
  },

  /**
   * Option
   */
  {
    /** 
     * suffix
     * optional, default '.cache'
     * - the cache file name = md5(url).suffix
     * - $hunter.run prefer to use cache (if exists), except when force=true
     * - You can use this to manage the caches of this page
     */
    suffix :moment.utc().format('YYYY_MM_DD')+'.html',

    /**
     * removeTags
     * optional, default []
     * remove tags from the returned page.content() when save to cache
     */
    removeTags : ['script','svg'],

    /** 
     * play
     * Do actions like scroll, wait or anyting on page in browse.
     */
    async play( url, ctx, $hunter ){
      await new Promise( r => setTimeout(r,200) ) // wait for 1 second
    },

  }
)
```

# More Sample and Test

see **/test** folder

you could test by run:

```node test/simpleCrawl {url?} {force?}```
```node test/batchCrawl {url1} {url2} {url3} ...```

---
_Source: https://npm.io/package/hunt-crawler · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._