# scrapebp

> Boilerplate code for a Node.js scraper with CLI

Latest version **0.5.0** (published 2015-06-25) · MIT license · 0 weekly downloads

> **Deprecated.** This package is deprecated.

## Install

```sh
npm install scrapebp
pnpm add scrapebp
yarn add scrapebp
bun add scrapebp
```

Provides the command `scrapebp`.

## Health

**Score 10/100 (F)** — status: deprecated.

Negative: deprecated.

## Facts

| | |
|---|---|
| Version | 0.5.0 |
| Published | 2015-06-25 |
| First published | 2014-02-05 |
| Weekly downloads | 0 |
| License | MIT |
| TypeScript types | none |
| Module format | CommonJS |
| Dependencies | 6 |
| Known vulnerabilities | 0 (+1 in 1 direct dependencies) |
| Install scripts | no |
| GitHub stars | 2 |
| Author | KY LEE |
| Maintainers | leesei |
| Keywords | scraper, crawler, boilerplate |

## Links

- npm: https://www.npmjs.com/package/scrapebp
- Repository: https://github.com/leesei/node-scrapebp
- Homepage: https://github.com/leesei/node-scrapebp#readme
- Issues: http://github.com/leesei/node-scrapebp/issues
- npm.io page: https://npm.io/package/scrapebp

## Dependencies (6)

- [clone](https://npm.io/package/clone.md) *
- [debug](https://npm.io/package/debug.md) *
- [needle](https://npm.io/package/needle.md) *
- [nomnom](https://npm.io/package/nomnom.md) *
- [cheerio](https://npm.io/package/cheerio.md) *
- [deep-defaults](https://npm.io/package/deep-defaults.md) *

## Alternatives

- [@tsparticles/shape-image](https://npm.io/package/@tsparticles/shape-image.md) — 303.7K weekly downloads
- [@tsparticles/shape-line](https://npm.io/package/@tsparticles/shape-line.md) — 233.7K weekly downloads
- [stringify-attributes](https://npm.io/package/stringify-attributes.md) — 58.6K weekly downloads
- [mobile-drag-drop](https://npm.io/package/mobile-drag-drop.md) — 46.3K weekly downloads
- [@comunica/actor-rdf-parse-html](https://npm.io/package/@comunica/actor-rdf-parse-html.md) — 29.2K weekly downloads

## Recent versions

- 0.5.0 (latest) — 2015-06-25
- 0.4.0 — 2015-06-24
- 0.3.2 — 2015-05-12
- 0.2.2 — 2014-02-09
- 0.2.1 — 2014-02-08
- 0.2.0 — 2014-02-08
- 0.1.0 — 2014-02-05

## README

# node-scrapebp

[![npm version](https://img.shields.io/npm/v/scrapebp.svg?style=flat-square)](https://www.npmjs.com/scrapebp)
[![npm downloads](https://img.shields.io/npm/l/scrapebp.svg?style=flat-square)](https://www.npmjs.com/scrapebp)
[![dependency status](https://img.shields.io/david/leesei/node-scrapebp.svg?style=flat-square)](https://david-dm.org/leesei/node-scrapebp)

## Installation

```bash
npm install scrapebp
```

This module can be forked or depended upon for future scraping projects.  
Caller only need to specify `opts` and implement the custom scraper and scrape callback function.

## Usage

See [bin/scrapebp](bin/scrapebp) for details.

```javascript
var ScrapeBp = require('scrapebp');

// DemoScraper and scrapeCallback are defined
// opts for ScrapeBp is prepared

var scrapebp = ScrapeBp(opts);

scrapebp.on('headers', function (headers) {
  console.log("- %s headers ready", opts.method);
  if (argv.dumpHeader) {
    console.log(headers);
  }
});

scrapebp.on('redirect', function (url, remaining) {
  console.log("- redirects to: %s (%d remaining)", url, remaining);
});

scrapebp.on('error', function (err) {
  console.error(err);
});

scrapebp.on('$ready', function(url, $) {
  console.log("- $ ready");
  // $ is the cheerio object
  // use $.html() to get the response body
  // useful if the response is not html/xml

  if (argv.dumpBody) {
    console.log("body:");
    console.log($.html());
  }

  // invoke our scraper
  DemoScraper.scrape(url, $, scrapeCallback);
});
```

## Debug

Following `needle`, `scrapebp` uses [visionmedia/debug](https://github.com/visionmedia/debug).

```sh
DEBUG=scrapebp bin/scrapebp www.yahoo.com
```

## Design choice

Originally [hyperquest](https://github.com/hyperquest/hyperquest), [hyperdirect](https://github.com/hyperquest/hyperdirect) and [hyperzip](https://github.com/hyperquest/hyperzip) is used as the HTTP stack.
Then I switched to [tomas/needle](https://github.com/tomas/needle), which supports all of the above and `iconv` conversion.

## Reference for dependencies 

[cheeriojs/cheerio](https://github.com/cheeriojs/cheerio)

[tomas/needle](https://github.com/tomas/needle)

## TODO

write tests that covers:
- GET with query string
- POST with payload
- redirects
- use of compression (`-z` and check response header and decoded body)
- error handling

features:
- character set detection with [aadsm/jschardet](https://github.com/aadsm/jschardet)? (in case HTTP header and HTML meta did not signals charset)
- promisify?
- browserify

bug:
- multi-byte cut-off (https://github.com/tomas/needle/issues/88)

---
_Source: https://npm.io/package/scrapebp · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._