# smart-crawler

> This module is propose to scrapy website pages and extract information from doms which selected by jQuery-like selectors

Latest version **0.0.0** (published 2014-07-03) · MIT license · 0 weekly downloads

## Install

```sh
npm install smart-crawler
pnpm add smart-crawler
yarn add smart-crawler
bun add smart-crawler
```

## Health

**Score 25/100 (F)** — status: abandoned.

Positive: no vulnerabilities.

Warnings: low downloads; no types; no esm support; pre 1.0.

Negative: abandoned.

## Facts

| | |
|---|---|
| Version | 0.0.0 |
| Published | 2014-07-03 |
| First published | 2014-07-03 |
| Weekly downloads | 0 |
| License | MIT |
| TypeScript types | none |
| Module format | CommonJS |
| Dependencies | 3 |
| Known vulnerabilities | 0 (+1 in 1 direct dependencies) |
| Install scripts | no |
| GitHub stars | 2 |
| Author | hh54188 |
| Maintainers | hh54188 |
| Keywords | scrapy, crawler, robot, robots, hacker |

## Links

- npm: https://www.npmjs.com/package/smart-crawler
- Repository: https://github.com/hh54188/Smart-Crawler
- Issues: https://github.com/hh54188/Smart-Crawler/issues
- npm.io page: https://npm.io/package/smart-crawler

## Dependencies (3)

- [rsvp](https://npm.io/package/rsvp.md) ^3.0.9
- [cheerio](https://npm.io/package/cheerio.md) ^0.12.4
- [request](https://npm.io/package/request.md) ^2.29.0

## Recent versions

- 0.0.0 (latest) — 2014-07-03

## README

# Smart Crawler

## What is this

This module is propose to scrapy website pages and extract information from doms which selected by jQuery-like selectors

## Features

- Batch page scrapy support
- jQuery-like selector supported for extracting dom infomation 
- Promises/A style support

## How to use

- Scrapy website: `new Crawler([domain1, domain2], callback)`

```
var Crawler = require("crawler");
var domain = "http://example.com";

new Crawler(domain, function (err, result, mergedResult) {
	var $body = result[domain];
	console.log($body.html());
});
```
- Using selector: `new Crawler([domain1, domain2], queryString, callback)`

*Note: not all jQuery query style is supported, details on [cheerio](https://github.com/cheeriojs/cheerio)*

```
var Crawler = require("crawler");
var domain = "http://example.com";

new Crawler(domain, "p", function (err, result, mergedResult) {

	if (err) return;

	var paragraphs = result[domain];
	paragraphs.forEach(function ($p) {
		console.log($p.text());
	});
});
```

- Custom request options: `new Crawler([domain1, domain2], requestOptions,queryString, callback)`

```
var Crawler = require("crawler");
var domain = "http://example.com";

new Crawler(domain, {
	'timeout': 2000,
    'headers': {
        'User-Agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36'
    }
} "p", function (err, result, mergedResult) {

	if (err) return;
});
```
*Request Options available on [Request: Custom HTTP Headers](https://github.com/mikeal/request#custom-http-headers)*


More examples please visit: [examples](https://github.com/hh54188/Smart-Crawler/tree/master/examples)

## External API

- `.refetch()`: to refetch the same sites with same parameters;

---
_Source: https://npm.io/package/smart-crawler · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._