# simple-webscraper
> Web scraping toolkit
Latest version **1.5.4** (published 2019-12-04) · MIT license · 0 weekly downloads
## Install
```sh
npm install simple-webscraper
pnpm add simple-webscraper
yarn add simple-webscraper
bun add simple-webscraper
```
## Health
**Score 15/100 (F)** — status: abandoned.
Positive: no vulnerabilities.
Warnings: low downloads; no types; no esm support.
Negative: abandoned; low maintenance score.
## Facts
| | |
|---|---|
| Version | 1.5.4 |
| Published | 2019-12-04 |
| First published | 2019-03-06 |
| Weekly downloads | 0 |
| License | MIT |
| TypeScript types | none |
| Module format | CommonJS |
| Node | >=10.15.1 |
| Dependencies | 4 |
| Unpacked size | 29.9 KB |
| Known vulnerabilities | 0 (+5 in 1 direct dependencies) |
| Install scripts | no |
| Author | Norbert Logiewa |
| Maintainers | nl253 |
| Keywords | http, async, webscrape, webscraper, spider, scraper, scrape |
## Links
- npm: https://www.npmjs.com/package/simple-webscraper
- Repository: https://github.com/nl253/WebScraper
- Homepage: https://github.com/nl253/WebScraper/blob/master/README.md
- Issues: https://github.com/nl253/WebScraper/issues
- npm.io page: https://npm.io/package/simple-webscraper
## Dependencies (4)
- [cheerio](https://npm.io/package/cheerio.md) ^1.0.0-rc.3
- [sqlite3](https://npm.io/package/sqlite3.md) ^4.0.0
- [sequelize](https://npm.io/package/sequelize.md) ^5.21.2
- [node-fetch](https://npm.io/package/node-fetch.md) ^2.6.0
## Alternatives
- [@tsparticles/shape-image](https://npm.io/package/@tsparticles/shape-image.md) — 303.7K weekly downloads
- [@tsparticles/shape-line](https://npm.io/package/@tsparticles/shape-line.md) — 233.7K weekly downloads
- [stringify-attributes](https://npm.io/package/stringify-attributes.md) — 58.6K weekly downloads
- [mobile-drag-drop](https://npm.io/package/mobile-drag-drop.md) — 46.3K weekly downloads
- [@comunica/actor-rdf-parse-html](https://npm.io/package/@comunica/actor-rdf-parse-html.md) — 29.2K weekly downloads
## Recent versions
- 1.5.4 (latest) — 2019-12-04
- 1.5.2 — 2019-12-04
- 1.5.0 — 2019-12-04
- 1.4.0 — 2019-11-30
- 1.2.1 — 2019-03-06
- 1.1.1 — 2019-03-06
- 1.1.0 — 2019-03-06
- 1.0.2 — 2019-03-06
- 1.0.1 — 2019-03-06
- 1.0.0 — 2019-03-06
## README
# Web Scraper
- CSS selectors
- exporting function
- pre-configured to insert results into SQLite database and generate CSV
- stop conditions:
- time
- number of results
- number of websites
- filter function to check for results
- post- and pre-processing functions
- init with options or set them later with `spider.setVal1(v).setVal2(v2)`
- builder (call chaining) design pattern
- extensible
## API
Docs in gh-pages.
```js
const startURL = "https://stackoverflow.com/questions/...";
const crawler = new Spider(startURL);
crawler.setRespSecW8(20)
.appendSelector('p.info')
.appendSelector('p.more-info')
.appendFollowSelector('.btn.next')
.appendFollowSelector('.btn.next-page')
.setPostProcessTextFunct(text => text.replace('mother', 'yes'))
.setFilterFunct(txt => !!txt.match('sunflower'))
.setTimeLimit(120) // sec
.setThreadCount(8) // #workers
.setSiteCount(100) // distinct URLs
// run returns void, you might want to provide an export function for each result (see below)
// by default goes to sqlite ./db and prints to console
.run();
```
OR use init object in the constructor
```js
// DEFAULT init options
const spiderOpts = {
// Function
exportFunct: exports.combine(exports.console(), exports.sqlite()),
// predicate i.e. Function
filterFunct: (txt) => true,
// Array
followSelectors: [],
// String
logInfoFile: undefined, // logging goes to console
// String
logInfoFile: undefined, // logging goes to console
// Integer
redirFollowCount: 3,
// Integer
respSecW8: 10,
// Array
selectors: [],
// Integer
resultCount: 100,
// Integer
siteCount: 10, // #sites
// Integer
threadCount: 4,
// Integer
timeLimit: 60, // sec
};
const startURL = "https://stackoverflow.com/questions/...";
const crawler = new Spider(startURL, spiderOpts);
crawler.run();
```
```js
const startURL = "https://stackoverflow.com/questions/...";
const crawler = new Spider(startURL);
crawler.setRespSecW8(20)
.appendSelector('p.info')
.appendSelector('p.more-info')
.appendFollowSelector('.btn.next')
.appendFollowSelector('.btn.next-page')
.setPostProcessTextFunct(text => text.replace('mother', 'yes'))
.setFilterFunct(txt => !!txt.match('sunflower'))
.setTimeLimit(120) // sec
.setThreadCount(8) // #workers
.setSiteCount(100) // distinct URLs
// run returns void, you might want to provide an export function for each result (see below)
// by default goes to sqlite ./db and prints to console
.run();
```
See export functions below to save results.
## Export Function
Must be of type `(uri: string, selector: string, text: string) => Promise<*>`.
There is a few configurable export functions that you can use:
Import the exporting module:
```js
const { exporting, Spider } = require('simple-webscraper');
```
Declare a spider:
```js
const spider = new Spider(uri, { /* opts */ });
```
- `sqlite`
Generates a `Result` table with columns: `id INT`, `text TEXT`, `selector TEXT`, `uri TEXT` columns.
```js
spider.setExportFunct(exporting.sqlite()) // generate output db name
.run();
```
```js
spider.setExportFunct(exporting.sqlite('my-database.sqlite'))
.run();
```
- `console`
```js
spider.setExportFunct(exporting.console()) // default formatter
.run();
```
```js
spider.setExportFunct(exporting.console('%s :: %s => %s')) // string formatter for (uri, selector, text)
.run();
```
```js
spider.setExportFunct(exporting.console((uri, selector, text) => `${uri} :: ${text.slice(0, 100)}`))
.run();
```
- `file`
```js
spider.setExportFunct(exporting.file()) // default file name, default formatter
.run();
```
```js
spider.setExportFunct(exporting.file('results.csv')) // custom file name, default csv formatter
.run();
```
```js
spider.setExportFunct(exporting.file('results.log', 'INFO %s, %s, %s')) // custom file name, string formatter
.run();
```
```js
spider.setExportFunct(exporting.file('results.log', (uri, selector, text) => `${uri} :: ${text.slice(0, 100)}`))
.run();
```
- `combine` (used to broadcast results to many exports)
```js
spider.setExportFunct(exporting.combine(
exporting.sqlite(),
exporting.console(),
exporting.file(),
)).run();
```
- `db`
```js
spider.setExportFunct(exporting.db(dbURI)) // look at sequelize docs
.run();
```
- `default` (enabled by default, sends to console, CSV file and sqlite database)
It's very easy to define your own export function. E.g. imagine wanting to POST each result to some 3rd party API.
```js
const myExportFunction = async (uri, selector, text) => {
const res = await http.post(myURI, { uri, selector, text });
return;
};
```
## Example
More examples in `./examples`.
```js
const { Spider, exporting } = require('simple-webscraper');
(async function() {
const s = new Spider('https://www.jobsite.co.uk/jobs/javascript');
const sqliteExport = await exporting.sqlite('./db', true /* force wipe if exists */);
s.setExportFunct(sqliteExport)
.appendSelector(".job > .row > .col-sm-12")
// don't look for jobs in London, make sure they are graduate!
.setFilterFunct(txt => !!txt.match('raduate') && !txt.match('London'))
// next page
.appendFollowSelector(".results-footer-links-container ul.pagination li a[href*='page=']")
// stop after 3 websites (urls)
.setSiteCount(3)
// run for 30 sec
.setTimeLimit(30)
.run();
})();
```
---
_Source: https://npm.io/package/simple-webscraper · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._