# spiderweb

> Crawl multiple domains using one or more entry URLs.

Latest version **0.0.14** (published 2013-09-15) · 0 weekly downloads

## Install

```sh
npm install spiderweb
pnpm add spiderweb
yarn add spiderweb
bun add spiderweb
```

## Health

**Score 15/100 (F)** — status: abandoned.

Positive: no vulnerabilities.

Warnings: low downloads; no types; no esm support; pre 1.0.

Negative: abandoned; low maintenance score.

## Facts

| | |
|---|---|
| Version | 0.0.14 |
| Published | 2013-09-15 |
| First published | 2013-06-23 |
| Weekly downloads | 0 |
| TypeScript types | none |
| Module format | CommonJS |
| Dependencies | 5 |
| Known vulnerabilities | 0 (+6 in 3 direct dependencies) |
| Install scripts | no |
| GitHub stars | 0 |
| Author | James Hight |
| Maintainers | jmerrick |
| Keywords | spiderweb, spider, web, crawl, url, page, website |

## Links

- npm: https://www.npmjs.com/package/spiderweb
- Repository: git@github.com:JamesHight/node-spiderweb
- Issues: https://github.com/JamesHight/node-spiderweb/issues
- npm.io page: https://npm.io/package/spiderweb

## Dependencies (5)

- [mime](https://npm.io/package/mime.md) ~1.2.9
- [moment](https://npm.io/package/moment.md) ~2.0.0
- [cheerio](https://npm.io/package/cheerio.md) ~0.12.0
- [request](https://npm.io/package/request.md) ~2.21.0
- [config-extend](https://npm.io/package/config-extend.md) 0.0.1

## Alternatives

- [base64url](https://npm.io/package/base64url.md) — 6.1M weekly downloads
- [get-installed-path](https://npm.io/package/get-installed-path.md) — 502.9K weekly downloads
- [@uppy/url](https://npm.io/package/@uppy/url.md) — 185.8K weekly downloads
- [@d3fc/d3fc-shape](https://npm.io/package/@d3fc/d3fc-shape.md) — 16.2K weekly downloads
- [localizer](https://npm.io/package/localizer.md) — 226 weekly downloads

## Recent versions

- 0.0.14 (latest) — 2013-09-15
- 0.0.13 — 2013-09-15
- 0.0.12 — 2013-09-15
- 0.0.11 — 2013-09-08
- 0.0.10 — 2013-07-01
- 0.0.9 — 2013-06-30
- 0.0.8 — 2013-06-28
- 0.0.7 — 2013-06-28
- 0.0.6 — 2013-06-28
- 0.0.5 — 2013-06-28
- 0.0.4 — 2013-06-28
- 0.0.3 — 2013-06-27
- 0.0.2 — 2013-06-27
- 0.0.1 — 2013-06-23

## README

Spiderweb
=========

Crawl multiple domains using one or more entry URLs.


Installation
------------

````bash
npm install spiderweb
````

````javascript
var Spiderweb = require('spiderweb'),
	urls, options, spiderweb;

urls = ['bar.com', 'foo.bar.com'];

options = {
	strictSSL: false,
	images: true,
	excludedUrls: ['*biz.foo.com*', '*/admin/*']
};

spiderweb = new Spiderweb(urls, options);

spider.pageHandler = function(err, resp, body, entry) {
	console.log(entry.url);
	Spiderweb.prototype.pageHandler.apply(this, arguments);
}

spiderweb.start(function(err, log) {
	if (err) {
		console.log('ERROR: ', err)
	}

	if (log.length) {
		console.log('DONE: ' + log.length + ' page errors');
	}
	else {
		console.log('DONE: no errors');
	}
});
````

---
_Source: https://npm.io/package/spiderweb · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._