# wwweb

> An autonomous webcrawler for indexing robots.txt files.

Latest version **1.0.3** (published 2016-06-07) · MIT license · 0 weekly downloads

## Install

```sh
npm install wwweb
pnpm add wwweb
yarn add wwweb
bun add wwweb
```

Provides the command `wwweb`.

## Health

**Score 15/100 (F)** — status: abandoned.

Positive: no vulnerabilities.

Warnings: low downloads; no types; no esm support.

Negative: abandoned; low maintenance score.

## Facts

| | |
|---|---|
| Version | 1.0.3 |
| Published | 2016-06-07 |
| First published | 2016-06-04 |
| Weekly downloads | 0 |
| License | MIT |
| TypeScript types | none |
| Module format | CommonJS |
| Dependencies | 3 |
| Known vulnerabilities | 0 (+25 in 1 direct dependencies) |
| Install scripts | no |
| Author | Florian Pircher |
| Maintainers | addpixel |
| Keywords | software-agent, autonomous, bot, webcrawler, robots |

## Links

- npm: https://www.npmjs.com/package/wwweb
- Repository: https://github.com/SoftwareAgenten/WWWEB
- Homepage: https://github.com/SoftwareAgenten/WWWEB#readme
- Issues: https://github.com/SoftwareAgenten/WWWEB/issues
- npm.io page: https://npm.io/package/wwweb

## Dependencies (3)

- [axios](https://npm.io/package/axios.md) ^0.12.0
- [yargs](https://npm.io/package/yargs.md) ^4.7.1
- [colors](https://npm.io/package/colors.md) ^1.1.2

## Recent versions

- 1.0.3 (latest) — 2016-06-07
- 1.0.2 — 2016-06-04
- 1.0.1 — 2016-06-04
- 1.0.0 — 2016-06-04

## README

# WWWEB

![](https://software-agenten.addpixel.net/static/wwweb.svg)

An autonomous webcrawler for indexing robots.txt files.

## Requirements

* node.js ^6.0.0

## Usage

```sh
wwweb -d <domain> [-s <interval>] -o <directory> [--rest <seconds>] [[-v] -v] [-t <timeout>]
```

## Options

| Flag | Alias | Description | Info |
|:--|:--|:--|:--|
| `--domain` | `-d` | Initial domain | required |
| `--save-interval` | `-s` | Interval in seconds for outputting reports | default: 30 |
| `--output` | `-o` | Name of the output directory | required |
| `--help` | `-h` | Show help | |
| `--rest` | `-r` | Seconds to rest between requests | default: 0 |
| `--timeout` | `-t` | Milliseconds before a request times out | default: 15000 |
| `--verbose` | `-v` | Verbose output of what is going on | -vv for debug output |
| `--no-color` | | Disable colorful output | |

## Examples

Crawl from example.org and output files to the current working directory:

```sh
wwweb -d example.org -o .
```

Crawl from example.org, output files to ./reports/, output warning, wait eight seconds for files to load and save a report every minute.

```sh
wwweb -d example.org -o reports/ -v -t 8000 -s 60
```

---
_Source: https://npm.io/package/wwweb · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._