Webscraper-exporter NPM

Webscraper Exporter

A simple yet powerful prometheus exporter for website performance metrics built using puppeteer.

Webscraper Exporter

CLI

Currently only tested on macOS and Linux, should support Windows

Getting started

Installation of the CLI

You can install the cli included to get started quickly with barely no configuration:

npm install --global webscraper-exporter

yarn global add webscraper-exporter

You can then check if everything went OK by running:

wsce --version

This should print the package's version

Configuration

In your current working directory, create a file named wsce.config.js with the following example content:

module.exports = {
    scraper: {
        urls: ["https://google.com"],
        puppeteerOptions: {},
        addons: [],
        interval: 60_000,
    },
    exporter: {
        port: 9924
    },
};

For further information on the configuration, see CONFIGURING.md

Using the CLI

To see all available options for a command, see wsce <command> -h;

Starting

You can start the exporter by simply executing:

wsce start

You can then access the metrics at http://localhost:9924

If you want more detailled logs, you can pass the -v argument.

To pass a custom config file, use -c path/to/file

Creating a project

Create a project from one of the included templates by running:

wsce init my-project

You will then be prompted for the project template. You can also directly pass the template to wsce init, e.g. :

wsce init my-project --template javascript

By default, wsce includes a typings file for the config so it's easier to fill. If you don't want that, you can pass --typings=false to the init command.

Adding, Removing and Listing Templates

wsce template has 3 subcommands:

add: Add a template from a local directory / remote repository
remove: Remove a saved template
list: List all saved templates

For more information on templates, see TEMPLATES.md

NodeJS module

Installation of the module

npm install --save webscraper-exporter

yarn add webscraper-exporter

Using the module

import { Exporter, Scraper } from "webscraper-exporter";

const scraper = new Scraper({
    urls: ["https://cstef.dev"],
    addons: [
        {
            name: "Logger",
            when: "before",
            run: (browser, page, URL) => {
                console.log(`I am running on ${URL}`);
            },
        },
    ],
    interval: 60_000,
});
scraper.start();
const exporter = new Exporter({ 
    scraper, 
    port: 9924, 
});
exporter.start();

For more examples, see the examples folder.