2.2.2 • Published 11 months ago

site-archive v2.2.2

Weekly downloads
-
License
MIT
Repository
github
Last release
11 months ago

Site Archive

This is a site crawler designed to track visual and content differences between crawls.

Example usage:

import Crawler from "site-archive";
import path from "path";

const entryPointRaw = process.argv[2];
const crawler = new Crawler(entryPointRaw, path.resolve('.', 'captures'), 1, console.log);
await crawler.crawl();
yarn run start https://yoursite.com

Options:

Default options:

let options = {
    ignoreHead: false, // If true skip any urls found in the <head> element.
    screenshotsOnly: false, // If true doesn't save any page HTML.
    screenshotSizes: [], // An array of {width: number, height: number} for screenshots to take.
    htmlOnly:  false, // Only attempt to follow links that might be HTML documents.
    logLevel:  2, // The detail level of messages to display. (1, 2, or 3 with 1 being every single message generated).
    onEvent: (msg) => console.log(msg), // Function that handles messaging from the crawler.
    htmlTypes: ['html', 'htm', 'xhtml', 'asp', 'aspx', 'shtml', 'dhtml', 'php', 'php5', 'jsp'], // Which file extensions (in addition to no-extension) are considered to be possible HTML links.
    ignoreQueryString: false, // If true, doesn't consider query strings to be part of a URL.
    ignoreAnchors: false, // If true, doesn't consider anchors to be part of a URL.
    selectorsToRemove: [], // An array of selectors for elements to be removed from the page before taking screenshots.
    timeout:  30000, // How long to wait for a page to finish load before timing out.
    redirect: 'follow', // How to treat HTTP redirects. (https://chromestatus.com/feature/4614142321229824)
}

Status:

Still very much a work in progress

2.2.1

1 year ago

2.2.2

11 months ago

2.2.0

1 year ago

2.1.27

1 year ago

2.1.28

1 year ago

2.1.18

2 years ago

2.1.19

2 years ago

2.1.2

2 years ago

2.1.1

2 years ago

2.1.25

2 years ago

2.1.4

2 years ago

2.1.3

2 years ago

2.1.23

2 years ago

2.1.6

2 years ago

2.1.24

2 years ago

2.1.5

2 years ago

2.1.21

2 years ago

2.1.8

2 years ago

2.1.7

2 years ago

2.1.20

2 years ago

2.1.0

2 years ago

2.1.9

2 years ago

2.1.17

2 years ago

2.1.14

2 years ago

2.1.12

2 years ago

2.1.13

2 years ago

2.1.10

2 years ago

2.1.11

2 years ago

2.0.2

2 years ago

2.0.1

2 years ago

2.0.0

2 years ago

1.0.16

2 years ago

1.0.15

3 years ago

1.0.14

3 years ago

1.0.13

3 years ago

1.0.12

3 years ago

1.0.11

3 years ago

1.0.10

3 years ago

1.0.9

3 years ago

1.0.8

3 years ago

1.0.7

3 years ago

1.0.6

3 years ago

1.0.5

3 years ago

1.0.4

3 years ago

1.0.3

3 years ago

1.0.2

3 years ago

1.0.1

3 years ago

1.0.0

3 years ago