2.2.2 • Published 11 months ago
site-archive v2.2.2
Site Archive
This is a site crawler designed to track visual and content differences between crawls.
Example usage:
import Crawler from "site-archive";
import path from "path";
const entryPointRaw = process.argv[2];
const crawler = new Crawler(entryPointRaw, path.resolve('.', 'captures'), 1, console.log);
await crawler.crawl();
yarn run start https://yoursite.com
Options:
Default options:
let options = {
ignoreHead: false, // If true skip any urls found in the <head> element.
screenshotsOnly: false, // If true doesn't save any page HTML.
screenshotSizes: [], // An array of {width: number, height: number} for screenshots to take.
htmlOnly: false, // Only attempt to follow links that might be HTML documents.
logLevel: 2, // The detail level of messages to display. (1, 2, or 3 with 1 being every single message generated).
onEvent: (msg) => console.log(msg), // Function that handles messaging from the crawler.
htmlTypes: ['html', 'htm', 'xhtml', 'asp', 'aspx', 'shtml', 'dhtml', 'php', 'php5', 'jsp'], // Which file extensions (in addition to no-extension) are considered to be possible HTML links.
ignoreQueryString: false, // If true, doesn't consider query strings to be part of a URL.
ignoreAnchors: false, // If true, doesn't consider anchors to be part of a URL.
selectorsToRemove: [], // An array of selectors for elements to be removed from the page before taking screenshots.
timeout: 30000, // How long to wait for a page to finish load before timing out.
redirect: 'follow', // How to treat HTTP redirects. (https://chromestatus.com/feature/4614142321229824)
}
Status:
Still very much a work in progress
2.2.1
1 year ago
2.2.2
11 months ago
2.2.0
1 year ago
2.1.27
1 year ago
2.1.28
1 year ago
2.1.18
2 years ago
2.1.19
2 years ago
2.1.2
2 years ago
2.1.1
2 years ago
2.1.25
2 years ago
2.1.4
2 years ago
2.1.3
2 years ago
2.1.23
2 years ago
2.1.6
2 years ago
2.1.24
2 years ago
2.1.5
2 years ago
2.1.21
2 years ago
2.1.8
2 years ago
2.1.7
2 years ago
2.1.20
2 years ago
2.1.0
2 years ago
2.1.9
2 years ago
2.1.17
2 years ago
2.1.14
2 years ago
2.1.12
2 years ago
2.1.13
2 years ago
2.1.10
2 years ago
2.1.11
2 years ago
2.0.2
2 years ago
2.0.1
2 years ago
2.0.0
2 years ago
1.0.16
2 years ago
1.0.15
3 years ago
1.0.14
3 years ago
1.0.13
3 years ago
1.0.12
3 years ago
1.0.11
3 years ago
1.0.10
3 years ago
1.0.9
3 years ago
1.0.8
3 years ago
1.0.7
3 years ago
1.0.6
3 years ago
1.0.5
3 years ago
1.0.4
3 years ago
1.0.3
3 years ago
1.0.2
3 years ago
1.0.1
3 years ago
1.0.0
3 years ago