Crawler Packages

A lightweight scraper for SleepCycle's online SleepSecure data.

This CLI tool will find same domain urls in a web page and requesting them to find even more urls until server crash (or at the end of benchmark). It is used to test maximun capacity of server or finding for glitches that users might encounter.

A robust GitHub API crawler that walks a queue of GitHub entities retrieving and storing their contents.

Multi environment web page parser

Follows and collects breadcrumbs accross the web

headless chromium library for takaing website screenshots.

Rate-limiting/throttling for limit-request-native

Bower dependency graph crawler

crawler parser

crawler prser core

Crawl website for ads.txt entries.

Builds parametrized Amazon URIs

Crawls product questions from Amazon.

Allocine web site crawling

Simple assets crawler that allows you to get all assets from a particular website via command-line

Legacy Apifier runtime for JavaScript

A crawler

Aragog web scraping framework client

API that crawls through both Google Play Store and Apple App Store

简单爬虫