Crawling Packages

crawler service

Crawler Second-system effect,the second development

A simple web crawler

Crawler is a web spider written with Nodejs. It gives you the full power of jQuery on the server to parse a big number of pages as they are downloaded, asynchronously. Scraping should be simple and fun!

Web crawler

crawlx is a Lightweight web crawler with powerful plugins!

make web scraping easy

Node.js module for crawling the web

A small package to crawl a site and return a redirect template. This is helpful for migration from one to another website with different url schemes.

Simple and powerful crawler. It scraps content and collects links from websites using request or phantomjs. The whole magic and simplicity is behind configuration.

A version of goldwasher that runs as a module on AWS Lambda.

Plugin for goldwasher to add needle for easy HTTP requests.

Scheduled goldwasher requests, using goldwasher-needle and node-schedule.

An interactive Command-Line Interface Build in NodeJS for downloading a single image or multiple images to disk from URL

An API to get data off of IMDB using Puppeteer.

A plugin for Hapi.js to run goldwasher as a scraping API on the web.

Distributed web crawler powered by Headless Chrome

This extracts the top five news metadata from NAVER headlines.

Fast and lightweight web crawler with built-in cheerio, xml and json parser.

PhantomJS and JSDOM based crawling tool. Used PhantomJS for full load of asynchronously-loaded resources and JSDOM for quick crawls. Allows custom [tough-cookie](https://www.npmjs.com/package/tough-cookie) insertion. Refer to [cheerio](https://www.npmj