Crawler Packages

Web crawler for Node and browsers

A fast and stable DHT crawler.

A nodejs cli tool for fetching Bing wallpapers regularly

Web crawler wrapper around puppeteer module to simply the crawling on ajax/java script enabled pages.

A simple crawler for retriving important data from web pages.

A simple in depth links crawler. You can easily collect all the links available on a website.

Identifies bots/crawlers

Flowesh is the non-cluster version of floodesh. It's a middleware based web spider which is lightweight and easy to maintain

This app will crawl and fully load a list of URLs or sitemap.xml(soon) using Puppeteer (aka headless Chromium). It's the ONLY crawler that (1) fully loads pages and (2) mimics browser HTTP headers to NGINX or Varnish. At the same time, it's optimized to n

'recursive fs and https crawlers'

Simple selenium web crawler

File System Crawler helps read the file system info for any user selected folder. It also helps extract text from files including pdf files. It can also perform OCR on image files and extract legible texts from them. Support for reading many other popular

nodejs crawler/spider which provides a simple interface for crawling the Web

A crawler framework base on jsdom.

Wordpress posts crawler for node.js

Node.js crawler frame.

subscrape is a very simple tool to download the latest images from a specific sub reddit.

middleware based lightweight crawler framework

A tibia crawler module for Node.

nodejs to crawler tieba