Crawler Packages

A web crawler/spider

Asynchronous control flow wrapper to crawl websites

NodeJS Crawler for Twitter

A util tool

crawler prser core

here is a package for crawling and downloading videos from youtube

Web Download CLI

Automatically extracts structured information from webpages

A simple webcrawler that prints out the URLs of the pages it encounters. Runs in parallel, up to a limit you specify.

Fetches a page based on the configuration. Handles JSON and jQuery types of request.

Scraplink library, for scraping links and assets url from a webpage

Scrapew Node SDK

generalized scraping using a single config object for any site that can be scraped without a headless browser

손으로 모래를 모아 모래성을 쌓듯 한손 한손 데이터를 긁어모아서 만든 숭실대학교 데이터 크롤러

A crawler

nodejs lightweight scraper/crawler

Distributed web crawler powered by Headless Chrome

casperjs server for scraping, with it you can simple write the script file , it can be executed in the browser and response you the result you expect. it can load ajax content as you are using browser.

Send custom response to Facebook and Twitter agents