Crawl Packages

Lightweight crawler written in TypeScript using ES6 generators.

Lightweight crawler written in TypeScript using ES6 generators.

Website extensions for the Sajari API. Automatically index site content, add user profiles, render search and recommendations, etc.

Magnetic is a tool that makes it easy to fetch web pages that may have dynamic content and render them to static HTML.

A node.js module to crawl product reviews from Amazon.

Crawl websites for accessibility issues from the command line.

A 2nd generation spider to crawl any article site, automatic reading title and content.

A CLI tool for validating pages while crawling a website.

A web crawler engine

Magnetic example - Magnetic is an Express middleware that makes it easy for you to add automatic server-side static rendering to your dynamic single-page app.

Magnetic Express is an Express middleware for Magnetic - a tool that makes it easy for you to add automatic server-side static rendering to your dynamic single-page app.

node.js web crawler

Fetch website with all the resources/responses/requests to local files, using puppeteer.

web scraping made simple

A mirroring plugin for webcheck

A webcheck plugin to raise wait paramerter by delay

A web crawler/spider

NodeJS Crawler for Twitter

Tutorial for web scraping / crawling with Node.js.

A node.js module to scrape player points from Repubblica Sport match report