Crawler Packages

Web crawler to use as API

A light weight nodejs library to helps you grab the specified content on any web page on internet.

simple polite crawling of the web.

A scraping crawler

novel downloader for node-novel style , include site ( dmzj / wenku8 / syosetu / ...etc )

Paginator enriches ability to paginate over the pages in Goose Parser

Finds broken links and resources on websites

And ExpressJS middleware for detecting search engine crawlers/spiders and handling crawler-specific logic such as additional logging for requests from spiders

Parser for Filmaffinity site

http-cache driver for x-ray

A @0y0/scraper expansion pack.

Crawls a website, gets PageSpeed Insights data for each page, and exports an HTML report

Get Aliexpress product details in JSON

Crawl the content of any instagram public page with no token or login

🚀 An easy-to-handle Node.js scraper that allow you to scrape them all in a record time.

Fetch pure HTML from a webserver and save it to disk

A crawler crawling rental information, based Nodejs

A web crawler module designed to scarp data from Ptt.

a ptt crawler

Generate sitemap just throw any link.