Crawl Packages

Fetch html with node-fetch.

Scrapoxy is a proxy for scrapers

Extract values out of deep structures using a safe and simple extraction language.

Scrapes/Crawls the logo from a provided url(s)/website for your Node.js applications.

get fetch parse crawl sitemap links recursively.

Get a website's text

A simple, RFC-compliant robots.txt parser

Spy on files

This is a function that accepts 3 arguments, "url", "tag" and "output", and writes to a file, in the "output" path, the content of an html "tag", relative to a specific "url".

galaxiat.serve.seo Node.js package allows you to easily cron crawl path that you want to have an html version (for dynamic rendering like React) without have to make SSR when request is passed.

Scrap and collect data from urls, html, json and stuff..

a web crawler based on crawlee, use file to cache result. Easy to maintain as Singleton Service.