schabbi-webscraper
Lightweight and easy to use crawling solution for websites.
1.2.2 • Published 2 years ago
Webcrawler Packages
redis-web-crawler
Web Crawler to create directed graph of links among connected sites. Runs with Node.js and stores data with Redis
readme-crawler
Download README files from GitHub repository links
sitemapper_mos
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
@public-square/sitemapper
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
@context-labs/sitemapper
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
@drorgl/sitemapper
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
@myagizmaktav/sitemapper
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
@rodrigogs/spiderman
Yet to describe
@qbbsh/sitemapper
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
@mastixmc/sitemapper
Parser for XML Sitemaps to be used with Robots.txt and web crawlers. (Extended version by mastixmc)
@raing3/sitemapper
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
@jarronb/site-mapper
Site mapper utilizing puppeteer to generate a xml sitemap