Crawler Packages

crawl youtube without api key (search videos channels or get all channel/playlist's videos)

Extract emails from text and also from a site page

A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.

Verify that a request is from Apple crawlers using DNS verification steps

Verify that a request is from Twitter crawlers using DNS verification steps

Verify that a request is from Bing crawlers using Bing's DNS verification steps

Verify that a request is from Baidu crawlers using Baidu's DNS verification

Verify that a request is from DuckDuckBot, the Web crawler for DuckDuckGo

Verify that a request is from Google crawlers using Google's DNS verification steps

This plugin links your Netlify site with Algolia's Crawler. It will trigger a crawl on each successful build.

jarvis crawler core

The unofficial HLTV Node.js API

Web crawler for Node.js

An easy-to-use Node web crawler storing cookies, following redirects, traversing pages and submitting forms.

Simple crawler for Single Page Applications

A library for checking basic SEO signals of a website

Crawler(robots) decision middleware for Express

Unofficial API lib for Nintendo Switch eShop game listing and pricing information.

crawler tool

模仿scrapy的node爬虫框架