Crawler | npm.io

Headless Chrome abstraction to simplify the interaction with the browser. It may be used for crawling sites, test automation, etc

headless chrome chromium headless-browser phantom-js nightmare casper horseman crawler test-automation

4.3.10 • Published 8 years ago

Using differents rules, try to know if a user agent string comes from a spider

bot user-agent crawler spider

2.0.1 • Published 10 years ago

[![npm](https://img.shields.io/npm/v/recrawl.svg)](https://www.npmjs.com/package/recrawl) [![ci](https://github.com/aleclarson/recrawl/actions/workflows/release.yml/badge.svg)](https://github.com/aleclarson/recrawl/actions/workflows/release.yml) [![codeco

fs crawl crawler directory readdir glob globby recrawl tiny

2.2.1 • Published 4 years ago

HTTP request module customized for crawlers.

crawler request http pdf-crawler pdf2text pdf-to-text pdf2json axios pdf pdf-parser

1.2.2 • Published 8 years ago

Beautiful-dom is a lightweight library that mirrors the capabilities of the HTML DOM API needed for parsing crawled HTML/XML pages. It models the methods and properties of HTML nodes that are relevant for extracting data from HTML nodes. It is written in

dom html xml parser crawler

1.0.9 • Published 4 years ago

use google play protobuf api in node

google play store play api crawler

4.5.0 • Published 6 years ago

Keep Bots Away From Your Express App

node express bot crawler protection block googlebot facebookexternalhit

1.0.5 • Published 6 years ago

Dependency free module for scraping and crawling websites using [ProxyCrawl](https://proxycrawl.com) API

scraping crawling scraper scrape crawler proxycrawl scraping-websites scraping-framework proxycrawl-api leads

2.5.0 • Published 3 years ago

Web scraper for NodeJS

web scraper crawler html xml dom parser

1.1.10 • Published 6 years ago

A web crawler for Nodejs.

crawler crawling spider spidering scraping scraper robot bot cheerio

0.8.2 • Published 11 years ago

Extract the article list from its raw news HTML

crawler scraper spider indexer html cheerio news headlines articles

1.0.6 • Published 3 years ago

A simple email extractor for obfuscated emails.

email email-address extractor obfuscated obfuscated-email obfuscated-emails parse regex obfuscator obfuscation

1.1.3 • Published 8 years ago

Priority based Semantic Web Crawler.

semantic-crawler pdf-crawler text-crawler priority priority-crawler scraper crawling spider scraper scraping

0.0.2 • Published 8 years ago

An `URL` parser for crawling purpose.

crawler-url-parser url-parser extract-url url-parse is-parent-url is-child-url url parser parse crawler

2.0.5 • Published 8 years ago

NexPloit CLI is a CLI tool that can initialize, stop, polling and maintain scans in NeuraLegions solutions (Such as NexPloit).

nexploit cli api nexploit-cli cyber-security security har oas crawler nexmock

9.2.0 • Published 2 years ago

A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.

robots txt robots.txt parser crawler spider bot robotstxt scraper

2.0.3 • Published 3 years ago

Simple, lightweight and expressive web scraping with Node.js

web html css selector scraping scrape scraper scrapy crawler

0.5.0 • Published 5 years ago

Verify that a request is from Google using Google's recommended DNS verification steps

googlebot google crawler bot

0.1.3 • Published 2 years ago

linkedin scraper for 2020 website

linkedin scraper crawler

1.0.21 • Published 5 years ago

A web scraper for NodeJs

scraper web scraper crawler web crawler web scraper web-crawler web-scraper nodejs node

6.1.3 • Published 3 years ago