isbot
🤖/👨🦰 Recognise bots/crawlers/spiders using the user agent string.
5.1.4 • Published 17 days ago
Crawlers Packages
robot-directives
Parse robot directives within HTML meta and/or HTTP headers.
sitemapper
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
simplecrawling
Crawler made simple
planisphere
A straightforward sitemap generator written in TypeScript.
bot-detector
It uses the user-agents.org xml file for detecting bots.
crawler-js
Opensource Framework Crawler in Node.js
robots-txt-component
Lightweight robots.txt parsing component without any external dependencies for Node.js.
@cabbiepete/sitemapper
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
isbot-bundle
isbot Bundle UMD
hidemyemail
A jQuery plugin that helps you to hide your email on your page and prevent crawlers to get it!
@crawlee/utils
A set of shared utilities that can be used by crawlers
@ibragim64/robots-txt-component
Lightweight robots.txt parsing component without any external dependencies for Node.js.
sitemapper_mos
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
@public-square/sitemapper
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
@jarronb/site-mapper
Site mapper utilizing puppeteer to generate a xml sitemap
@javivelasco/isbot
🤖 detect bots/crawlers/spiders via the user agent.
@mastixmc/sitemapper
Parser for XML Sitemaps to be used with Robots.txt and web crawlers. (Extended version by mastixmc)
@microlink/ua
A simple redis primitives to incr() and top() user agents
@myagizmaktav/sitemapper
Parser for XML Sitemaps to be used with Robots.txt and web crawlers