# crawler-user-agents > This repository contains a list of of HTTP user-agents used by robots, crawlers, and spiders as in single JSON file. Latest version **1.60.0** (published 2026-08-07) · MIT license · 0 weekly downloads ## Install ```sh npm install crawler-user-agents pnpm add crawler-user-agents yarn add crawler-user-agents bun add crawler-user-agents ``` ## Health **Score 70/100 (B)** — status: active. Positive: has types; esm support; no vulnerabilities; has provenance; recently updated; high maintenance score. Warnings: low downloads. ## Facts | | | |---|---| | Version | 1.60.0 | | Published | 2026-08-07 | | First published | 2015-06-23 | | Weekly downloads | 0 | | License | MIT | | TypeScript types | bundled | | Module format | ESM + CommonJS | | Dependencies | 0 | | Unpacked size | 587.8 KB | | Known vulnerabilities | 0 | | Install scripts | no | | Provenance | attested (GitHub Actions) | | GitHub stars | 1406 | | Author | Martin Monperrus | | Maintainers | monperrus | ## Links - npm: https://www.npmjs.com/package/crawler-user-agents - Repository: https://github.com/monperrus/crawler-user-agents - Homepage: https://github.com/monperrus/crawler-user-agents#readme - Issues: https://github.com/monperrus/crawler-user-agents/issues - npm.io page: https://npm.io/package/crawler-user-agents ## Recent versions - 1.60.0 (latest) — 2026-08-07 - 1.59.0 — 2026-08-07 - 1.58.0 — 2026-08-07 - 1.57.0 — 2026-08-07 - 1.56.0 — 2026-07-02 - 1.55.0 — 2026-07-02 - 1.54.0 — 2026-06-24 - 1.53.0 — 2026-06-23 - 1.52.0 — 2026-06-19 - 1.51.0 — 2026-06-17 - 1.50.0 — 2026-06-01 - 1.49.0 — 2026-06-01 - 1.48.0 — 2026-06-01 - 1.47.0 — 2026-06-01 - 1.46.0 — 2026-06-01 - … 213 more at https://npm.io/package/crawler-user-agents/versions ## README # crawler-user-agents This repository contains a list of of HTTP user-agents used by robots, crawlers, and spiders as in single JSON file. * NPM package: * Go package: * PyPi package: Each `pattern` is a regular expression. It should work out-of-the-box wih your favorite regex library. ## Sponsor 💼 **Using crawler-user-agents in a commercial product?** This package is free to use, but it takes real time to maintain and expand. If it's providing value (and it probably is), please consider [sponsoring at the commercial tier](https://github.com/sponsors/monperrus?frequency=recurring). It keeps the project alive and actively maintained. Your company can afford it. 🙏 ## Install ### Direct download Download the [`crawler-user-agents.json` file](https://raw.githubusercontent.com/monperrus/crawler-user-agents/master/crawler-user-agents.json) from this repository directly. ### Javascript crawler-user-agents is deployed on npmjs.com: To use it using npm or yarn: ```sh npm install --save crawler-user-agents # OR yarn add crawler-user-agents ``` In Node.js, you can `require` the package to get an array of crawler user agents. ```js const crawlers = require('crawler-user-agents'); console.log(crawlers); ``` ### Python Install with `pip install crawler-user-agents` Then: ```python import crawleruseragents if crawleruseragents.is_crawler("Googlebot/"): # do something ``` or: ```python import crawleruseragents indices = crawleruseragents.matching_crawlers("bingbot/2.0") print("crawlers' indices:", indices) print( "crawler's URL:", crawleruseragents.CRAWLER_USER_AGENTS_DATA[indices[0]]["url"] ) ``` Note that `matching_crawlers` is much slower than `is_crawler`, if the given User-Agent does indeed match any crawlers. ### Go Go: use [this package](https://pkg.go.dev/github.com/monperrus/crawler-user-agents), it provides global variable `Crawlers` (it is synchronized with `crawler-user-agents.json`), functions `IsCrawler` and `MatchingCrawlers`. Example of Go program: ```go package main import ( "fmt" "github.com/monperrus/crawler-user-agents" ) func main() { userAgent := "Mozilla/5.0 (compatible; Discordbot/2.0; +https://discordapp.com)" isCrawler := agents.IsCrawler(userAgent) fmt.Println("isCrawler:", isCrawler) indices := agents.MatchingCrawlers(userAgent) fmt.Println("crawlers' indices:", indices) fmt.Println("crawler's URL:", agents.Crawlers[indices[0]].URL) } ``` Output: ``` isCrawler: true crawlers' indices: [237] crawler' URL: https://discordapp.com ``` ## Contributing I do welcome additions contributed as pull requests. The pull requests should: * contain a single addition * specify a discriminant relevant syntactic fragment (for example "totobot" and not "Mozilla/5 totobot v20131212.alpha1") * contain the pattern (generic regular expression), the discovery date (year/month/day) and the official url of the robot * result in a valid JSON file (don't forget the comma between items) Example: { "pattern": "rogerbot", "addition_date": "2014/02/28", "url": "http://moz.com/help/pro/what-is-rogerbot-", "instances" : ["rogerbot/2.3 example UA"], "tags": ["seo"] } ## License The list is under a [MIT License](https://opensource.org/licenses/MIT). The versions prior to Nov 7, 2016 were under a [CC-SA](http://creativecommons.org/licenses/by-sa/3.0/) license. ## Related work There are a few wrapper libraries that use this data to detect bots: * [Voight-Kampff](https://github.com/biola/Voight-Kampff) (Ruby) * [isbot](https://github.com/Hentioe/isbot) (Ruby) * [crawlers](https://github.com/Olical/crawlers) (Clojure) * [isBot](https://github.com/omrilotan/isbot) (Node.JS) Other systems for spotting robots, crawlers, and spiders that you may want to consider are: * [Crawler-Detect](https://github.com/JayBizzle/Crawler-Detect) (PHP) * [BrowserDetector](https://github.com/mimmi20/BrowserDetector) (PHP) * [browscap](https://github.com/browscap/browscap) (JSON files) --- _Source: https://npm.io/package/crawler-user-agents · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._