# fetchfox

> AI based web scraping library

Latest version **0.0.40** (published 2025-03-26) · MIT license · 0 weekly downloads

## Install

```sh
npm install fetchfox
pnpm add fetchfox
yarn add fetchfox
bun add fetchfox
```

## Health

**Score 30/100 (F)** — status: maintenance-mode.

Positive: esm support; no vulnerabilities.

Warnings: low downloads; no types; large bundle; pre 1.0.

Negative: stale; low maintenance score.

## Facts

| | |
|---|---|
| Version | 0.0.40 |
| Published | 2025-03-26 |
| First published | 2024-10-05 |
| Weekly downloads | 0 |
| License | MIT |
| TypeScript types | none |
| Module format | ESM + CommonJS |
| Dependencies | 20 |
| Unpacked size | 200.9 MB |
| Known vulnerabilities | 0 |
| Install scripts | no |
| GitHub stars | 159 |
| Maintainers | ortutay |

## Links

- npm: https://www.npmjs.com/package/fetchfox
- Repository: https://github.com/fetchfox/fetchfox
- Issues: https://github.com/fetchfox/fetchfox/issues
- npm.io page: https://npm.io/package/fetchfox

## Dependencies (20)

- [chalk](https://npm.io/package/chalk.md) ^5.3.0
- [openai](https://npm.io/package/openai.md) ^4.58.1
- [pretty](https://npm.io/package/pretty.md) ^2.0.0
- [ioredis](https://npm.io/package/ioredis.md) ^5.4.1
- [p-queue](https://npm.io/package/p-queue.md) ^8.0.1
- [groq-sdk](https://npm.io/package/groq-sdk.md) ^0.7.0
- [loglevel](https://npm.io/package/loglevel.md) ^1.9.2
- [tiktoken](https://npm.io/package/tiktoken.md) ^1.0.17
- [p-timeout](https://npm.io/package/p-timeout.md) ^6.1.4
- [node-fetch](https://npm.io/package/node-fetch.md) ^3.3.2
- [playwright](https://npm.io/package/playwright.md) ^1.48.0
- [whatwg-url](https://npm.io/package/whatwg-url.md) ^14.0.0
- [url-polyfill](https://npm.io/package/url-polyfill.md) ^1.1.12
- [node-html-parser](https://npm.io/package/node-html-parser.md) ^6.1.13
- [playwright-extra](https://npm.io/package/playwright-extra.md) ^4.3.6
- [@anthropic-ai/sdk](https://npm.io/package/@anthropic-ai/sdk.md) ^0.27.2
- [@aws-sdk/client-s3](https://npm.io/package/@aws-sdk/client-s3.md) ^3.669.0
- [@google/generative-ai](https://npm.io/package/@google/generative-ai.md) ^0.19.0
- [loglevel-plugin-prefix](https://npm.io/package/loglevel-plugin-prefix.md) ^0.8.4
- [@aws-sdk/s3-request-presigner](https://npm.io/package/@aws-sdk/s3-request-presigner.md) ^3.686.0

## Recent versions

- 0.0.40 (latest) — 2025-03-26
- 0.0.39 — 2025-03-18
- 0.0.38 — 2025-02-11
- 0.0.37 — 2025-01-07
- 0.0.36 — 2024-12-30
- 0.0.35 — 2024-12-11
- 0.0.34 — 2024-11-11
- 0.0.33 — 2024-11-04
- 0.0.32 — 2024-10-29
- 0.0.31 — 2024-10-23
- 0.0.30 — 2024-10-23
- 0.0.28 — 2024-10-21
- 0.0.27 — 2024-10-15
- 0.0.26 — 2024-10-14
- 0.0.25 — 2024-10-13
- … 16 more at https://npm.io/package/fetchfox/versions

## README

<div align="center">
  <h1>FetchFox</h1>
  <div>
    <img width="515" alt="Screenshot 2024-10-13 at 1 14 28 AM" src="https://github.com/user-attachments/assets/290d26c5-f0a0-48ba-985a-8052ad23f252">
  </div>

<p>FetchFox is an AI powered scraping, automation, and data extraction library.</p>
  
<p>It can scrape data from any webpage using just plain English. It is made by the developers of the <a href="https://fetchfox.ai">FetchFox AI scraper</a>.</p>
</div>

<div align="center">
  
<a href="https://twitter.com/FetchFoxAI"><img src="https://img.shields.io/twitter/follow/FetchFoxAI?style=social"></a> [![GitHub stars](https://img.shields.io/github/stars/fetchfox/fetchfox.svg?style=social&label=Star)](https://github.com/fetchfox/fetchfox) <a href="https://badge.fury.io/js/fetchfox"><img src="https://badge.fury.io/js/fetchfox.svg" alt="npm version" height="18"></a> <a href="https://discord.gg/mM54bwdu59"><img src="https://img.shields.io/discord/1180618526436888586?label=discord&logo=discord&logoColor=white&style=flat"></a>

</div>

# Getting started

Install the package and playwright:

```bash
npm i fetchfox
npx playwright install-deps
npx playwright install
```

Then use it. Here is the callback style:

```javascript
import { fox } from 'fetchfox';

const workflow = await fox
  .init('https://pokemondb.net/pokedex/national')
  .extract({ name: 'Pokemon name', number: 'Pokemon number' })
  .limit(3)
  .plan();

const results = await workflow
  .run(null, (delta) => { console.log(delta.item) });
  
for (const item of results.items) {
  console.log('Item:', item);
}
```

If you prefer, you can use the streaming style:

```javascript
import { fox } from 'fetchfox';

const stream = fox
  .init('https://pokemondb.net/pokedex/national')
  .extract({ name: 'Pokemon name', number: 'Pokemon number' })
  .stream();

for await (const delta of stream) {
  console.log(delta.item);
}
```

## Following URLs

You'll often want to scrape over multiple levels. You can do this using the `url` field. If you extract a `url` field, FetchFox will follow that URL on the next step.

For example, you can get HP and attack on the second page of the Pokedex:

```javascript
const workflow = await fox
  .init('https://pokemondb.net/pokedex/national')
  .extract({ 
    url: 'URL of pokemon profile', 
    name: 'Pokemon name', 
    number: 'Pokemon number'
  })
  .extract({ 
    hp: 'Pokemon HP', 
    attack: 'Pokemon attack power', 
  })
  .limit(3)
  .plan();

const results = await workflow
  .run(null, (delta) => { console.log(delta.item) });
  
for (const item of results.items) {
  console.log('Item:', item);
}
```

This scraper will start at https://pokemondb.net/pokedex/national, and then go to detail pages like https://pokemondb.net/pokedex/pikachu to get the HP and attack values.

## Enter your API key

You'll need to give an API key for the AI provider you are using, such as OpenAI. There are a few ways to do this.

The easiest option is to set the `OPENAI_API_KEY` environment variable. This will get picked up by the FetchFox library, and all AI calls will go through that key. To use this option, run your code like this:

```bash
OPENAI_API_KEY=sk-your-key node index.js
```

Alternatively, you can pass in your API key in code, like this:

```javascript
import { fox } from 'fetchfox';

const results = await fox
  .config({ ai: ['openai:gpt-4o-mini', { apiKey: 'sk-your-key' }]})
  .init('https://pokemondb.net/pokedex/national')
  .extract({ name: 'Pokemon name', number: 'Pokemon number' })
  .limit(3)
  .run();
```
 
This will use OpenAI's `gpt-4o-mini` model, and the API key you specify. You can also use OpenRouter to access AI models from other providers:

```javascript
const results = await fox
  .config({ ai: ['openrouter:google/gemini-flash-1.5', { apiKey: 'your-openrouter-key' }]})
  .init('https://pokemondb.net/pokedex/national')
  .extract({ name: 'Pokemon name', number: 'Pokemon number' })
  .limit(3)
  .run();
```

Choose the AI model that best suits your needs.

The following providers are supported

* __OpenAI__: Model strings are `openai:...`, for example `openai:gpt-4o`
* __Google__: Model strings are `google:...`, for example `google:gemini-1.5-flash`
* __OpenRouter__: Model strings are `openrouter:...`, for example `openrouter:anthropic/claude-3.5-haiku`


By default, FetchFox uses OpenAI's `gpt-4o-mini` model. We've found this model to provide a good tradeoff between cost, runtime, and accuracy. We have a [public benchmarks dashboard](http://dashboard.fetchfox.ai/) where you can review performance data on recent commits.

---
_Source: https://npm.io/package/fetchfox · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._