@typicalninja21/puppeteer-fetch NPM

puppeteer-fetch

node-fetch but with puppeteer

This is for a personal project, but is public for others to view/use

Install

npm i @typicalninja21/puppeteer-fetch

Usage

Simple Example

const fetch = require('@typicalninja21/puppeteer-fetch');


(async () => {
    const response = await fetch('https://www.npmjs.com/package/node-fetch').catch(console.log)

    // we use .catch(console.log) above, if error happens, response = undefined
    if(!response) return;

    console.log(response)
})()

Response:

The following object will be returned with the fetch()

{
    body: '<body of the site, usually html>',
    url: '<the url visited>,
    headers: <object, headers>,
    status: <status code, ex: 200>,
    ok: <status code is above 200 and below 299>,
    json: <function(), returns the body JSON.parsed || error>,
    text: <function(), returns the body || error>,
    _request: {
           headers: <object, headers>,
           method: <fetch method, "get">
    }
}

Getting Json Data

const fetch = require('@typicalninja21/puppeteer-fetch');


(async () => {
    const response = await fetch('https://jsonplaceholder.typicode.com/todos/1').catch(console.log)

    // we use .catch(console.log) above, if error happens, response = undefined
    if(!response) return;

// make sure to await it as it returns a promise
    console.log(await response.json())
})()

Response

The following object will be returned with the fetch(), (it is json data from jsonplaceholder.typicode.com)

{ userId: 1, id: 1, title: 'delectus aut autem', completed: false }

<response>.text() works the same way, but returns a string instead of json data, and does "NOT" error if returned content is not a valid json

Options

Default options are:

headers: {}

userAgent: ''

onPage: null

waitTimeOut: 0

puppeteerLaunchOptions: {}

puppeteerArgs: []

timeOut: 9000

Options Explained

headers - Headers to send with the request (duh)

userAgent - userAgent to be set as

onPage - want to run a function when the page get created? then use this, should be a async function

waitTimeOut - if for some reason, you want the module to stay on the page for a bit use this, should be in, milliseconds

puppeteerLaunchOptions - options for puppeteer.launch

puppeteerArgs - options to include args option in puppeteer.launch (as the package itself has 2 default args '--no-sandbox', '--disable-setuid-sandbox', will be merged with this)

timeOut - will exit after this time, if the request doesn't resolve

Plugins

The package itself doesn't have any plugins, but the package uses puppeteer-extra instead of normal puppeteer to enable plugin support

Example

Single plugin example

using Plugin puppeteer-extra-plugin-stealth as a example

const fetch = require('@typicalninja21/puppeteer-fetch');
const StealthPlugin = require('puppeteer-extra-plugin-stealth');

fetch.extend(StealthPlugin());

(async () => {

})()

<fetch>.extend Returns fetch(), and extend is a function attached to fetch() so you can do something like below

Like this

This is also how to use multiple plugins, at once

const fetch = require('@typicalninja21/puppeteer-fetch');
const StealthPlugin = require('puppeteer-extra-plugin-stealth');
const AdblockerPlugin = require('puppeteer-extra-plugin-adblocker');


fetch.extend(StealthPlugin()).extend(AdblockerPlugin({ blockTrackers: true }))('https://jsonplaceholder.typicode.com/todos/1').then((res) => {
    console.log(res)
}).catch((err) => {
    console.log(err)
});

Thats it

ya thats all for now, you can read some of the things i suggest you to use/do when using this

suggestions

use puppeteer-extra-plugin-recaptcha in a case where the webpage give you captcha, this will come in handy
not needed but works well, puppeteer-extra-plugin-stealth
using proxies when needed, you can use proxy by using the plugin puppeteer-extra-plugin-proxy
using the normal non puppeteer node-fetch, this is great but is pretty slow sometimes, and overkill for simple apis that are just json stuff, you would use this when you would need the page to load javascript or any kind of script, as normal fetch would not wait for javascript to run before resolving

Support

3 years ago

3 years ago