open-graph-scraper-bypasshtmlcheck v4.11.0-1

openGraphScraper

A simple node module for scraping Open Graph and Twitter Card info off a site. For browser usage, we recommend using ky to make the requests(or a backend service) then pass in the html into open-graph-scraper using the html option.

Installation

npm install open-graph-scraper --save

Usage

Callback Example:

const ogs = require('open-graph-scraper');
const options = { url: 'http://ogp.me/' };
ogs(options, (error, results, response) => {
  console.log('error:', error); // This returns true or false. True if there was an error. The error itself is inside the results object.
  console.log('results:', results); // This contains all of the Open Graph results
  console.log('response:', response); // This contains the HTML of page
});

Promise Example:

const ogs = require('open-graph-scraper');
const options = { url: 'http://ogp.me/' };
ogs(options)
  .then((data) => {
    const { error, result, response } = data;
    console.log('error:', error);  // This returns true or false. True if there was an error. The error itself is inside the results object.
    console.log('result:', result); // This contains all of the Open Graph results
    console.log('response:', response); // This contains the HTML of page
  })

Results JSON

Check the return for a success flag. If success is set to true, then the url input was valid. Otherwise it will be set to false. The above example will return something like...

{
  ogTitle: 'Open Graph protocol',
  ogType: 'website',
  ogUrl: 'http://ogp.me/',
  ogDescription: 'The Open Graph protocol enables any web page to become a rich object in a social graph.',
  ogImage: {
    url: 'http://ogp.me/logo.png',
    width: '300',
    height: '300',
    type: 'image/png'
  },
  requestUrl: 'http://ogp.me/',
  success: true
}

Options

Note: open-graph-scraper uses got for requests and most of got's options should work as open-graph-scraper options.

Custom Meta Tag Example

const ogs = require('open-graph-scraper');
const options = {
  url: 'https://github.com/jshemas/openGraphScraper',
  customMetaTags: [{
    multiple: false, // is there more than one of these tags on a page (normally this is false)
    property: 'hostname', // meta tag name/property attribute
    fieldName: 'hostnameMetaTag', // name of the result variable
  }],
};
ogs(options)
  .then((data) => {
    const { error, result, response } = data;
    console.log('hostnameMetaTag:', result.hostnameMetaTag); // hostnameMetaTag: github.com
  })

Proxy Example

Look here for more info on how to use proxies.

const ogs = require('open-graph-scraper');
const tunnel = require('tunnel');
const options = {
  url: 'https://whatismyipaddress.com/',
  timeout: 15000,
  agent: {
    // setting proxy agent for https requests
    https: tunnel.httpsOverHttp({
      // test proxies can be found here: https://hidemy.name/en/proxy-list/?country=US&type=h#list or http://free-proxy.cz/en/proxylist/country/US/https/ping/all
      proxy: {
        host: 'proxy_ip',
        port: proxyPort,
        rejectUnauthorized: false,
      }
    })
  }
};
ogs(options)
  .then((data) => {
    const { error, result, response } = data;
    console.log('response:', response); // you should see the proxy IP in here
  })

User Agent Example

const ogs = require("open-graph-scraper");
const options = {
  url: "https://twitter.com/elonmusk/status/1364826301027115008",
  headers: {
    "user-agent": "Googlebot/2.1 (+http://www.google.com/bot.html)",
  },
};
ogs(options, (error, results) => {
  console.log("error:", error); // This returns true or false. True if there was an error. The error itself is inside the results object.
  console.log("results:", results); // This contains all of the Open Graph results
});

Tests

Then you can run the tests by running...

npm run test