Scraperkit NPM

Scraperkit

Common utility functions that regularly need in web scrapper apps.

getHTML

const { getHTML } = require('scraperkit');
const res = await getHTML('https://example.com', {
  prerender: true,
  puppeteerLaunchConfig: {
    executablePath: `/Applications/Google Chrome.app/Contents/MacOS/Google Chrome`
  }
});
console.log(res);

getSelectors

const htmlContent = `<html>...</html>`;
const selectors = getSelectors(htmlContent, ['meta', 'h1']);

TODO

Add proxy support for getHTML
Make it easy to debug issues

Resources

https://github.com/axios/axios/issues/925

@sindresorhus/is axios cheerio ow puppeteer-core

@infinitebrahmanuniverse/nolb-scra @everything-registry/sub-chunk-2718

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago