@tuplo/fletcher v2.19.9

@tuplo/fletcher

HTTP request library, focused on web scraping.

Install

$ npm install @tuplo/fletcher

# or with yarn
$ yarn add @tuplo/fletcher

Usage

Fetch a HTML page and parse it using cheerio.

import fetch from '@tuplo/fletcher';

const $page = await fetch.html('https://foo.com/page.html');
const heading = $page.find('body > h1');

Fetch a JSON file and parse it.

const { foo } = await fetch.json('https://foo.com/page.html');

Find a script on a page and evaluate it.

const { foo } = await fetch.script('https://foo.com/page.html', {
  scriptPath: 'script:nth-of-type(3)',
});

Find JSON-LD metadata on a page.

const [jsonld] = await fetch.jsonld('https://foo.com/page.html');

Work with the raw Response.

const res = await fetch.response('https://foo.com');
console.log(res.headers);
console.log(await res.text());

Work with Puppeteer for headless browser automation.

const client = fetch.create({
  browserWSEndpoint: 'ws://localhost:3000',
});
const $page = await client.browser.html('https://foo.com');
const { foo } = await client.browser.json('https://foo.com', /ajax-list/);

Options

API

fletcher(url: string, options?: FletcherOptions) => http.Response

Generic utility to return a HTTP Response

fletcher.html(url: string, options?: FletcherOptions) => Cheerio<AnyNode>

Requests a HTTP resource, parses it using Cheerio and returns its

const $page = await fletcher.html('https://foo.com/page.html');
const heading = $page.find('body > h1');

fletcher.script<T>(url: string, options?: FletcherOptions) => T

Requests a HTTP resource, finds a script on it, executes and returns its global context.

const { foo } = await fletcher.script('https://foo.com/page.html', {
	scriptPath: 'script:nth-of-type(3)',
});

fletcher.text(url: string, options?: FletcherOptions) => string

Requests a HTTP resource, returning it as a string

fletcher.cookies(url: string, options?: FletcherOptions) => CookieJar

Requests a HTTP resources, returning the cookies returned with it.

fletcher.json<T>(url: string, options?: FletcherOptions) => T

Requests a HTTP resource, returning it as a JSON object

fletcher.jsonld(url: string, options?: FletcherOptions) => unknown[]

Requests a HTTP resource, retrieving all the JSON-LD blocks found on the document

fletcher.response(url: string, options?: FletcherOptions) => Response

Requests a HTTP resource, returning the full HTTP Response object

fletcher.browser.html(url: string) => Cheerio<AnyNode>

Requests a HTTP resource using Puppeteer/Chrome, parses it using Cheerio and returns its.

fletcher.browser.json<T>(url: string, requestUrl: string | RegExp) => T

Requests a HTTP resource using Puppeteer/Chrome, intercepts a request made by that page and returns it as a JSON object

fletcher.create(options: FletcherOptions) => Object

Creates a new instance of fletcher with a custom config

const instance = fletcher.create({ headers: { foo: 'bar' } });
await instance.json('http://foo.com');