httpclient-nodejs v1.3.0

httpclient-nodejs

Simple but powerful HTTP client for NodeJS.

HTTP Client utilises HTTP protocol to connect to the HTTP server which can be a web server, API or something else. After I noticed that there's no efficient HTTP Client for NodeJS this library was developed. It is made for the NodeJS environment and uses both HTTP and HTTPS.

Features

• URL auto correction
• auto detect secure https protocol
• handle 301 redirects
• retry on HTTP response error
• HTTP response is formatted nicely and returned as JS Promise so async/await can be used
• response is an array where elements are responses from every stage of 301 HTTP redirection
• response in JSON format for APIs
• rich formatted error responses
• decompress gzip responses automatically
• do not parse JavaScript in the HTML document
• no npm dependencies

Installation

$ npm install --save httplient-node

Example

/*** NodeJS script ***/
const { HttpClient } = require('httpclient-nodejs');

const getUrl = async () => {
  try {
    const opts = {
      encodeURI: false,
      encoding: 'utf8',
      timeout: 3000,
      retry: 2,
      retryDelay: 2100,
      maxRedirects: 3,
      headers: {
        'authorization': '',
        'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36',
        'accept': '*/*', // 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9'
        'cache-control': 'no-cache',
        'host': '',
        'accept-encoding': 'gzip',
        'connection': 'close', // keep-alive
        'content-type': 'text/html; charset=UTF-8'
      },
      debug: false
    };

    const hcn = new HttpClient(opts); // http client instance
    const answers = await hcn.ask('http://www.adsuu.com');
    console.log(answers);

  } catch (err) { throw err; }
};


getUrl().catch(console.error);

Other examples are in /tests/ folder.

API

constructor(opts:{encodeURI:boolean, timeout:number, retry:number, retryDelay:number, maxRedirects:number, headers:object})

• encodeURI Encode URI before request is sent. (false)
• timeout Close socket on certain period of time in milliseconds. Same as timeout in NodeJS HTTP library. (8000)
• retry When HTTP Client receives an error response it will try to send requests repeatedly. The retry number determines the max allowed retries. (3)
• retryDelay Time delay after each retry in milliseconds. (5500)
• maxRedirects When HTTP Client receives 301 in Header response it will try to send new request to redirected URL. Number maxRedirects determines max redirects allowed to prevent infinite loops. (3)
• headers Definition of HTTP Headers used in HTTP request. (see below)

headers::
{
  'authorization': '',
  'user-agent': `DEX8-SDK/${pkg_json.version} https://dex8.com`, // 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36'
  'accept': '*/*', // 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9'
  'cache-control': 'no-cache',
  'host': '',
  'accept-encoding': 'gzip',
  'connection': 'close', // keep-alive
  'content-type': 'text/html; charset=UTF-8'
}

askOnce(url, method = 'GET', body_obj)

Send one time HTTP/HTTPS request. Redirection is not handled. Response is a Promise so async/await can be used. hcn.askOnce('https://www.dummy-api.com/create', 'POST', {first_name: 'Saša'});

answer (HTTP response) is formatted as simple object
------------------------------------------------------------
{
  requestURL: 'http://www.adsuu.com',
  requestMethod: 'GET',
  status: 200,
  statusMessage: 'OK',
  httpVersion: '1.1',
  gzip: true,
  https: false,
  req: {
    headers: {
      authorization: '',
      'user-agent': 'DEX8-SDK/2.0.5 https://dex8.com',
      accept: '*/*',
      'cache-control': 'no-cache',
      host: '',
      'accept-encoding': 'gzip',
      connection: 'close',
      'content-type': 'text/html; charset=UTF-8'
    },
    payload: undefined
  },
  res: {
    headers: {
      server: 'nginx',
      date: 'Fri, 06 Mar 2020 11:20:54 GMT',
      'content-type': 'text/html; charset=UTF-8',
      'transfer-encoding': 'chunked',
      connection: 'close',
      vary: 'Accept-Encoding',
      'x-powered-by': 'PHP/5.6.40',
      'x-xss-protection': '1; mode=block',
      'x-content-type-options': 'nosniff',
      'x-nginx-cache-status': 'MISS',
      'x-server-powered-by': 'Engintron',
      'content-encoding': 'gzip'
    },
    content: '\n' +
      '\n' +
      ''
  }

ask(url, method = 'GET', body_obj)

Sends HTTP/HTTPS request to HTTP server. Redirection is handled maxRedirects times. Response is an array of resolved responses for every redirection stage. If there's no redirects then this array will contain only one response. hcn.ask('www.yahoo.com');

answers:
-----------------------------
[
  {
    requestURL: 'http://bing.com',
    requestMethod: 'GET',
    status: 301,
    statusMessage: 'Moved Permanently',
    httpVersion: '1.1',
    gzip: false,
    https: false,
    req: {
      headers: {
        authorization: '',
        'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36',
        accept: '*/*',
        'cache-control': 'no-cache',
        host: '',
        'accept-encoding': 'gzip',
        connection: 'close',
        'content-type': 'text/html; charset=UTF-8'
      },
      payload: undefined
    },
    res: {
      headers: {
        location: 'http://www.bing.com/',
        server: 'Microsoft-IIS/10.0',
        'x-msedge-ref': 'Ref A: BDA43350AD8448E0BF90BD7557179CC9 Ref B: ZAG30EDGE0120 Ref C: 2020-03-06T11:28:13Z',
        'set-cookie': [Array],
        date: 'Fri, 06 Mar 2020 11:28:13 GMT',
        connection: 'close',
        'content-length': '0'
      },
      content: ''
    }
  },
  {
    requestURL: 'http://www.bing.com/',
    requestMethod: 'GET',
    status: 302,
    statusMessage: '',
    httpVersion: '1.1',
    gzip: true,
    https: false,
    req: {
      headers: {
        authorization: '',
        'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36',
        accept: '*/*',
        'cache-control': 'no-cache',
        host: '',
        'accept-encoding': 'gzip',
        connection: 'close',
        'content-type': 'text/html; charset=UTF-8'
      },
      payload: undefined
    },
    res: {
      headers: {
        'cache-control': 'private',
        'content-length': '179',
        'content-type': 'text/html; charset=utf-8',
        'content-encoding': 'gzip',
        location: 'https://www.bing.com:443/?toHttps=1&redig=D1B8D19DDBFC4CD8A6B9FA690AD3919B',
        vary: 'Accept-Encoding',
        'x-msedge-ref': 'Ref A: 41FC9D16CE464F90A17D18B339B3A0A4 Ref B: ZAG30EDGE0116 Ref C: 2020-03-06T11:28:13Z',
        'set-cookie': [Array],
        date: 'Fri, 06 Mar 2020 11:28:13 GMT',
        connection: 'close'
      },
      content: '\r\n' +
        'Object moved to here.\r\n' +
        '\r\n'
    }
  },
  {
    requestURL: 'https://www.bing.com:443/?toHttps=1&redig=D1B8D19DDBFC4CD8A6B9FA690AD3919B',
    requestMethod: 'GET',
    status: 200,
    statusMessage: 'OK',
    httpVersion: '1.1',
    gzip: true,
    https: true,
    req: {
      headers: {
        authorization: '',
        'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36',
        accept: '*/*',
        'cache-control': 'no-cache',
        host: '',
        'accept-encoding': 'gzip',
        connection: 'close',
        'content-type': 'text/html; charset=UTF-8'
      },
      payload: undefined
    },
    res: {
      headers: {
        'cache-control': 'private',
        'transfer-encoding': 'chunked',
        'content-type': 'text/html; charset=utf-8',
        'content-encoding': 'gzip',
        vary: 'Accept-Encoding',
        p3p: 'CP="NON UNI COM NAV STA LOC CURa DEVa PSAa PSDa OUR IND"',
        'set-cookie': [Array],
        'strict-transport-security': 'max-age=31536000; includeSubDomains; preload',
        'x-msedge-ref': 'Ref A: 7F6C67E1D8364C0DA87DE69A2455A213 Ref B: ZAG30EDGE0220 Ref C: 2020-03-06T11:28:14Z',
        date: 'Fri, 06 Mar 2020 11:28:13 GMT',
        connection: 'close'
      },
      content: ' ... '
    }
  }
]

askJSON(url, method = 'GET', body)

Send HTTP/HTTPS request to API with JSON response. Redirection is not handled because we suppose that APIs are not using redirections. Parameter body can be either string or object type. As HTTP Client receives responses as string it will be automatically converted into object. hcn.askJSON('http://dummy.restapiexample.com/api/v1/employees');

JSON answer:
----------------------------------------
{
  requestURL: 'http://dummy.restapiexample.com/api/v1/employees',
  requestMethod: 'GET',
  status: 200,
  statusMessage: 'OK',
  httpVersion: '1.1',
  gzip: true,
  https: false,
  req: {
    headers: {
      authorization: '',
      'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36',
      accept: 'application/json',
      'cache-control': 'no-cache',
      host: '',
      'accept-encoding': 'gzip',
      connection: 'close',
      'content-type': 'application/json; charset=utf-8'
    },
    payload: undefined
  },
  res: {
    headers: {
      'access-control-allow-origin': '*',
      'access-control-expose-headers': 'Content-Type, X-Requested-With, X-authentication, X-client',
      'cache-control': 'no-store, no-cache, must-revalidate',
      'content-encoding': 'gzip',
      'content-type': 'application/json;charset=utf-8',
      date: 'Fri, 06 Mar 2020 11:46:31 GMT',
      expires: 'Thu, 19 Nov 1981 08:52:00 GMT',
      'host-header': 'c2hhcmVkLmJsdWVob3N0LmNvbQ==',
      pragma: 'no-cache',
      'referrer-policy': '',
      response: '200',
      server: 'nginx/1.16.0',
      'set-cookie': [
        'PHPSESSID=5df0597c3f284dc14c0c7e564466867c; path=/',
        'ezoadgid_133674=-1; Path=/; Domain=restapiexample.com; Expires=Fri, 06 Mar 2020 12:16:30 UTC',
        'ezoref_133674=; Path=/; Domain=restapiexample.com; Expires=Fri, 06 Mar 2020 13:46:30 UTC',
        'ezoab_133674=mod89-c; Path=/; Domain=restapiexample.com; Expires=Fri, 06 Mar 2020 13:46:30 UTC',
        'active_template::133674=pub_site.1583495190; Path=/; Domain=restapiexample.com; Expires=Sun, 08 Mar 2020 11:46:30 UTC'
      ],
      vary: 'Accept-Encoding,X-APP-JSON',
      'x-middleton-response': '200',
      'x-sol': 'pub_site',
      'content-length': '595',
      connection: 'close'
    },
    content: {
      status: 'success',
      data: [
        [Object], [Object], [Object],
        [Object], [Object], [Object],
        [Object], [Object], [Object],
        [Object], [Object], [Object],
        [Object], [Object], [Object],
        [Object], [Object], [Object],
        [Object], [Object], [Object],
        [Object], [Object], [Object]
      ]
    }
  }
}

grabStreams(url, method = 'GET', body)

Get request and response streams which can be used for piping. For example: clientResponse.pipe(file) hcn.grabStreams('http://www.dex8.com');

setHeaders(headerObj)

Change request header object. Previously defined "this.headers" properties will be overwritten. headerObj - {'authorization', 'user-agent', accept, 'cache-control', 'host', 'accept-encoding', 'connection'} hcn.setHeaders({authorization: 'myToken, 'content-type': 'application/json; charset=utf-8', accept: 'application/json'});

setHeader(headerName, headerValue)

Change only one request header. headerName - header field name headerValue - header field value hcn.setHeader('authorization', 'myToken);

delHeaders(headerNames)

Delete the request headers. headerNames - array of header names 'content-type', 'accept' hcn.delHeaders('content-type', 'accept');

getHeaders()

Get the current request headers.

AddOns

Additional libraries.

RobotsTxt

const { HttpClient, RobotsTxt } = require('httpclient-nodejs');

const robotsTxt = new RobotsTxt(HttpClient);

const fja = async () => {
  const url = 'https://www.yahoo.com/lifestyle/mom-makes-upsetting-discovery-walmart-134505752.html';
  const robotsText = await robotsTxt.fetch(url);
  console.log('robotsText::');
  console.log(robotsText);

  const robotsTextObj = robotsTxt.parse(robotsText);
  console.log('robotsTextObj::', robotsTextObj);

  const follow_urls = robotsTxt.whatToFollow(robotsTextObj, '*');
  console.log('follow_urls::', follow_urls);

  const unfollow_urls = robotsTxt.whatToUnfollow(robotsTextObj, '*');
  console.log('unfollow_urls::', unfollow_urls);
};

fja();

License

The software licensed under AGPL-3.