Web-logger-lib NPM

README

Description

This is a browser based node library that allows you to track and log various events in your frontend application. Some of the features included are:

Tracking uncaught errors
Tracking route changes
Tracking click events
Tracking requests and responses
Tracking load times

How do I get set up?

Step one: Install the library

npm install web-logger-lib

Step two: Import the library into your codebase as follows:

import * as Logger from 'web-logger-lib';

The Logger object will contain all of the Core and Wrapper APIs (see the API section for more details).

Demo app

To see an example of how to use this library in your application, please access the demo repository at this URL: https://bitbucket.org/michaelcoxon/poc-webevents-ui/src/master/

Architecture

Architecture diagram

The library is broken down into three pieces: Core Implementation, Core API and Wrapper API.

Core Implementation

This part of the library is the main "guts" of the system. This is where the Logger class will consume events from the exposed APIs and buffer them using the WebEventBuffer. The WebEventBuffer is a simply a queue datastructure adopting the FIFO principle. If the number of events in the buffer is greater or equal to the flushSize (configurable during initialisation) or the flushTimer (also configurable during initialisation) has lapsed, the Logger will commit all of the present events in the WebEventBuffer using the WebEventCommitter. The WebEventCommitter simply accepts a list of events and posts a request to the targetted backend service (this endpoint will be provided by the consumer during initialisation). If the server responds with a 2XX response and successfully accepted the events, the system will then flush the committed events out of the buffer. If the server fails to respond with a 2XX response, the events are persisted in the buffer and will be retried on the next trigger (flushTimer lapsing or if the next event inserted makes the number of events in the buffer greater than the flushSize).

It is entirely possible that during the commit phase while the log request to the server is still in flight, the consuming application adds more events to the buffer as the user continues to interact with the given application. These subsequent events will simply be added to the buffer but will not affect the current batch that is being currently committed. Once the commit phase is over and assuming the phase succeeded, the current batch will be flushed out of the buffer and the recently added events will move to the front of buffer. These newly added events will be committed and flushed upon the next trigger (the flushTimer lapsing or number of events exceeding the flushSize). Note that if the commit phase failed, the current batch of events will be persisted and the recently added events will stay in place but the next trigger will then commit all events (the old batch and the newly added events) to the backend service in one request. Furthermore, it also possible that during the commit phase, a trigger is fired (eg. flushTimer lapses again while the library is waiting for a response from the backend server). In this case, the library will simply ignore the trigger since there is already a pending request. The library will only make another request if there is no pending request. This way, we do not overload the backend service with too many requests.

There is also another case where the server simply refuses or is unable to accept events for a prolonged period of time (eg. server is down). In this case, although the triggers will attempt to commit and flush events to the backend service, the events sent will be categorically rejected. As a result, the buffer will continuously grow and as we persist events that failed to be committed to the server. To prevent memory overflow in the frontend that can potentially crash the user's browser, we have set a maxBufferSize limit which should ideally be far larger than the flushSize. In the event that the buffer size reaches this limit, the library will shutdown by no longer accepting events and stop firing requests to the backend. This is a rare but potentially catastrophic case to keep in mind.

Core API

The Core API wraps itself around the implementation below and gives the consumer the ability to interact with the library. At this layer, we provide raw-like but important APIs. Please see the API section for more details.

Wrapper API

The Wrapper API provides enhanced functionality and takes care of the heavy lifting for the consumer. We provide out of the box tools like:

tracking load times of React components (including lazy loaded components)
tracking requests
tracking route changes
tracking click events
tracking uncaught errors

The Wrapper API is simply a higher level abstraction that builds upon the Core API. Please see the API section for more details.

API

Core

`init(options: Settings): Promise<void>`

Params

init({
  // The server endpoint that will accept the events.
  // It should be a full URL like: https://logger.com/log
  endpoint: string;

  // Track user click based events.
  trackClickEvents?: boolean; // Default: false

  // Track any uncaught errors in the application.
  trackUncaughtErrors?: boolean; // Default: false

  // Track the DOMContentLoaded event or not.
  ignoreDOMContentLoaded?: boolean; // Default: false

  // The number of events to determine when to commit
  // and flush events.
  flushSize?: number; // Default: 5

  // The interval time to determine when to commit and
  // flush events.
  flushTimer?: number; // Default: 1000 (in ms)

  // The maximum buffer size. If buffer reaches this limit,
  // the library will shutdown and no longer track events.
  maxBufferSize?: number; // Default: 20000

  // This is where you can insert additional contextual properties
  // that will be stamped in all subsequent captured events.
  contextForEvent?: {
    [string]: string;
  };
});

Example

init({ endpoint: 'https://logger.com/api' });
// Your code continues...

OR;

init({ endpoint: 'https://logger.com/api' }).then(() => {
  // Your code continues...
});

Example logged events that will be sent to the server:

{
  "action": "logger-initialised",
  "level": "INFO",
  "ip": "255.345.65.139",
  "conversationId": "190f0477-6269-431d-a117-4a59d9d1e234",
  "timestamp": "2021-03-03T19:01:11:816+1100",
  "url": "https://yourapp.com/",
  "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36"
}

Initialises the library. This API should be called ideally during the boot sequence of your application. Note that although the initialisation step is internally an asynchronous task, you can treat the API as if it is a synchronous task. We will simply queue your immediately fired events in a separate area of the library while the internal asynchronous task is completing. Once the asynchronous work is done, the separately queued events will be replayed onto the main buffer. If this is not ideal for your application, we also return a promise where your application can continue the boot sequence after the library has fully initialised itself (completed the asynchronous task). Note that once the library has been initialised, it will log an initialised event.

`logEvent(options: CoreEvent): void`

Params

logEvent({
  // This specifies the severity of the event.
  level: 'INFO' | 'ERROR';

  // This is where you can specify what exactly happened
  // (eg. "click", "route-change", "form-submitted").
  action: string;

  // The subject of the event (eg. a component, page, button etc.).
  target?: string; // Default: undefined.

  // An optional ID to use to link a logged event
  // to another operation (eg. an API request)
  correlationId?: string; // Default: undefined

  // You can override the timestamp generated by the library
  // with your own version.
  timestamp?: string; // Default: generated in this format - "yyyy-mm-dd'T'HH:MM:ss.ms"

  // The current URL when the event was logged. This is
  // captured by default but can be overriden too.
  url?: string; // Default: window.location.href

  // Here, you can provide additional properties for the given event.
  message?: {
    // Used to track load times of events.
    elementId?: string; // Default: undefined.

    // Used to describe anything about the event.
    text: string;

    // Used to track the status of a given API request.
    status?: string; // Default: undefined.
  }
});

Example

logEvent({
  level: 'ERROR',
  action: 'app-crashed',
  target: 'App',
  message: {
    text: 'Somehow the app crashed :/',
  },
});

Example logged events that will be sent to the server (based on above example):

{
  "action": "app-crashed",
  "level": "ERROR",
  "target": "App",
  "ip": "255.345.65.139",
  "conversationId": "190f0477-6269-431d-a117-4a59d9d1e234",
  "timestamp": "2021-03-03T19:01:11:816+1100",
  "url": "https://yourapp.com/",
  "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36",
  "message": {
    "text": "Somehow the app crashed :/"
  }
}

This API is how the consumer application can log events to the library. You must initialise the library before calling this function. Most of the time, you will be using the Wrapper APIs rather than directly interfacing with this API. However, you can use this API to log events that are not captured via the Wrapper APIs.

`getConversationId(): string | undefined`

Example

const conversationId = getConversationId();

This API returns the conversation id that is stamped on all logged events for the given session. You must initialise the library before calling this function.

`updateContextForEvent(context: CustomContext): void`

Params

updateContextForEvent({
  [string]: string;
})

Example

updateContextForEvent({
  userId: '1234-john',
  authId: '1234-2223',
});

Example logged events that will be sent to the server (based on example below):

updateContextForEvent({
  userId: '1234-john',
  authId: '1234-2223',
});

logEvent({
  level: 'ERROR',
  action: 'app-crashed',
  target: 'App',
  message: {
    text: 'Somehow the app crashed :/',
  },
  correlationId: '123',
});

{
  "action": "app-crashed",
  "level": "ERROR",
  "target": "App",
  "ip": "255.345.65.139",
  "conversationId": "190f0477-6269-431d-a117-4a59d9d1e234",
  "timestamp": "2021-03-03T19:01:11:816+1100",
  "url": "https://yourapp.com/",
  "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36",
  "message": {
    "text": "Somehow the app crashed :/"
  },
  "correlationId": "123",
  "customContext": {
    "userId": "1234-john",
    "authId": "1234-2223"
  }
}

This API allows you to update the custom context that will be stamped on all future events. A common usecase might be that you want to add additional IDs of the current user to each event captured. This way, you can identify events based on a given user in the backend. You must initialise the library before calling this function.

`getEndpoint(): string | undefined`

Example

const endpoint = getEndpoint();

This API returns the logging endpoint registered. You must initialise the library before calling this function.

Wrapper

`withNonLazyComponentProfiler(component: React.Component): React.Component`

Example

const TrackedComponent = withNonLazyComponentProfiler(App);

Example logged events that will be sent to the server:

[
  {
    "level": "INFO",
    "action": "component-load-start",
    "target": "Application",
    "message": {
      "elementId": "58f1fdd7-7902-4c7e-9561-41903228bbe4"
    },
    "ip": "234.234.212.139",
    "conversationId": "190f0477-6269-431d-a117-4a59d9d1e234",
    "timestamp": "2021-03-03T19:01:11:819+1100",
    "url": "https://yourapp.com/app",
    "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36"
  },
  {
    "level": "INFO",
    "action": "component-load-end",
    "target": "Application",
    "message": {
      "elementId": "58f1fdd7-7902-4c7e-9561-41903228bbe4"
    },
    "ip": "101.188.83.139",
    "conversationId": "190f0477-6269-431d-a117-4a59d9d1e234",
    "timestamp": "2021-03-03T19:01:11:848+1100",
    "url": "https://yourapp.com/app",
    "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36"
  }
]

This API will track the load time of your given component. It will fire a component load start event and component end event. From there, your backend service can calculate the time difference between to the two events. This API will commonly be used at the root level of your React application. You must initialise the library before using this API.

`LazyLoad(() => Promise<{ default: React.Component; }>, ComponentId: string)`

Params

LazyLoad(
  // Provide a dynamic import statment here.
  () => Promise<{ default: React.Component; }>,

  // Provide an ID for the given lazy component
  // to identify the component in the logs.
  ComponentId: string
);

Example

// Instead of this:
const HomeComponent = React.lazy(() => import('./HomeComponent'));

// Use this:
const HomeComponent = LazyLoad(() => import('./HomeComponent'), 'HomeComponent');

Example logged events that will be sent to the server:

[
  {
    "level": "INFO",
    "action": "component-load-start",
    "target": "Login",
    "message": {
      "elementId": "420f7c57-eebe-4653-ab0a-79739b23a86b"
    },
    "ip": "223.3.4543.139",
    "conversationId": "190f0477-6269-431d-a117-4a59d9d1e234",
    "timestamp": "2021-03-03T19:01:18:682+1100",
    "url": "https://yourapp.com/login",
    "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36"
  },
  {
    // Only if the component is being downloaded for the first time.
    "level": "INFO",
    "action": "module-download-start",
    "target": "Login",
    "ip": "33.188.11.139",
    "conversationId": "190f0477-6269-431d-a117-4a59d9d1e234",
    "timestamp": "2021-03-03T19:01:18:682+1100",
    "url": "https://yourapp.com/login",
    "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36"
  },
  {
    // Only if the component is being downloaded for the first time.
    "level": "INFO",
    "action": "module-download-end",
    "target": "Login",
    "ip": "101.188.83.139",
    "conversationId": "190f0477-6269-431d-a117-4a59d9d1e234",
    "timestamp": "2021-03-03T19:01:18:811+1100",
    "url": "https://yourapp.com/login",
    "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36"
  },
  {
    "level": "INFO",
    "action": "component-load-end",
    "target": "Login",
    "message": {
      "elementId": "420f7c57-eebe-4653-ab0a-79739b23a86b"
    },
    "ip": "23.188.33.139",
    "conversationId": "190f0477-6269-431d-a117-4a59d9d1e234",
    "timestamp": "2021-03-03T19:01:18:814+1100",
    "url": "https://yourapp.com/login",
    "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36"
  }
]

This API will track the load time of a given lazy component. Note that this API is a direct drop in replacement of React.lazy. Internally, the API simply extends from React.lazy but will also log events. It will fire a component load start event and component end event. In addition, if the component is being downloaded for the first time, it will also track the module download start time and module download end time. This API will should be used strictly for lazy based components. You must initialise the library before using this API.

`<UrlListener /> (React Component)`

Example

import React from 'react';
import { BrowserRouter as Router, Switch, Route } from 'react-router-dom';

import { UrlListener } from 'web-logger-lib';

const App = () => {
  return (
    <Router basename="/">
      <UrlListener /> // Insert here.
      <Switch>
        <Route path="/login">The login component.</Route>
        <Route path="/">The home component.</Route>
      </Switch>
    </Router>
  );
};

export default App;

Example logged events that will be sent to the server:

{
  "url": "https://yourapp.com/login",
  "level": "INFO",
  "action": "route-change",
  "target": "https://yourapp.com/status",
  "ip": "234.188.236.139",
  "conversationId": "190f0477-6269-431d-a117-4a59d9d1e234",
  "timestamp": "2021-03-03T19:01:20:419+1100",
  "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36"
}

This API ensures that all route changes in a React application are logged. You simply insert the <UrlListener /> component just under your standard <Router /> component and everytime the user changes the route, a route change event will be logged.

`trackRequestStart({ method: string | undefined, url: string }): string`

Params

trackRequestStart({
  // The HTTP method.
  method: 'GET'
          | 'DELETE'
          | 'HEAD'
          | 'OPTIONS'
          | 'POST'
          | 'PUT'
          | 'PATCH'
          | 'PURGE'
          | 'LINK'
          | 'UNLINK'
          | 'UNKNOWN-HTTP-METHOD';

  // The targetted url.
  url: string;
});

Example

trackRequestStart({
  method: 'POST',
  url: 'https://api.domain.com/web',
});

Example logged events that will be sent to the server:

{
  "level": "INFO",
  "action": "POST-request",
  "target": "https://api.yourapp.com/login",
  "correlationId": "4e5a3902-3036-47fe-952e-d11552c900da",
  "ip": "23.12.56.139",
  "conversationId": "190f0477-6269-431d-a117-4a59d9d1e234",
  "timestamp": "2021-03-03T19:01:27:662+1100",
  "url": "https://yourapp.com/login",
  "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36"
}

This API allows you to track a given request. You can log the request method and targetted URL. The API also returns a string identifying correlation id. This id is to help correlate logs with the tracked request and is essentially an identifier of the request. The API will internally log a request started event and stamp the event with the generated correlation ID which is returned to the consumer. The ID can also be used to stamp the response event (see trackRequestEnd API below). You must initialise the library before using this API.

A common usecase is to intercept requests and log details about the request. For example, this API can be used in a given Axios interceptor as follows:

import axios from 'axios';
import * as Logger from 'web-logger-lib';

axios.interceptors.request.use(function (config) {
  // Attach the correlation ID to the request header so we can link the request and the event.
  config.headers['X-Correlation-Id'] = Logger.trackRequestStart({
    method: config.method ? (config.method.toUpperCase() as Logger.RequestType) : undefined,
    url: config.url,
  });

  return config;
});

`trackRequestEnd(options: RequestEndOptions): void`

Params

trackRequestEnd({
  // The HTTP method.
  method: 'GET'
          | 'DELETE'
          | 'HEAD'
          | 'OPTIONS'
          | 'POST'
          | 'PUT'
          | 'PATCH'
          | 'PURGE'
          | 'LINK'
          | 'UNLINK'
          | 'UNKNOWN-HTTP-METHOD';

  // The targetted url.
  url?: string; // Default: undefined.

  status?: number; // Default: undefined.
  statusText?: string; // Default: undefined.
  correlationId: string;
  errorText?: string // Default: undefined.
});

Example

trackRequestEnd({
  method: 'POST'
  url: 'https://api.domain.com/web',
  status: 200,
  statusText: 'Success.',
  correlationId: '12343e', // Should be the same one that was used when tracking the request.
  errorText: 'No error.'
});

Example logged events that will be sent to the server:

{
  "level": "ERROR",
  "action": "POST-response",
  "target": "https://api.yourapp.com/login",
  "correlationId": "4e5a3902-3036-47fe-952e-d11552c900da",
  "message": {
    "status": 401,
    "text": "Request failed with status code 401"
  },
  "ip": "23.12.56.139",
  "conversationId": "190f0477-6269-431d-a117-4a59d9d1e234",
  "timestamp": "2021-03-03T19:01:30:305+1100",
  "url": "https://yourapp.com/login",
  "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36"
}

This API allows you to track a given response. You must initialise the library before using this API.

A common usecase is to intercept responses and log details about the response. For example, this API can be used in a given Axios interceptor as follows:

import axios from 'axios';
import * as Logger from 'web-logger-lib';

const trackResponse = (config: AxiosRequestConfig, response?: AxiosResponse<any>, errorText?: string) => {
  if (config.url === getEndpoint()) return;

  trackRequestEnd({
    method: config.method ? (config.method.toUpperCase() as RequestType) : 'UNKNOWN-HTTP-METHOD',
    url: config.url,
    status: response ? response.status : undefined,
    statusText: response ? response.statusText : undefined,
    errorText,
    correlationId: config.headers['X-Correlation-Id'], // Assuming the correlation id was originally attached to the request (see trackRequestStart API above).
  });
};

axios.interceptors.response.use(
  (response) => {
    // Track responses.
    trackResponse(response.config, response);
    return response;
  },
  (error) => {
    // Track any errors that happened during the request.
    trackResponse(error.config, error.response, error.message);

    return Promise.reject(error);
  },
);

`trackAllAxiosRequests(axios: AxiosStatic): void`

Params

trackAllAxiosRequests(
  // Your axios library.
  axios: AxiosStatic
);

Example

import axios from 'axios';

trackAllAxiosRequests(axios);

To easily track all requests (and their corresponding responses), we have exposed a simple API as shown above. This API essentially extends from the trackRequestStart and trackRequestEnd APIs and implements the Axios interceptors as shown in the two APIs examples. You simply provide your axios library and the library will track the request and responses. Note that we ignore tracking API requests initiated from our library. This API should commonly be used during the boot sequence of your application. You must initialise the library before using this API.

How to quickly setup the library in your React App?

First, setup the library at the root of your application. The following code should be entered into your entrypoint file (usually, this will be your index.js/ index.ts file).

import ReactDOM from 'react-dom';
import axios from 'axios';

import * as Logger from 'web-logger-lib';

import App from './App';

Logger.init({
  endpoint: 'https://logger.com/log',
  trackUncaughtErrors: true,
  trackClickEvents: true,
  ignoreDOMContentLoaded: false,
  flushSize: 15,
  flushTimer: 100000,
  maxBufferSize: 20000,
});

Logger.trackAllAxiosRequests(axios);

// Do some asynchronous work to fetch data
// that can added to all future events.
fetchUser((userId) => {
  Logger.updateContextForEvent({
    userId: userId,
  });
});

// Track the load time of your root component.
const Root = Logger.withNonLazyComponentProfiler(App, 'Application');

// Finally render your application.
ReactDOM.render(<Root />, document.getElementById('root'));

The setup code above will setup the library to track:

Uncaught errors,
Click events
DOMContent loaded event
Track all axios based requests and responses initiated from your application.
Load time of your root app component

Once the boot sequence is setup, you can then move onto tracking route changes and load time of lazy loaded components. Here is an example:

import React, { Suspense } from 'react';
import { BrowserRouter as Router, Switch, Route } from 'react-router-dom';

import { UrlListener, LazyLoad } from 'web-logger-lib';

import { Link, Nav } from './styles';

// When rendered, the load times of these components will be tracked.
const Login = LazyLoad(() => import('../pages/Login'), 'Login');
const ApiError = LazyLoad(() => import('../pages/ApiError'), 'ApiError');
const Error = LazyLoad(() => import('../pages/Error'), 'Error');
const MultiForm = LazyLoad(() => import('../pages/MultiForm'), 'MultiForm');
const FormStatus = LazyLoad(() => import('../pages/FormStatus'), 'FormStatus');

const App = () => {
  return (
    <Router basename="/">
      <UrlListener /> // Inject url listener so we can track the route changes.
      <Suspense fallback={<div>Loading page...</div>}>
        <Nav>
          <Link to="/">Home</Link>
          <Link to="/login">Login</Link>
          <Link to="/error">Error</Link>
          <Link to="/api-error">API Error</Link>
          <Link to="/multi-form">Form with multiple steps</Link>
          <Link to="/form-status">Form Status</Link>
        </Nav>
        <Switch>
          <Route path="/api-error" component={ApiError} />
          <Route path="/error" component={Error} />
          <Route path="/login" component={Login} />
          <Route path="/multi-form" component={MultiForm} />
          <Route path="/form-status" component={FormStatus} />
          <Route path="/">
            <div>Click on one of the routes above to test various cases.</div>
          </Route>
        </Switch>
      </Suspense>
    </Router>
  );
};

export default App;

Note that you can track loading time of any nested React components too.

Contribution guidelines

Writing tests

npm run test

All code changes must be accompanied by tests. We use Jest and react-testing-library to write unit/integration tests.

Formatting and linting

npm run format && npm run lint

Please put up a PR for review when making changes. Team owners will then approve and publish the changes when necessary. Do not bump the version yourself as that will be done in a separate PR by the team owners.