@chipsgg/openservice NPM

Openservice Microservice Framework

A modern micro-service framework to improve your application development, testing, deployment and maintenance.

Install

npm install -g @chipsgg/openservice

Run

You must have a configuration file, env, or json and your service code. See the API section for more information.

openservice config.js

QuickStart

Quick reference for getting started.

Service Definition

Services are designed as closures. Return your services api as an object of functions.

module.exports = async (config,services,emit)=>{
  //initialize service here...

  //return service api
  return {
    ping(){
      return 'pong'
    }
  }
}

Config Definition

An example configuration that shows some basic setup.

//in config.js
module.exports = {
  //the name of the service, used for logging
  name:'helloworld',
  //define all paths where service files can be found
  paths:[
    process.cwd(),
    __dirname
  ],
  //defines which services to start and in what order
  //service startup is blocking and will fail the entire app if
  //one service fails
  start:[
    'helloworld.service'
  ],
  //defines your named transports
  transports:{
    //the name of your local transport is the key 'local'
    local:{
      //openservice includes a local stream transport if your app is completely contained as a single process
      require:'openservice/transports/local'
    },
  },
  //helloworld is the namespace of this group of services
  //services must belong to a namespace. 
  helloworld:{
    //service is the name of this service, it would be accessed like helloworld.service
    service:{
      //uses a file called helloWorld.js, you are telling openservice how to find the file
      require:'helloWorld',
      //uses the local transport
      transport:'local',
      //has no service dependencies (clients)
      clients:[],
      //has no injected configuration
      config:{},
    }
  }
}

Secrets & ENV

In .env file or environment variables. Openservice uses lodash.set/get notation for creating a json object from environment variables which get merged into your config.

helloworld.service.config.secret=12345

Starting Service

From the command line: openservice config.js

Or from within a js file

const OpenService = require('openservice')
const config = require('./config')

OpenService(config).then(([service])=>{
  console.log('Services started')
}).catch(err=>console.log(err)

Getting Started

Services

In this framework a service is a single function which returns some methods. You can imagine it as a library accessible over some kind of transport or inter process communication layer. The functions your service returns are the public functions accessible to any other service on the network.

async function(config, services, emit) => {}

config - This is a plain js object with options specified by the user. In practice these will map to environment variables to allow configuration of the service at run time.
services - this is a js object keyed by external service names. These represent external services which the current service is dependent on and are injected in when the service starts. Each service has a client which allows the developer to call functions in the typical async await pattern.
emit - this is a function which allows the service to emit messages for interested listeners. You will treat this similiar to a node event emitter in which you specify the topic and then the data.

A service has the option of returning serveral things:

nothing - service returns nothing on instantiation, this means nothing can call it externally
a function - service returns a single function which can be called by other services
a class - service returns a class or key value object of functions, this will be exposed to external services as its api

Service Example

//services are exposed as an asyncronous function with some standard parameters.
//this allows the service to take in all the information about the world it needs

module.exports = async (config,services,emit)=>{
  //initialize service here...

  //return service api
  return {
    ping(){
      return 'pong'
    }
  }
}

Service Clients API

The framework takes care of wrapping up your services and exposing them to the transport layer, and a client is created and injected into your service as a dependency if you specify a service needs it. Clients connect to the transport for you and create an interface to allow local-like interactions with an external service. You can call your external service as if it was a local library.

Simple Service Example

//imagine you have a service with some dependencies
//one of your clients is a users table with some on it.
module.exports = async (config, services, emit) => {
  const { users, wallets } = services

  //services has users, wallets, notify clients 
  async function withdraw(userid,amount){
    //first make sure the user exists
    const user = await users.get(userid)
    
    //get withdraw from users wallet, which has same id as user
    return wallets.withdraw(userid,amount)
  }

  return {
    withdraw
  }
}

Advanced Service Example

Here is an example of the flexibility of a service client:

module.exports = async (config,services)=>{
  //we have an external service called time
  const { time } = services

  //there are several ways to interface with this

  //we can make a request response call
  let now = await time.getTime()

  //we can emit a fire and forget message. Time has a function called sync, and this framwork
  //attaches a utility to "emit" to the function by emiting a 1 way message where we dont care
  //about the response. We still await though as network needs to confirm message sent.
  await time.sync.emit(Date.now(),'my service')

  //we can listen for updates. The time service is emitting an event on the channel
  //called "tick". The client uses the keyword "listen" to listen on that channel
  //and callback the updates.
  await time.tick.listen(current=>{
    now = current
  })


  //streams are also available to you, wrapped wiht the highland js library
  //in this instance we want to know if sync is being called by other services
  //streams give you access to a more advanced event structure. 

  const syncStream = time.sync.listen()

  //the event structure contains the services response to the call
  //as well as any of the callers arguments. Events are also available
  //for request response calls

  syncStream.filter(event=>{
    //args: [service response, caller argument 1, caller argument 2]
    const {args:[response,time,serviceid]} = event

    //highland has a filter operation which works like js filter. 
    //here we just filer out our own calls to sync
    return serviceid != 'my service'
  }).each(event=>{
    const {args:[response,time,serviceid]} = event
    console.log(serviceid,'synced the time')
  })

  return {
    now(){
      return now
    }
  }

}

Listening to Events

All services produce events as they get called and return data. We can tap into this data stream using a callback API. This will be called when services return data from calls.

services.wallets.withdraw.listen((result,...arguments)=>{}) or services.wallets.on('listen',(result,...arguments)=>{})

In an example in a statistics service:

module.exports = async (config,services) => {
  //services has wallets
  const stats = {
    totalWithdrawn:0
  }

  services.wallets.on('withdraw',(wallet,userid,amount)=>{
    //update total withdrawn amount
    stats.totalWithdrawn += amount
  })

  return {
    getStats(){
      return stats
    }
  }
}

Listening to Event Streams

OpenService uses the node stream compatible library highland to wrap streams and give you some extra functionality. We can access the underlying service streams to do processing on rather than the event emitter. The stream give you extra meta data about the event.

const stream = services.wallets.listen()

An event object has this schema

{
  //event channel: requests, responses, streams, errors
  channel: 'string',
  //service function path like, ['withdraw']
  path: { type: 'array', items: 'string' },
  //function arguments, for requests this is simply the arguments to the function
  //for responses, the response is first, and the arguments start at args[1]
  args: 'array',
}

module.exports = async (config,services) => {
  //services has wallets
  const stats = {
    totalWithdrawn:0
  }

  const walletStream = services.wallets.listen()

  //highland exposes the each callback
  walletStream.each(({path,args})=>{
    //we can switch on path, this will listen for all
    //calls to the wallet
    switch(path[0]){
      case 'withdraw':
        stats.totalWithdrawn += args[2] //result, userid, amount
        break
    }
  })

  return {
    getStats(){
      return stats
    }
  }
}

The Configuration File

Configs can be represented through env vars or through a JS object. The configuration tells openservice where to find your services files, what dependencies they need and any additional data the service needs. Configurations should be able to be merged together. In order to facilitate this theres a specific convention to define environment variables.

//this is the top level of the config file
module.exports = {
  name:string,    //required, gives a name for logs and errors for this service
  start:string[], //the services you want to start in this processs using the string path
  config:object,  //global configuration you want passed into every service defined
  paths:string[], //paths to where your service files are located
  transports:object, //key value of all available transports
}

Each individual service configuration is defined in the json object at a path. Services need to be namespaced in a top level directory in the config. This can be achieved like this.

require:string,   //the path of the service file
transport:string, //the name of the transport this service uses
clients:string[], //the path to any clients this service needs
config:object,    //configuration object passed into this service

See examples/basic/config.js or examples/advanced/config.js

Environment and Secrets

Environment variables are very important for configuring your services and this architecture accepts a convention for injecting variables into your service defintion. For example if you need to pass sensitive data you do not want committed to your project you can specify it in an .env file or your env variables. The convetion for injecting variables follows lodash's set interface. Furthermore the architecture will ignore any envs which begin with an uppercase, so only lowercase envs will be observed. In practice it produces something like this:

.env

express.cookieSecret=1234qwerty
users.username=admin
users.password=qwerty
auth.systemToken=abcdefg

config.js

{
  ...,
  users:{
   file:'./services/users', 
   clients:[],
   //additional configuration for database table
   table:'users',
   //merged from env:
   username:'admin',
   password:'querty',
  },
  auth:{
   file:'./services/auth', 
   clients:['users'],
   //merged from env:
   systemToken:abcdefg
  },
  express:{
   file:'./services/express', 
   clients:['api'],
   port:80,
   //merged from env:
   cookieSecret:1234qwerty
  },
  ...
}

Service Multiplexing

Traditionally services are thought of as 1 to 1 to an application, kubenetes pod or docker container. This architecture discards that notion. It has no opinion on how many services you run in a single application. This allows flexibilty for economizing server usage by intelligently bundling many services together into a single container or pod. The downside is that this leads to yet another container configuration specification, as these services are much like containers, but hopefully a bit simpler. Most of these specifications can be bundled with the source code as it defines mainly service names and dependencies.

Configuring Transports

Services typically need to talk to other services. This architecture requires the developer do this explicitly. Each application will require its own service definitions. These specify which services you want to run, where to find them, what you want to name them, how to configure them, and what dependencies. These are all specified in a json object.

module.exports = {
  //these are the services you want to run in your application. These names are what is exposed
  //in your service dependencies, so name carefully. You want your services names to be unique.
  //This also has a side effect of ordering the startup of each service. So place the most important
  //services first so that they can be relied on as dependents of other services. Otherwise
  //the application may deadlock on startup.
  services:[
    'users',
    'auth',
    'api',
    'express',
  ],
  //here is where you specificy transport type. We are using a nats streaming driver.
  //it has certain configuration requirments which you pass in here.
  transport: 'local',
  //all of these represent configuration for each service and get passed
  //in the service as the first argument "config".
  users:{
   //the service file is found here
   file:'./services/users', 
   clients:[],
   //additional configuration for database table
   table:'users',
  },
  auth:{
   file:'./services/auth', 
   clients:['users'],
  },
  api:{
   file:'./services/api', 
   //you can also specify remote services by name. Wallets could run in a seperate application
   //but is avaialble through the transport layer.
   clients:['auth','users','wallets'],
  },
  express:{
   file:'./services/express', 
   clients:['api'],
   port:80
  },
}