Zcrypto-service-manager NPM

zcrypto-service-manager

Purpose

This repository holds the code for our Hyper Protect Crypto Services Service Manager. It serves as the brains of all our operations. It's major functions are:

Deploy new instances of our service within a given region. It will have the knowledge of where all of the servers are via it's database (Cloudant).
Allow SREs to view and manage the service. It will have information about the containers which are deployed when a new client comes on board as well as interacting with them to debug and resolve issues.
It will have have information on the Crypto Express cards which are a part of this service as well as the LPARs. This information is call gathered through the SCaaS Manager.

Getting started with Development

Requirements

Node.js version 12.16.0
You will need to get a .env file from an existing team member. It should go in the root directory of the project.

For development you will be running 2 servers allowing for hot loading of the react UI code when changes are made: 1) An express server - running on port 3000 so as not to interfere with the react server 2) A react server - which runs on port 3001, which will hot load the client side code when you make changes. Requires an environment variable to be set for development mode. See below.

.env for secrets manager

Secrets manager secrets used for deployment and in various scripts are listed in the secretsmanager.env. Edit this file by adding your api key to the SECRETS_MANAGER_API_KEY property, uncommenting the necessary datacenter properties and save the file as .env which can then be used with deployment scripts.

This env file needs to be updated when any new secrets are added or modified or when a new datacenter is added.

Artifactory NPM resolver

When building the Service Manager and the React Server using NPM, it's important to resolve all dependencies using the Artifactory NPM. To do this you need to create an .npmrc file in your home directory with the following content:

registry=https://eu.artifactory.swg-devops.com/artifactory/api/npm/sys-zaas-team-dev-npm-virtual/
username = <the IBM email address you use to log into artifact>
_password = <the base64 encoded Artifactory API key for your account>
email = <the IBM email address you use to log into artifact>
always-auth = true

E.g.,

registry=https://eu.artifactory.swg-devops.com/artifactory/api/npm/sys-zaas-team-dev-npm-virtual/
username = mattarno@uk.ibm.com
_password = <REDACTED>
email = mattarno@uk.ibm.com
always-auth = true

Becasue the .npmrc file contains a secret, the base64 encoded (not encrypted!) API key, do not upload this file to GHE.

If you have not generated an Artifactory API key, log in to Artfactory then go to https://eu.artifactory.swg-devops.com/ui/admin/artifactory/user_profile and create an API key. Remember to base64-encode this API key before you add it to your .npmrc file.

Steps

:large_orange_diamond: Ensure correct version of Node.js is installed - see above 1) Once you have cloned the repository, simply run the following commands from the root directory to start the express server. The "npm ci" commands only need to be run when first cloning the project or when an update to the project was made that requires new modules to be installed. Usually you can skip over these commands and just start the express server.

```
npm ci
npm i --only=dev
npm run start
```

2) Use or a browser or API Testing tool like postman or curl to test the server is running. Call http://localhost:3000/v1/healthcheck and look for the following output:

```
{"name":"zcrypto-service-manager","version":"0.0.1"}
```

3) Bring up a second terminal in order to start the react server. The react server code is stored in the GHE repo ZaaS/zcrypto-service-manager-react-app to follow the same steps to install the dependencies for the react code found in the directory "react-client". Same notes apply about the install commands. The PORT environment variable should be set to 3001 for the react server to run, it will then proxy requests to port 3000 where the express server is running. Note in windows, setting an environment variable is different than *nix systems. 4) Generate .npmrc file following the console-pipeline instructions. 5) Install node packages.

git clone git@github.ibm.com:ZaaS/zcrypto-service-manager-react-app.git
mv zcrypto-service-manager-react-app react-client
cd react-client
npm ci
npm i --only=dev
PORT=3001 npm run start

For Windows use:

git clone git@github.ibm.com:ZaaS/zcrypto-service-manager-react-app.git
mv zcrypto-service-manager-react-app react-client
cd react-client
npm ci
npm i --only=dev
set PORT=3001 
npm run start

6) Access the app by launching http://localhost:3001.

##### Automatically Restart Servers on Code Change
Install the nodemon package globally and use in place of node to start your 
app. This tool allow the app to restart automatically upon detection of code change. For example:

```
npm i -g nodemon
nodemon start
```

React and Carbon Components

Information

In order to achieve a unified IBM Cloud look and feel, the UI is rendered using the React versions of Carbon components. It is important to understand the Carbon makes use of Sass to handle styling. Sass compiles .scss files in corresponding .css files. This is step is done every time you start the react server or create a build. So you should only be making changes to the .scss files for styling rules. Refer to scripts section of pacakge.json for more information on what scripts are called.

Debugging React

Install the React Dev Tools plugin for your browser to look at react specific problems:

Building a Staging or Production-like Package for React Code Locally

A staging or production build looks different than the current development setup. We will refer to this mode as non-development mode. For production, all of the react javascript and css files are packaged to be served statically from the same server/port as the express server (port 3000). A non-development build can be created by doing the following:

cd react-client
npm run build

:large_orange_diamond: Note: This may be change in the future once we implement a proper automated build process using something like Gulp.

To run the non-development build locally from your machine, only the express server needs to be launched while setting the NODE_ENV environment variable to 'non-development'. This will cause the express app to update the location of where to find the static files to be served and also setup routes to handle the react client side routing used in the navigation menu for the single-page app.

NODE_ENV=non-development npm start

Note: on windows setting an environment variable like NODE_ENV=non-development is handled differently. Use the following command:

SET NODE_ENV=non-development
npm run start

Launch your browser and then point to https://manager.local.hs-crypto.test.cloud.ibm.com:3000/. Note the use of a domain and also a port change in non-development mode since we are no longer using 2 servers. You can not use localhost:3000 when running in non-development mode.

Contributing code

A good practice is to develop your code in a separate branch instead of master and then opening a pull request when you feel your code is ready to pull into the master branch. Committing and pushing code to your remote branch will trigger a Jenkins build of the service manager and run a series of unit/functional tests. See Continuous Integration below for more details. Opening a pull request (PR) and pulling code into master will also trigger a Jenkins build.

Before committing any code, one should always lint the code to fix any style issues. Additionally testing in development and non-development server mode should be considered based on the types of changes that are being committed to the remote master.

Run lint from the project root:

npm run lint

Appropriate testing should be completed before checking code into the Git repository. Please see the testing notes README.md file: Testing README.md in the project/test directory.

Continuous Integration

The zcrypto-service-manager job on running on http://9.42.74.230:8080/job/zcrypto-service-manager/ will detect can this repository for code changes and attempt to build a test docker image/container and run any available testcases. This applies to changes in any remote branches or pull requests.

A special extra step is made for the master branch that currently builds a github release tag and uses that tag information as input calling the remote build job called Build_And_Publish_HPCS_Service_Manager on the Z Build machine. This remote job has the task of checking out the master branch at the specified release tag level and building a signed docker image that is pushed to dockerhub. These images can be used to deploy instances of the service manager on Z architecture. You can view the job here: https://s215bchain17.pok.stglabs.ibm.com:8080/job/Build_And_Publish_HPCS_Service_Manager

Ensure the Jenkins job has successfully completed. If a failure occurs investigate why by examining the console output or TAP test results.

Deploying and Upgrading the Service Manager

Gulp tasks have been created to help with the process of deploying an instance of the service manager on a specified LPAR that is known by the hosting manager. The scripts that are executed make REST API calls to the hosting manager in order to drive docker commands and manage the associated VServer.

To deploy an instance: node_modules/gulp/bin/gulp.js deployAndBind --lpar <lpar-name> --port <port>
:large_orange_diamond:Note: The standard port for the service manager is 4000. This is not the port that docker image exposes. It is the host port that is mapped to the instance port. 4000 was chosen in order to avoid a conflict with the TKE server which runs on port 3000.

To upgrade running instances of the service manager: node_modules/gulp/bin/gulp.js UpgradeService
:large_orange_diamond:Note: This task relies on the value of the HpcsServiceManager.tag key found in the dev document of the hscs_images cloudant db in order to upgrade the service manager. This value must be manually updated before running the task until a process has been defined in order to programmatically update this field when a new docker image has been created. See example:

  "HpcsServiceManager": {
    "repository": "HpcsServiceManager",
    "tag": "0.0.1-ff8281c0",
    "ports": [
      {
        "port": 3000,
        "type": "tcp",
        "need_exposed": "yes"
      }
    ]
  },

Commit and Pull Request message convention

Please ensure that each commit relates to a single issue, to help avoid orphaned commits, and facilitate the review process
Include the issue number in the commit message title
Use a .gitmessage template to encourage this convention ( example template below )
Update local Git config ( ~/.gitconfig ) to automate use of this template via commit.template=~/.gitmessage

Example `.gitmessage` template

cat ~/.gitmessage

########50 characters############################
Issue nnnn: Description
########72 characters##################################################
Content of commit

Signed-off by: Dave Hay <david_hay@uk.ibm.com>

Validating `.gitmessage` template

git config -l

...
user.name=Dave Hay
user.email=david_hay@uk.ibm.com
url.git@github.ibm.com:.insteadof=https://github.ibm.com
...
commit.template=~/.gitmessage
...
core.repositoryformatversion=0
...