dr_back_training_project v0.0.6

NodeJS Express Backend Application Project Template in TypeScript

This repo demonstrates a NodeJS Express backend project template in TypeScript. It is a complete and runnable project with a few RESTFul APIs. It contains the essential configurations and dependencies for a TypeScript-based backend project and followed the conventions outlined in the documentation server. To find detailed explaination and usage of this template and other specifications, refer to the documentation server here.

⚠️ Important Notices

Modify this template accordingly:

1. This template uses the cors and helmet npm packages, they are handled by the API gateway when deployed online. Delete these packages and related code in the template.
2. The README.md file in the root folder (this file) is too verbose. The readme file here keeps verbosity for completeness, but in real projects, one should simplify this file to include only three sections: description, features, environment variables.
3. The metadata in the package.json file should be modified according to each project.

Environment Variables

Run the Project

The project requires NodeJS version >= 14.10.0 and Docker version >= 20. Make sure you have these installed before continuing. The project supports the following operations (all commands are run under the root folder):

• Source code linting with ESLint

  To lint the code in src folder, run the following command:

  npm run lint

  To generate a report in HTML from the linter outputs:

  npm run lint:report

  The report is generated under the folder outputs/eslint.

• TypeScript code type checking

  To statically check code correctness, run the type checking process with tsc:

  npm run check

• Documentation generation (use ts-doc to generate documentation from TypeScript source code comments)

  Using ts-doc to annotate the source code, the documentation on the Classes, Functions etc. defined in the code is automatically generated with the TypeDoc tool.

  npm run doc

  The output documents are under the outputs/docs folder in HTML format.

• Babel transpiling TypeScript into JavaScript

  To transpile the TypeScript code into JavaScript for NodeJS to execute, use babel:

  npm run build

  The transpiled code is under folder build, compatibile with your local NodeJS version.

  To transpile the source code into production version:

  npm run build:prod

  The transpiled and compressed code is under folder build, compatibile with NodeJS version 14.

  Note: the built code always uses CommonJS modules instead of ES6+ modules to avoid issues in production.

• Run the transpiled JavaScript with PM2

  Before running the project, make sure you have already built the code and have a non-empty build folder (PM2 is configured to target at the build folder).

  npm run start

  and then check the running status by pm2 status (Note: this command requires PM2 to be installed globally).

  The logs are stored in logs/pm2_out.log for stdout and logs/pm2_err.log for stderr. On success, there's no error outputs in pm2_err.log and no application restarts in pm2 status outputs.

  To check the RESTFul APIs, send Ajax request using Postman or other tools to the endpoint http://localhost:3000/weather. The API supports POST actions as follows:

  1. POST http://localhost:3000/weather/queryWeather with body {"username": "<name>","city": "<value>"}(replace <name> and <value> with your string)
  2. POST http://localhost:3000/weather/addWeather with body {"username": "<name>","city": "<value>", "weather": "<data>"} (replace <value> and <name> and <data> with your string)
  3. POST http://localhost:3000/weather/updateWeather with body {"username": "<name>","city": "<value>", "weather": "<data>"} (replace <vale> and <data> and <name> with your string)
  4. POST http://localhost:3000/weather/deleteWeather with body {"username": "<name>","city": "<value>"} (replace <name> and <value> with your string)

  On success, you should get responses and corresponding logs in the pm2_out.log file.

• Build a Docker image and push it to github container registry

  To build a docker image locally, run:

  npm run docker:build

  The image is tagged by "ghcr.io/dataReachable/dr_nodejs_backend_sample:v0.0.0".

  To push the image to ghcr.io, run:

  npm run docker:push

  To run the built image locally, run:

  npm run docker:run

  Port 3000 on localhost is exposed by default.

  To kill the running docker container, run:

  npm run docker:kill

  To remove the exited docker container, run:

  npm run docker:remove

  Note: docker related environment variables are defined in config/envars/.env.global.

Contents

The project structure follows this spec. Apart from the essential package.json and tsconfig.json configs, in this repo, we give samples for Docker configurations, ESLint configurations, Babel configurations, PM2 configurations, Jest configurations, GitHub Actions for CI/CD, etc.

1. Environment variables config/envars.
2. A sample package.json file with sample NPM production dependencies and dev dependencies (should be changed accordingly).
3. A tsconfig.json file to configure the TypeScript compiler tsc and the TypeScript documentation generator typedoc.
4. A babel.config.json to configure Babel as the ESLint parser and also transpile code to match the production environment.
5. Docker configurations Dockerfile to build and run NodeJS application image in both production and development.
6. PM2 sample configurations ecosystem.config.cjs for local development use only.
7. ESLint configurations .eslintrc.json to regularize coding styles.
8. Testing configurations using Jest.
9. Useful GitHub Actions in .github/workflows.

Description in Details

This section provides the descriptions of the above contents.

1. Environment variables config/envars

Use dotenv to define all environment variables in dotenv format. Normally, a global .env.global is created to define project-wide envars, and a local .env.app is created to define app-wide envars. The global variables are ONLY used by project-wide scripts, e.g. scripts in package.json file, whereas the application-level environment variables are used to define envars inside the application.

In this example, .env.global contains Docker related envars, e.g. tag name, container name etc., and PM2 related envars. .env.app contains all necessary envars required by the application, e.g. MongoDB connection string, port number etc.

2. package.json

Important contents in this file

• Production and development dependencies Must distinguish dev and prod dependencies. Putting useless dev dependencies into prod is forbidden.
• Forces to use NodeJS version >= 14.8.0
• Forces to use CommonJS modules
• Pre-defined useful npm scripts

The scripts uses cross-var to allow variable substitution across different platforms. No need to change the scripts in whichever operating system.

3. TypeScript configurations tsconfig.json

TypeScript and TypeDoc related configurations are defined in the root folder tsconfig.json file.

For TypeScript compiler tsc, the configuration suppresses the outputs from the compiler to use tsc only as a type checking tool (transpiling is done by Babel). For details on compiler options, refer to the official doc.

For TypeDoc, it suppresses the inclusion of this README.md file as the main page, and you can assign another *.md file for the main page. For configuration details, refer to the official doc.

4. Babel configurations babel.config.json

The babel configurations are defined in babel.config.json under the root folder. It defines a default configuration and 3 BABEL_ENV controlled configurations:

• lint: ESLint running environment configs
• test: Jest running environment configs
• prod: Production environment configs to transpile source code into production code

Babel plugins: @babel/preset-typescript, @babel/preset-env, babel-preset-minify

Module format: CommonJS to avoid compatibility issues

ES Module format still has some limitations with TypeScript, e.g. this issue.

Configurations: 1. Default configuration:

• Target: your locally installed NodeJS version (the same as process.versions.node)
• Module Format: CommonJS

2. ESLint configuration:

• Target: NodeJS version 14
• Module Format: CommonJS

3. Jest testing configuration:

• Target: NodeJS version 14
• Module Format: CommonJS

4. Production configuration:

• Target: NodeJS version 14
• Module Format: CommonJS
• Minification: Babel minifier and source code comment removal

5. Docker configurations Dockerfile

The docker configuration consists of a Dockerfile file and a .dockerignore file, both in the root folder. Docker operations involved are mainly image building and running locally.

• Building docker image:

  The root folder is the docker image building context. .dockerignore file ignores most irrelevant folders and files. The Dockerfile adopts multi-stage building with the first stage transpiling the source code using babel and second stage building the final image.

  The DOCKER_IMAGE_TAG global environment variable is required to tag the built image. The variable is defined in config/envars/.env.global.

• Running the built image locally:

  Locally running the built image is recommended to check the image before publishing to a container registry.

  The docker container uses application-level envars defined in config/envars/.env.app.

  Several global environment variables are required: DOCKER_IMAGE_TAG is the image to run, DOCKER_CONTAINER_NAME is the name of the running container, DOCKER_HOST_PORT and DOCKER_CONTAINER_PORT defines the port mapping.

6. PM2 configurations ecosystem.config.cjs

PM2 configurations are defined in config/pm2/ecosystem.config.cjs (Note: the file name must be EXACTLY this one). By default, it runs in development environment, i.e. NODE_ENV=development, but can also be switched to production environment.

The logs are stored into logs folder with names of pm2_err.log and pm2_out.log.

PM2 is only used in development. Don't include it in production. You're free to change the PM2 configurations.

7. ESLint configurations .eslintrc.json

ESLint is applied to regularise the coding styles, best practices and avoid potential errors in programming. It's also useful to avoid git merge conflicts in collaboration.

The configuration targets at TypeScript projects. The linter rules are a combination of pre-defined rules in popular community ESLint plugins and a set of customised rules. The rule plugins required are:

• ESLint defines the core rules in general
• TypeScript ESLint defines the core rules specific to TypeScript
• ESLint Import defines the rules for module import/export
• ESLint Comments defines the rules for ESLint comments
• ESLint Promise defines the rules for Promise in TypeScript
• ESLint Unicorn defines a set of community developed awesome rules
• ESLint Jest defines a set of rules for Jest testing code
• ESLint Jest Formatting defines the rules for Jest testing code styles
• ESLint TS-Doc defines the rules for tsdoc comment syntax

For the above rule plugins, we adopt the recommended rules in each plugin. Refer to the above links for rule details.

The customised rules are defined and commented in the single .eslintrc.json file in the root folder. Refer to this file for details on those rules.

A more detailed description on those rules can be found on the document server.

Note: Suggestions and modifications are welcome. However, do NOT change those rules without discussing with Travis (travis.yuan@datareachable.com).

In addition, in the root folder, the .eslintignore file is created to ignore linting on irrelevant folders/files, e.g. node_modules, config, build, etc. You are free to change this file to meet your needs.

8. Testing configurations

We adopt Jest testing framework to test TypeScript projects.

General instructions

• The tests include unit and integration testing. Unit testing focuses on isolated modules, e.g. controllers, services, utilities, etc. Integration testing focuses on the whole project and tests on the RestFul API endpoints.
• Jest configuration files in config/envars/testing folder with different settings for unit and integration testing, i.e. jest.config.js for unit testing and jest.int.config.js for integration testing.
• Code coverage is automatically caculated by Jest and the output reports are under the outputs/jest folder.
• The testing code is under folder test. For unit testing the name of the subfolder is the same as the module under test. For integration testing, the name of the folder is called integration. All testing code is only allowed in TypeScript. The names of the testing code file are *.ts or *.(test|spec).ts.
• When mocking a module is essential in unit testing, the mocked module is defined in __mocks__ folder under the corresponding module folder in src. For example, to mock a service, create the mocked module under module/service/__mocks__. When mocking functions, they can be directly created in the testing code file.
• Testing environment variables are defined in config/envars/.env.test, e.g. testing database connection string.
• When necessary, we use the supertest package to test HTTP Restful APIs.

Details

• Unit Testing

  Unit testing covers tests on isolated modules in controller, middleware, model/service and utils. Code coverage is automatically caculated on the above folders.

  Unit testing might start emphemeral HTTP servers to listen to HTTP requests.

  In unit testing, mocking functions and modules are often needed. When database connections are required, you can either use mocked databased, e.g. @shelf/jest-mongodb, or connect to a real testing database.

  Examples of unit testing are given under test/controller and test/service.

• Integration Testing

  Integration testing covers the whole project so that modules not included in unit testing are tested, e.g. router, bin. Code coverage is caculated on all folders in the project.

  Integration testing starts an HTTP server that listens to HTTP requests. The HTTP server should be gracefully shutdown after the testing.

  In integration testing, when database connections are required, connect to a real testing database.

  Example of integration testing is given under test/integration.

Note: clean up databases after testing if you connect to a real database.

Running the Examples

In this sample project, we provide two testing related npm commands in package.json for unit testing and integration testing respectively.

• Unit testing:

  npm run test

  This runs the testing code in test/controller and test/service folder. When testing controllers, we use mocked services. When testing services, we connect to a real database whose connection string is given in config/envars/.env.test.

• Integration testing:

  npm run test:int

  This runs the integration testing code in test/integration folder. The testing connects to a real database given in config/envars/.env.test.

  The code coverage reports are in the outputs/jest folder. Open the index.html file in a browser to see the HTML version of the report.

9. GitHub Actions

GitHub Actions is the main CI/CD tool. This sample project provides a few action templates.

The developers should learn the basics on GitHub Actions, because the developers are required to make small modifications on them to meet specific requirements.

List of action templates provided:

Sample Project Structure

Template users should follow the project structure in general. Minor changes are allowed, but the backbone structure should NEVER be modified without discussing with Travis, especially, the src folder.

Overall structure

Details inside the src folder

Details inside test folder