document-service v1.714.0
Document Service
Document Service allows consumers to get and update data about Endpoint Documents.
Table of contents
Description
The Document Service manages the state related to documents that are shared between our customers and operations teams. All requests to create, access, or modify documents - by both internal and external systems - are serviced here.
Further documentation on the service can be found here.
Note: In order to support multi-tenancy in our platform, apps/document-service/src/AppModule.ts#32
configures the application to utilize the common endpoint auth guard. It is configured to only be used in non-local development environments to enable better developer experience.
Technologies
This service is written in TypeScript, using the NestJS framework. It is packaged and run as a Docker container, which internally contains the NodeJS runtime.
Other significant technologies to note are:
- ExpressJS the web framework used (read more about Nest and Express here)
- DynamoDB the database service used
- PostgreSQL the relational database service used
- TypeORM manages the repository layer and migrations.
Installation
In order to run the service locally, you will at minimum need to have docker installed. See local development instructions for details. If you want to run scripts from the package.json
outside of the docker container, then you will need to ensure that your local node version is compatible with the version range listed in package.json -> engines -> node
.
Database (PostgreSQL)
The service is backed by PostgreSQL. TypeORM manages the repository layer and migrations.
Run the database locally.
docker-compose up -d db
Migration run scripts
The run scripts in package.json
allow easy execution of the 5 basic migration run scripts: create
, generate
, show
, run
and revert
,
Show pending migrations
In the local environment
yarn orm:mig:show:local
It shows all ready-for-execution migrations. It iterates all created files in the dist/migrations folder.
Notice: you must build the app to get the javascript-format migrations.
Add a new migration
Empty migration:
yarn orm:mig:create -n <migration_name>
It generates a migration class with empty methods under the following path:
./migrations/<numeric_timestamp>-<migration_name>.ts
, with class name <migration_name><numeric_timestamp>
Example
migrations/1655932765632-MigrationName.ts
Revert migrations
In the local environment
yarn orm:mig:revert:local
It reverts all the already-executed queries in a previous run. e.g. the ones defined in the down()
method of the migration classes.
In other environments
Ask devops to perform the following actions:
- backup the database
- navigate to the running container via shell
- DEPLOY_ENV= yarn orm:mig:revert
Local Development
It is possible to execute the Document Service almost entirely using local infrastructure. This is made possible by auto-mocking several external services and using localstack to mock AWS infrastructure.
If one has not been created already, create a .env
file at the root of the application. This should be automatically git-ignored. The file should contain values for the following variables:
1. (Required) Set NPM_TOKEN=<your NPM token>
2. (Optional) Set the LAUNCH_DARKLY_SDK_KEY
for local development in the root level .env
file. This can be found in AWS secrets manager
.
To start the document-service and all necessary infrastructure, run
docker-compose up
This will start the service using local
as the DEPLOY_ENV
and watching for any changes via the start:dev
script.
To start the document-service in debug mode, run 1. start the document-service and all necessary infrastructure
docker-compose up
- list all running containers
docker container ls
- scan the list for the container ID of document-service
- stop only document-service container
docker container stop <container-id-of-document-service>
- start the document-service in debug mode
DEPLOY_ENV={ENV} yarn start:dev
Known Issues
- The Token Exchange Service is not mocked and is still being used by the auth guard. We haven't found a way yet to mock a valid JWT response from the service, and as a result the service's call to OCR Service is failing. This call initiates a side-effect and should not inhibit testing of the Document Service.
Run Scripts
Run dev
DEPLOY_ENV=local yarn start:dev
Scaffolds commit messages and enforces required format using Endpoint's Commitizen configuration.
yarn commit
Runs linter
yarn lint
Fixes linter errors
yarn lint:fix
Resets node_modules, clears yarn cache, and reinstalls node_modules
yarn reset-modules
Runs integration tests using cypress.
yarn run test:cypress
Document Service SLA
Swagger Links
Production
Use this space to describe any requirements that need to be set up before the app can be deployed to production.
Environment variables
DEPLOY_ENV
This variable designates which of our environments this service is running in (i.e. local
, development
, integration
, test
, sandbox
, staging
, production
).
DYNAMO_REGION
The AWS region to connect to for dynamo. In development or testing environments with a localstack dynamo instance this value should be local
.
TODO: Document why this env variable needs to be set and what precisely it influences.
PG_HOST
The URL of the PostgreSQL (RDS) host to connect to.
- For environments migrated into environment-specific AWS accounts, this value is specified in the corresponding file in the
ep-k8s
repo - For environments still running in the "legacy" AWS account, this vlaue is specified in the corresponding
document-service/config/.ENV_NAME.env
file
PG_PASSWORD
The password for the specified PG_HOST. This value is configured and stored in AWS secrets manager.
SENTRY_DSN
The sentryDSN
url from Sentry to enable sentry logging of errors, if not set it will disable sentry reporting for errors.
S3_SERVER_SIDE_ENCRYPTION
The type of encryption to set for objects stored in S3. Information about server side encryption (SSE) can be found in the S3 SSE documentation.
S3_REGION
The AWS region where the S3 bucket for this service resides.
S3_BUCKET_NAME
The name of the S3 bucket to connect to.
ENABLE_REQUEST_LOGGING
Enable logging for incoming requests through the Endpoint Core Module when set to true
(see docs).
PROVIDER_TEMPLATES_SYNCHRONIZE_CIRCUIT_BREAKER_MILLISECONDS
Max number of milliseconds of timeout for synchronizing document templates from provider
Testing
Unit Tests
To run unit tests:
yarn test
Testing DynamoDB Queries
We use jest-dynalite to enable validation of dynamo queries.
Because not all unit test suites need an instance of dynalite, it is only spun up by manually including it in your tests. The simplest way is to include the following import at the top of your test suite:
import "jest-dynalite/withDb";
This will add the required setup and teardown hooks for dynalite. If you need more granular control, see further options in the Advanced Setup instructions.
Cypress Tests
User Admin credentials are required to run the cypress test. The credentials are stored in the
Cypress Test User (dev, int, stg)
entry in the Engineering vault.
1. Create a cypress.env.json
file at the project root. This is already git-ignored and thus safe for secrets.
2. Add the following, replace {username} and {password} with the values retrieved from the Engineering Vault
{
"ADMIN_USER_NAME": "{username}",
"ADMIN_USER_PASSWORD": "{password}"
}
Example: run the following in two shells to run the cypress tests locally against a local service process using the development config. Cypress is setup to run integration test.
yarn run test:cypress
By default, the test will run against your local server. An env variable can be passed to cypress to
specify which environment you want to run the test (development, integration, staging
).
yarn run test:cypress -e name=staging
Note:
- For non-production environments a
transactionId
anddocumentId
will be auto provided as env vars when running the cypress run script. For more details see file/cypress/plugins/setUpTransaction/setUpTransaction.ts
. - If you need to add a new dependency to the E2E test cases, you will need to update test-automation.yml pipeline to install the needed dependency for the test runner
Adding and running tagged test cases
Tags help us to organize and run tests based on the type of case they are. Right now, we only tag integration
test cases. We use the tagging from the cypress/grep package to help us tag and run cases.
Adding tags to a test case
To add a tag to your test case you'll need to add { tags: ['@integration'] }
to your it statement.
Ex.
it('C290718 - Open Order - Prelim conversion to Escrow', { tags: ['@integration'] }, () => {
Running tagged test cases
To run tagged test cases you'll need to add grepTags={tags}
to your cypress run script, where {tags}
is replaced with the text of your tag
Ex.
yarn cypress open --env name=integration,grepTags='@integration'
Performance Tests
We use K6 for our performance tests. You'll need to have K6 installed on your machine before attempting to execute the tests.
Scripts
Upload Latency
This script measures the time between a file upload completing and that file being available in Verse. It accomplishes this by uploading a file to document-service and then polling the service for that document's etag. It is configured to simulate 5 virtual users and have each user upload 100 files for a total of 500 uploads.
Environment Variables
The following environment variables are required:
TX
- a transaction IDTOKEN
- a token to include in theAuthorization
header of outgoing requestsFILE
- a path to a file on your local machine to upload
Running the Test
Make sure you're connected to the VPN, then run:
k6 run -e TX="TRANSACTION_ID" -e TOKEN="TOKEN" -e FILE="PATH_TO_FILE" upload-latency.js
Be sure to substitute a valid transaction ID and authorization token for the TX
and TOKEN
environment variables.
Download Availability
This script measures the availability of downloading a file. It is configured to simulate 5 virtual users and have each user download 300 files for a total of 1500 downloads. This script can set up a document, upload a document and then download it.
Environment Variables
This script uses the config.json file to set its environment variables:
EP2_URL
- ep2-services endpoint (auth token generator)DOCUMENT_SERVICE_URL
- document-service urlFILE_PATH
- file to upload and downloadCOGNITO_CLIENT_ID
- cognito client id to get Auth TokensCOGNITO_CLIENT_SECRET
- cognito client secret to get Auth Tokens
Running the Test
Make sure you're connected to the VPN, then run:
yarn build
k6 run download-availability.js
Upload Availability
This script measures the availability of uploading a file. It is configured to simulate 5 virtual users and have each user download 20 files for a total of 100 uploads. This script can set a document and upload the document's content
Environment Variables
This script uses the config.json file to set its environment variables:
EP2_URL
- ep2-services endpoint (auth token generator)DOCUMENT_SERVICE_URL
- document-service urlFILE_PATH
- file to upload and downloadCOGNITO_CLIENT_ID
- cognito client id to get Auth TokensCOGNITO_CLIENT_SECRET
- cognito client secret to get Auth Tokens
Running the Test
Make sure you're connected to the VPN, then run:
yarn build
We recommend execute just a single scenario to avoid uploading a lot of files to our environments. Scenario list:
- single_user (a single user uploading 100 files)
- single_user_parallel (a single user uploading 100 files in parallel in 30 seconds)
- five_users (fice users uploading 20 files)
- constant_request_rate (2 users uploading 100 files per minute in 5 minutes)
k6 run --env SCENARIO=constant_request_rate upload-availability.js