Polling-infinity NPM

Polling^Infinity

Poll an EHR webservice for changes and send them through the RedoxEngine.

Design: https://docs.google.com/document/d/1POBcWWog8eOp_FiLVjWDEOA13gIQrpvLfmVLGyPIpWM/edit

Making Changes to the Code

If functionality changes are made, it is expected that they are documented in the web sequence diagrams in the diagrams directory of this service. To view the diagrams and modify within a GUI, please install the VSCode Draw.io integration extension by Henning Dieterichs.

RedoxLocal (rl-k8s) Commands

Polling^Infinity uses rl to run local instances of the services that make up the entire polling platform. Refer to the RL setup documentation for instructions on getting your machine prepared to use RL.

The following manifest targets are available for this service:

rl run polling-infinity - Provisions and starts the Web API service in Kubernetes, running the migrations job (see below) and launching a polling-scripterator web API and rl-postgres-redoxservices services via redox.yml dependency (if not already running).
rl run polling-infinity:migrate - Creates the migration job in Kubernetes, which applies migrations to the pollinginfinity database in rl-postgres-redoxservices.
rl run polling-infinity:cleanup - Provisions and starts the clean-up cron job service in Kubernetes, immediately running all jobs.
rl run polling-infinity:workflow-sequential - Provisions and starts the Workflow stage FQS Sequential worker service in Kubernetes.
rl run polling-infinity:supplemental-sequential - Provisions and starts the Supplemental stage FQS Sequential worker service in Kubernetes.
rl run polling-infinity:workflowfinalize-sequential - Provisions and starts the Workflow-Finalizer stage FQS Sequential worker service in Kubernetes.
rl run polling-infinity:send-sequential - Provisions and starts the Send-To-Engine stage FQS Sequential worker service in Kubernetes.

Note that the above run commands use the latest built/released version of the Docker image for polling-infinity, not your local code/changes. In order to create an instance with your local code, you need to use rl dev <target> instead. Note that "automatically" run dependencies (like migrations and scripterator) don't "inherit" the dev command — rl dev polling-infinity will result in rl run polling-infinity:migrate being run, which can be surprising. Remember to run those "upstream" dependencies manually with rl dev <target> first to get local code in the containers.

If you need to reset the Docker image based on your local code (e.g. if rl dev <target> seems to show old code), you have to possibly do some of the following:

rl remove <target> to clear things out
Use Lens or similar commands to remove things like Jobs, ConfigMaps, or other things you want to rebuild "from scratch".
Delete the docker image on your local machine with something like docker image rm <image-ID>.

Running Local Tests

Local tests can be run with cicd module test, which uses the test-compose.yml file to instantiate services for integration tests. There are commented out volumes mount points and run-forever-do-nothing command versions in that file. By un-commenting the volume mount and swapping the command line for the commented out version, the local-test runner script will start up the container and do nothing, but apply your code edits to the container as you make them. That means you can docker exec -it into the test container and run tests yourself with something like npm run test and edit/re-run them without lift/stop delays.

Development Commands

npm run db:create creates a dev and test postgres database.
npm run db:setup runs latest migrations for dev and test databases.
npm run db:migrate:make creates a new migration file in /src/db/migrations.
npm run db:migrate:latest runs "up" migrations.
npm run db:migrate:rollback runs "down" migrations.
npm run lint runs tslint and displays errors.
npm run lint:clean runs tslint and fixes linting errors when possible.
npm run test compiles the typescript and then runs all tests in the lib/**/*.spec.ts files. A code coverage report is also generated, and will display coverage errors if there are any.
npm test:debug runs npm run test in debug mode.
npm start <worker type> starts a worker and watches for any changes to the .ts files. When there are changes, it recompiles the typescript and restarts. Worker types include:
- scheduler - Keep track of source build in the RedoxEngine and periodically place jobs on the polling queue table.
- workflow - Grab jobs off the polling queue table and perform HTTP requests to generate a batch of changed records. Place the batch of changed records on FQS using the fqs-enqueuer service.
- supplemental concurrent - Perform HTTP requests to generate extra information for changed records in a batch, then create a list of messages that are ready to be sent to the RedoxEngine.
- supplemental sequential - Place individual messages from the batch on FQS using the fqs-enqueuer service.
- send concurrent - No work to do in this step yet; messages are simply passed along to the the sequential worker.
- send sequential - Attempt to send the message into the RedoxEngine.
- cleanup - Every few minutes, drops the entries in error and fqs archive tables that are older than a day or two.