panther_campaign_bridge v2.0.0

Panther Campaign Bridge v2

Purpose

The bridge acts as the central sync-tool between the Panther (TMP) world and the already existing Perengo infrastructure.

It takes care of the following procedures:

• create new campaigns based on a new job entering an existing flight
• pauses / resumes campaigns based on the corresponding status of a Panther job
• keeps Panther <-> JPAL side up-to-date
• takes care of budget-and-goals actions (pause/resume) based on the spent budget
• synchronizes segment-/flight id on the corresponding Perengo supercampaign
• decommission supercampaigns in case the corresponding Panther job has been removed from the feed
• executes job interventions on supercampaigns in an existing flight

Prerequisites

In order to be able to run the bridge, we need to ensure the following

• has Node JS (incl. NPM) in a version >10.85
• has connection to a running Panther-API
• has connection to a running JPAL
• has access to the Postgres (RDS) database
• has access to an existing Redis server
• has access to Elasticsearch
• can send messages to Slack

First time installation

git clone git@bitbucket.org:openjob/panther-campaign-bridge.git
cd panther-campaign-bridge
npm i

Now, we also need to take care to set the required environment variables by either creating an .env file in the root of the project folder or by properly export-ing them

\# describes the current environment - will be sent along to Elasticsearch
ENVIRONMENT=local

\######################################################
\# current db environment
POSTGRES_HOST=127.0.0.1
POSTGRES_PORT=5432
POSTGRES_USER=root
POSTGRES_PASSWORD=\*\*\*\*\*\*
POSTGRES_DATABASE=perengo_v2
\######################################################

\######################################################
\# uncomment when you want to run in single thread mode
\#IS_MULTITHREADED=false
\######################################################

\######################################################
\# can be used to replay a previously recorded session
\# the session key needs to be available in the 
\# attached "current db environment" database
\# also, the replay just works in single_threaded mode
\#
IS_REPLAY=false
REPLAY_SESSION_KEY=\*\*\*\*\*\*
\######################################################

\######################################################
\# defines the global log level
LOG_LEVEL=debug
LOG_SQL=true
LOG_INSERTS=false
\######################################################

\######################################################
\# used to send n-jobs per chunk to jpal
MAX_CHUNK_SIZE=250
\######################################################

\######################################################
\# AWS settings
AWS_ACCESS_KEY_ID="\*\*\*\*\*\*"
AWS_SECRET_ACCESS_KEY="\*\*\*\*\*\*"
AWS_DEFAULT_REGION="us-east-1"
\######################################################

\######################################################
\# Panther-API and JPAL connection endpoints
PANTHER_API_ENDPOINT=http://localhost:3000/api/v1
PANTHER_API_TOKEN=\*\*\*\*\*\*
JPAL_ENDPOINT=http://localhost:5000/jobs/api/v2.0/dsp
\######################################################

\######################################################
\# db-migrate local-/stage-/stable- and prod definition
PANTHER_STAGE_DBUSER=root
PANTHER_STAGE_DBPASSWD=\*\*\*\*\*\*
PANTHER_STABLE_DBUSER=root
PANTHER_STABLE_DBPASSWD=\*\*\*\*\*\*
PANTHER_PROD_DBUSER=root
PANTHER_PROD_DBPASSWD=\*\*\*\*\*\*
PANTHER_LOCAL_DB=perengo_v2
PANTHER_LOCAL_DB_URL=postgres://localhost/perengo_v2
PANTHER_LOCAL_DBUSER=root
PANTHER_LOCAL_DBPASSWD=\*\*\*\*\*\*
\######################################################

Running Campaign Bridge

In its current version v2, the campaign bridge can be run in several modes.

1. Multi-threaded standalone mode:
   • start command npm run start
   • creates one master thread and up to n-worker threads (cpu cores -1)
   • master thread is responsible to spawn the worker threads
   • master thread fills the workers queue with customer ids to be processed (de-duplicated)
   • master thread creates a web interface and provides (basic) controlling
   • worker threads consume the workers queue, pick exactly one customer id and perform the customer processing
2. Multi-server mode (master):
   • start command npm run start-master
   • one server can be picked to run the master part (as sketched above)
3. Multi-server mode (worker):
   • start command npm run start-worker
   • unlimited amount of server instances which each spawn n-threads (cpu cores) to process the worker queue
4. Single-threaded standalone mode:
   • start command npm run start but environment variable IS_MULTITHREADED=false
   • infinitely loops in sequentially working through a list of customers
5. Single-threaded lambda (serverless) mode:
   • start command: no direct start command, but the file simulate.js can be executed to simulate a lambda run
   • sequentially works through a list of customers and stops (runs every 30 minutes)