menlolab-runner v0.13.2

Runner

The agent deployed on private and public infrastructure to manage tasks. Runner communicates back to MenloLab via Marathon-Runner authenticating a persistent web socket connection. The default runner state assumes it has been installed as a system service. See development below for information on running local runner instances.

Install - DEB

Short way

sudo apt-get update -y; sudo apt-get install wget -y; \
wget -qO - https://repo.menlolab.com/r-install.sh | sudo bash

Long way

sudo apt-get update -y; sudo  apt-get install gnupg1 wget -y; \
sudo echo 'deb https://repo.menlolab.com bionic main' > \
/etc/apt/sources.list.d/menlolab-runner.list; \
wget -qO - https://repo.menlolab.com/pub.gpg | sudo apt-key add -; \
sudo apt-get update -y; sudo apt-get install menlolab-runner -y

Install - RedHat Based Distributions

To install runner run the following command:

wget https://rpm.menlolab.com/rpm-install.sh && chmod u+x ./rpm-install.sh && ./rpm-install.sh

Server -> export GPG Key

gpg --export --armour email > /etc/pki/rpm-gpg/MenloLab-Runner

Client -> import GPG Key

rpm --import /etc/pki/rpm-gpg/MenloLab-Runner

Add Repository

yum-config-manager --add-repo repository_url

For more information click here

Commands

Menlolab Runner
All runners must be register before connecting to Menlolab.

Options:
  -V, --version          output the version number
  -c, --config <config>  specify the config file to be used, defaults to /etc/menlolab-runner/config.yml
  -j, --job-dir <path>   specify the location of the job folder, defaults to {HOME}/.menlolab/runner/
  -h, --help             output usage information

Commands:
  register-public        register as a private runner (MenloLab Only)
  register               register this runner to a user or group
  deregister             unregister this runner from a user or group
  add-tags [tags...]     add tags to this runner
  rm-tags [tags...]      remove tags from this runner
  start                  start the runner

Supporting Files and Folders

In the table above BOLD OPTIONS are configurable

Config File

The behavior of runner is determined be the config file. The config is manipulated by commands. The runner daemon will watch the config file for changes and update itself as necessary (to be developed).

Job Directory

The job directory is where the runner stores information about jobs and its current state. By default the runner will use the process user home folder {HOME}/.menlolab/runner/. A flag can used to specify the location of the job folder.

Development

There are several methods for running the runner for local development. See the package.json for the various scripts that support local development.

npm run debug

The npm run debug command uses a config.yml located at the root of this repo. Copying or linking to a config is the easiest way to satisfy the requirements of the npm run debug.

ln -s ./config/mlo_public.yml config.yml

Testing

S3rver is used to fake S3 interactions during testing. All testing should be developed function offline (with the exception of retrieving public images and boxes).

Persistent State

The runner uses xstate to keep track of the steps taken by the process. The current state is stored in the /var/lib/menlolab-runner/jobs/.persistence.json, which is used to restore running state in case runner crashes.

Task Handlers

Runner tasks can be executed using various supported methods including shell, docker and vagrant. Every handler will use the Job directory as a staging environment regardless of whether or not that execution environment has its own filesystem.

One Time Runners

The runner maybe deployed to hardware to handle a task designed for specific dedicated hardware. To be developed.

Sentry

To be developed The runner is configured with Sentry to configure catch errors in the wild. This option is on by default and can be flagged off in the config.

Deploy

pkg

pkg is used to create a single standalone binary. pkg handles file paths in a unique way in the binary. Please see the pkg documentation when working with file paths in runner.

DEB - node-deb

node-deb packaged the binary along with several other template generated files into a .deb package.

DEB Distribution - Aptly & Repo

The repo is managed by aptly running on GCP instance. Aptly manage a local DEB repo on its VM. Aptly updates necessary metadata files as new versions of the package are added to the repo. The local repo is copied to a GCP storage instance which is configured to be reachable by the web. The GCP storage repo is accessed by a 302 rewrite to avoid load balancer costs. This 302 is provided by a Google Domains synthetic record.

RPM

Docker container with rpm-build and redhat-rpm-config is used to build the .rpm package. The built package is then exported to GCP Bucket

RPM Distribution

A local repo is created with the link to MenloLab RPM repo. The local repo is used to allow the user to update the runner.

Scope

This package will be release as an opensource repository. At no time is necessary to bring any secrets or keys into this repository.