hesburgh_util v2.0.0

hesburgh_utilities

hesdeploy documentation

This project contains shared utilities to be used across all projects in multiple languages - currently Javascript and Python.

NOTE: Unless otherwise stated all utilities are avaialable in all supported languages, even if they don't have usage examples in that language.

NOTE: On AWS heslog defaults to only logging INFO, WARN, and ERROR levels while locally all default to being on. To override in either environment set the env var HESBURGH_DEBUG=true|false to the appropriate value for what you want.

Installation

This project requires yarn, so first run npm install -g yarn. To install projects locally run ./setup.sh in the home directory. This will install the python module and link the javascript module to your global namespace.

JS

To install in a javascript project run yarn add hesburgh_util in said project.

PY

To install the python library to another project, run pip install hesburgh-utilities --target hesburgh This will pip install to the target directory (hesburgh) so that all it may be packaged with the rest of the code.

Utilities:

Logger (heslog)

Sentry

Sentry is cross-platform application monitoring, with a focus on error reporting. Sentry is integrated into heslog. The basic workflow

1. create project in Sentry
2. config app to reference Sentry and add logging
3. check logs in Sentry

The function setHubContext('project') is used for the app to reference the Sentry project. It works by taking the argument passed in and going to AWS parameter store for the projects Sentry DSN. It should be stored at /all/sentry/production/project/dsn

By default messages with log level of warning or error are automatically sent to the speicified Sentry project. Any message can be logged to Sentry by adding a 'sentry' flag.

• JS EX: heslog.info('hello world','','sentry')
• PY EX: heslog.info('hello world',sentry='y')

Grock output

LEVELS (DEBUG|TEST|VERBOSE|INFO||WARN|ERROR)
%{TIMESTAMP_ISO8601:timestamp} ::%{LEVELS:level}:: %{GREEDYDATA:message}

Example Output

A basic log prints date, time, logtype, context (key,value pairs separated by |'s) and finally the message eg:

2017-02-08 21:06:27.328743 ::DEBUG:: lambda_requestId=some_id | request_code=github | api_requestId=some_other_id | printing this debug message

Note: date/time will not be printed on AWS as they already add that to logs. It can easily be added back if we ever need it though.

Formats output for easier, uniform parsing

Usage in Python

import heslog

heslog.setHubContext('snek') # references /all/sentry/production/snek/dns for Sentry config
heslog.setContext({"foo": "bar"})
heslog.debug("message")
heslog.addContext({"bar": "baz"})
heslog.error("message", foo="baz", context="here") # this overrides foo for this message only
heslog.removeContext("bar", "foo") # remove foo and bar from the context
...
heslog.setLevels(heslog.LEVEL_ERROR)
heslog.debug("message") # will not output to log
heslog.error("message") # only this function will output to log
...
heslog.setLevels(heslog.LEVEL_WARN, heslog.LEVEL_INFO)
# now only warn and info will output

Usage in JS

const hesburgh = require("hesburgh_util");
const heslog = hesburgh.heslog;

heslog.setHubContext('armor')
heslog.setContext({foo: "bar"});
heslog.debug("message", {foo: "baz", context: "here"});
heslog.setLevels(heslog.levels.debug);
heslog.setLevels(heslog.levels.debug, heslog.levels.error);
...
// same format as python from here on out (which is the whole point)

Utils (hesutil)

JS

const hesburgh = require("hesburgh_util");
const hesutil = hesburgh.hesutil;

var value = hesutil.getEnv("key", "defaultValue");

//safely get value from a dictionary
var foo = hesutil.dictGet(testDict, "key", "defaultVal");

// check if a dictionary has a key
var has = hesutil.dictHas(testDict, "key")

Test data (hestest)

NOTE: The file datakeys.json is expected to be in the test data (whatever you name it) folder, schema described below

datakeys.json

For the most part, this is just a generic json file, it will be what's returned if the netid is found in the get call. There is one extra option: if the value is { "load_file": "file.json" } that file will be loaded and replace the value with the contets of the data key contained within. The file.json value must be a relative path to the datakeys.json file and must contain json with a top level key data. The data key requirement allows the file to contain 1 string value or any other type if needed, eg "data": "long string value" or "data": { "obj": "value"...}

// datakeys.json
{
  "key": "value",
  "foo": { "load_file": "bar.json" }
}

// bar.json
{
  "data": {
    "barkey": "barvalue"
  }
}

An example of what the return will be with the above data:

foo = hestest.get("test_netid")
# foo = { "key": "value", "foo": { "barkey": "barvalue" } }

The load_file key can be used at any level, the entire tree will be traversed to load all files. So in the example bar.json could also contain a load_file to load another file.

Python

from hesburgh import hestest

# __file__ is required as the first param in python, it denotes the location of the calling file
# Then the second param is the relative path of the test data folder
hestest.init(__file__, "../testdata")
hestest.get("hbeachey")

JS

const hesburgh = require("hesburgh_util");
const hestest = hesburgh.hestest;

# In JS the setup is slightly different, the first param is the relative path
# The second param is the test data folder name
hestest.init("./", "testdata")
hestest.get("hbeachey")

Timer (in hesutil)

Python

from hesburgh import hesutil

timer = hesutil.Timer()
timer.start()
# do stuff
dt = timer.step()
# do stuff
totalTime = timer.end()
avgTime = timer.getAvgStep()