@moogsoft/mars-sdk NPM

The mars-sdk

A library of utilities intended to make the development of collector node.js script both easier, and more robust.

The core utilities can be required and used in the simple form:

const utils = require('@moogsoft/mars-sdk');

utils.debug('Mooo');

Or in the prefered deconstructed method:

const {
    debug,
} = require('@moogsoft/mars-sdk');

debug('Mooo');

Extended utilities can be loaded as follows:

const rest = require('@moogsoft/mars-sdk/rest');
const Event = require('@moogsoft/mars-sdk/Event');

Utility Reference

Core Utilities

Loggers

Discovery Utilities

Configuration Utilities

Shedulers

API Object builders and formatters

General Purpose Utilities

Extended Utilities

REST

REST

debug

debug(string)

Writes a JSON formatted string to stdout, with a "level" key set to "debug" and the supplied string as the value for "msg", which is processed by the collector as a log message.

info

An "info" level log message. See debug for details

warn

A "warn" level log message. See debug for details

error

An "error" level log message. See debug for details

jsonParse

jsonParse(string)

A wrapper for JSON.parse()

Returns: <Object> Parsed object

fileExists

fileExists(filePath)

Checks to see if a given file exists

filePath <String>
Returns: <boolean>

canRun

canRun(cmd)

Checks to see if a given command is executable

cmd <String>
Returns: <boolean>

hasCmd

hasCmd(cmd)

Checks to see if a given command exists, and is executable by the current user

cmd <String>
Returns: <boolean>

procRunning

procRunning(proc)

Checks to see if a given process/task (name) is running

proc <String> Name of processes
Returns: <boolean>

DiscoveryResult

const DiscoveryResult = require('@moogsoft/mars-sdk/DiscoveryResult');
const result = new DiscoveryResult()
    .<setMethod(value)>
    [...]

An external class to generate a well formed discovery result object.

Returns: <Object>
Methods:
- .setMoob(moob) Required (If single)
  - moob: <String> The Managed Object that has been enabled
- .setMoobs(moobs)Required (If multiple)
  - moob: <Object> A list of Managed Objects that have been enabled (if multiple are suported)
- .setReason(reason) Optional
  - reason: <Object> A well formed reason object built with the Reason method
- .setActive(active)
  - active <boolean>

Examples:

Initialize and turn on a MAR:

let builder = new DiscoveryResult.setMoob('system').setActive(true);

Disable a MAR based on subsequent logic:

builder = builder.setActive(false);

Reason

An external class to generate a well formed Reason to be used by the DiscoveryResult.setReason() method

Returns: <Object>
Methods:
- .setRecoverable(recoverable)
  - recoverable: <boolean> Is it completely undiscoverable, or should the user be prompted for miising information?
- .setMsg(msg)
  - msg: <String> A message that will be presented to the user
- .setType(type)
  - type: <String> A categorization of the type of discovery "failure". While this can be any string, the following categores must be set from the constants module:
    - INSUFFICIENT_PRIVILEGES
    - INVALID_CREDENTIALS
    - MISSING_CREDENTIALS
    - MISSING_PROCESS
    - MISSING_COMMAND

Examples:

Disable a MAR because a process was not detected:

const {
    MISSING_PROCESS,
    MISSING_COMMAND,
}  = require('@moogsoft/mars-sdk/constants');

builder = builder.setActive(false)
    .setReason(new Reason()
        .setRecoverable(false)
        .setMsg('redis server process not found')
        .setType(MISSING_PROCESS);

constants

The constants module exposes common constants used during discovery and collection. Available constants:

INSUFFICIENT_PRIVILEGES
INVALID_CREDENTIALS
MISSING_CREDENTIALS
MISSING_COMMAND
MISSING_PROCESS
COUNTER
GAUGE

sendDiscovery

sendDiscovery(discovery)

Writes a JSON string to stdout containing the results of the disovery, which is interpretted by the collector

discovery <Object> A well formed "disovery" object

getConfig

getConfig()

Returns: <Object> An object containing configuration (if any) for that MAR, for that collector instance

getCredentials

getCredentials()

Returns: <Object> An object containing credentials (if any) for that MAR, for that collector instance

exportConfig

exportConfig(config)

Writes a JSON string to stdout containing the config (see getConfig(), which is interpretted by the collector

config <Object>

getMarDir

getMarDir()

Returns the absolute path of the MAR script that is being execute. Useful for accessing external files/artifacts in the MAR

Returns: <String>

registerScheduled'

registerScheduled(func)

Takes a function and calls if if the first input argument is the functions name. Useful when calling separate functions on different schedules for a scheduled collector.

func: <function> A collector function that generates events/metrics

carousel'

carousel(interval, func, args)

Takes an interval, function of arity 1, and a list of arguments to pass to a function. Calls the function with each argument in turn, spreading the calls across the supplied interval. (A form of load balancer). e.g: If the interval is 10s, and there are 10 args, the function will be executed every second.

interval: <Number> Time interval in ms
func: <Function> The function to be executed
args: <Array> The list of arguments

Metric

An "extended" class to generate a well formed Metric to be used by the sendMetrics() method

Returns: <Object>
Methods:
- .setData(data)
  - data: <Number> a time series value to be observed
- .setMetric(metric)
  - metric: <String> The name of the metric or "check". (e.g: NetworkIO)
- .setSource(source)
  - source: <String> the unique name, or "namespace" of the resource the metric is associated with
- .setKey(key)
  - key: <String> A sub key of the resource (e.g: the network interface)
- .setTime(ts)
  - ts: <unixtime> A unix timestamp in ms. The collector will provide on if not set
- .setDescription(desc)
  - desc: <String> A description of the metric
- .setUtcOffset(offset)
  - offset: <String> utc_offset The timezone of the metric, defaults to the current timezone
- .setAdditionalData(ad)
  - ad: <Object> Any addition JSON metadata that is not suited for tags, but might be useful for upstream processing or for a user
- .setTags(tags)
  - tags: <Object> An object, containing a list/array of key:value pairs
- .setType(ad)
  - type: <String> The metric type, one of either counter or gauge, defaulting to gauge
- .setUnit(unit)
  - unit: <String> The metric unit. e.g: %, + (non-negative), or a unit like kb. Used by the UI for chart treatment
- .setWindow(window)
  - window: <Number> The window value
- .gauge()
  - Sets the metric type to gauge
- .counter()
  - Sets the metric type to counter
- .from(object)
  - object: <Object> Create a Metric from an object. The source object is expected to have the same property names as the Metric. Missing properties will be ignored (left as the default undefined).
    - data
    - metric
    - source
    - key
    - utc_offset
    - description
    - additional_data
    - tags
    - type
    - unit
    - window
    - time

Example

const Metric = require('@moogsoft/mars-sdk/Metric');

const metric = new Metric()
    .setMetric('duration')
    .setSource(`${jname}-${stage.name}`)
    .setData(stage.durationMillis)
    .setTime(stage.startTimeMillis)
    .setTags({ job: job.id })
    .setAdditionalData({
        status: stage.status,
        href: encodeURI(stage._links.self.href),
    });

Event

An "extended" class to generate a well formed Event to be used by the sendEvents() method

Returns: <Object>
Methods:
- .setSeverity(severity)
  - severity: <String> OR <Number> Can be one of the following
    enum String Notes
    0 CLEAR Clears a (mandatory) preceding event severity
    1 UNKNOWN The severity cannot be determined, usually indicates a problem, or unrecognized event
    2 WARNING
    3 MINOR
    4 MAJOR
    5 CRITICAL
- .setSource(source)
  - source: <String> the unique name, or "namespace" of the resource the metric is associated with
- .setCheck(check)
  - check <String> The check that failed to generate this event, for example "cpu load"
- .setDescription(description)
  - description <String> A description of the event
- .setTime(time)
  - time: <unixtime> The timestamp of the event in s, if omitted defaults to the current time
- .setUtcOffset(utc_offset)
  - utc_offset: <String> The UTC offset of the data, for example +01:00, defaults to the current TZ
- .setDedupeKey(dedupe_key)
  - dedupe_key: <String> Optional The deduplication key of this event, used when processing into an Incident
- .setManager(manager)
  - manager: <String> Optional The manager of this event
- .setService(service)
  - service: <String> Optional An array of services impacted by this event
- .setAlias(alias)
  - alias: <String> Optional An alias for this event
- .setClass(clazz)
  - clazz: <String> Optional The class of this event, for example Storage, AWS, Network, etc
- .setTags(tags)
  - tags: <Object> Optional Key-value pairs of metadata for the event
- .from(object)
  - object: <Object> Create an Event from an object. The source object is expected to have the same property names as the Event. Missing properties will be ignored (left as the default undefined).
    - severity
    - source
    - check
    - description
    - time
    - utc_offset
    - dedupe_key
    - manager
    - service
    - alias
    - class
    - tags

enum	String	Notes
`0`	`CLEAR`	Clears a (mandatory) preceding event severity
`1`	`UNKNOWN`	The severity cannot be determined, usually indicates a problem, or unrecognized event
`2`	`WARNING`
`3`	`MINOR`
`4`	`MAJOR`
`5`	`CRITICAL`

Bitmask

An "extended" class to generate a well formed Bitmask metric to be used by the sendMetrics() method

Returns: <Object>
Methods:
- .setKeys(keys)
  - keys: <Array> An array of strings, represnting the keys, or lables of the binary representation
- .setValues(values)
  - values: <Array> An array of boolean values (corresponding to the keys)
- .addValue(key, value) Appends a key-value pair to the value array

sendResult

sendResult(result)

Writes a JSON string to stdout returning a SINGLE datum, which is interpretted by the collector

result <Object> A well formed "Metric" object

sendMetrics

sendMetrics(metrics)

Writes a JSON string to stdout returning a batch of metrics, which is interpretted by the collector

metrics <Array> An array of well formed "Metric" objects

sendEvents

sendEvents(events)

Writes a JSON string to stdout returning a batch of events, which is interpretted by the collector

events <Array> An array of well formed "Event" objects

exec

exec(cmd, args)

Executes a shell process

cmd: <String> A command in the the collector execution path, or an absolute path
args: <Array> An array of arguments to the cmd
Returns: <Object>:
- status: <Number> shell return value
- stdout: <String> Text that was written to stdout
- stderr: <String> Text that was written to stderr

JSONToKv

JSONToKv(object, array, func)

Builds an array of key-value pairs from nested JSON/BSON, by "flattening" the nested structure, concatenating the keys with an underscore

object: <Object> The JSON object to be converted
array: <Array> The array to hold the resulting key:value pairs
func: <Function> A function to format the key:value pairs

Example:

function buildMetric(n, v) {
    const name = String(n).split(' ').join('_');
    let built = new Metric.setName(name).setValue(v);
    return built;
}

const mongoStat = [];
JSONToKv(info, mongoStat, buildMetric);

dehumanize

dehumanize(string)

Normalise (de-humanize) a value that may have been generatied in "human" readable form. * Takes an input like "10Mb", "5 Kb" or "12.3Kb" and returns a numeric value in bytes

Returns: <Number
string <String> The string to convert. e.g: 1 Kb will return 1024

jsonParse

jsonParse(string)

A wrappper for JSON.parse with error handling

Returns: <Object>
string: Input JSON as a string

isInFilters

isInFilters(item, filterList)

A filter utility that evaluates against lists of strings or regex

Returns: <Boolean>
item <String> The string we're looking for
filterList <Array> An array of regular expressions or stringto test Example

mongoStat.filter((m) => passFilter(m.name, filters)).forEach((m) => {
    filteredMongoStat.push(m);
});

Where the filter list is:

"filters": [
    "text",
    "/regex\\s/",
    "!/inverseRegex/"
]

passFilter

passFilter(item, flist)

A basic, but very quick exclusion filter

Returns: <Boolean>
item <String> The string we're looking for
flist <Array> An array of regular expressions to test, matching items will be filtered (dropped) Where the filter regex is of the following format:

"filters": [
    "'^tcmalloc'",
    "'foo'"
]

REST

head(url, params, httpOptions)
post(url, params, httpOptions)
get(url, params, httpOptions)

The REST module provides wrappers for the HEAD, POST and GET HTTP(S) requests, and returns the resulting content (if any), or a detailed error. Additionally timing is done at various stages, enabling the requests to be used as a basic synthetic test.

Returns: <Object>
url: <String> A URL, containing the protocol
params: <Object> OR <String> Query parameters, or post data. In most circumstances, the object will be appropriate, but a string can be accepted for instances where the query parameters have to be URI encoded
httpOptions: <Object> HTTP Options

Return Object:

Key	Type	Description
`exception`	`<String>`	Any non HTTP error
`status_code`	`<String>`	HTTP status code (if available)
`body`	`<String>`	HTTP response body
`roundTripTime`	`<Number>`	Round trip from request initiation to full body received
`responseTime`	`<Number>`	HTTP Header received
`errorTime`	`<Number>`	Time to detect / receive fatal error
`dnsTime`	`<Number>`	DNS lookup time from socket library
`connectTime`	`<Number>`	Connect time from socket library
`certValidTo`	`<String>`	Certificate expiry as a string
`certValidToMs`	`<Number>`	Certificate valid time remaining
`headers`	`<Object>`	HTTP headers

Example:

A simple synthetic test, using the Boolean detector for string validation, and Adaptive detector for Round Trip Time

const target = {
    "url":"http://example.com",
    "host": "example.com",
    "validString": "Example",
};

async function checkUrl(target) {
    const { url } = target;

    const request = async () => {
        const metrics = [];
        const result = await rest.get(url, {});
        let success = result.status_code === 200;

        //
        // If the request returned 200, move on to string validation
        //
        if (success) {
            const page = result.body.toString();
            if (page.match(target.validString) === null) {
                success = false;
                result.exception = `Validation string not found: '${target.validString}'`;
            }
        }

        const {
            roundTripTime,
            responseTime,
            errorTime,
            dnsTime,
            connectTime,
            certValidTo,
            certValidToMs,
        } = result;

        //
        // Build the RTT metric
        //
        const rttMetric = new Metric()
            .setMetric('roundTripTime')
            .setSource(target.host)
            .setData(roundTripTime)
            .setAdditionalData({
                responseTime,
                errorTime,
                dnsTime,
                connectTime,
                certValidTo,
                certValidToMs,
            });
        metrics.push(rttMetric);

        //
        // Build the status metric
        //
        const statusMetric = new Metric()
            .setMetric('urlcheck')
            .setSource(target.host)
            .setData(success)
            .setAdditionalData({
                exception: result.exception,
                status_code: result.status_code,
            });
        metrics.push(statusMetric);
        sendMetrics(metrics);
    };
    request();
}

hjson

@infinitebrahmanuniverse/nolb-_moo @everything-registry/sub-chunk-618

3 years ago

3 years ago

4 years ago

4 years ago