storinfo v1.0.1

node-storinfo

This repository is part of the Joyent Manta project. For contribution guidelines, issues, and general documentation, visit the main Manta project page.

node-storinfo is a NodeJS client for the Manta Storinfo service.

NOTE This module is load-bearing for both the muskie (webapi) and buckets-api MantaV2 services. Therefore, if you make changes to this node package, you should sanity check the functionality of both services before integration.

Creating a Storinfo Client

node-storinfo supports initializing either a standalone or full StorinfoClient object. A standalone client does not establish a network connection to the Storinfo service. It is primarily used for testing the object placement algorithm implemented by the choose method without requiring access to an actual Manta deployment.

A standalone client can be created as follows:

const mod_storinfo = require('storinfo');
           
var client = mod_storinfo.createClient({standalone: true});

A full client (one that actually connects to the Storinfo service in a Manta deployment) requires a number of additional properties.

For example:

var opts = {
    standalone: false,
    url: 'http://storinfo.domain'
    cueballOpts: {
        spares: 4,
        maximum: 10,
        recovery: {
            default: {
                timeout: 2000,
                retries: 5,
                delay: 250,
                maxDelay: 1000
            }
        },
        resolvers: ["nameservice.domain"]
    }
};

var client = mod_storinfo.createClient(opts);

Additionally, the following optional parameters can be specified when creating a StorinfoClient instance:

StorinfoClient methods

getStorageNodes

The getStorageNodes method returns the Storinfo services cached view of the entire manta_storage bucket, as an array of objects sorted by manta_storage_id. This method will return an error if invoked on a standalone client.

This asynchronous method takes a the following argument:

getStorageNode

The getStorageNode method returns the Storinfo services cached view of a single row from the manta_storage bucket, corresponding to the storage node with the specified manta_storage_id. This method will return an error if invoked on a standalone client.

This asynchronous method takes a the following arguments:

choose

The choose method takes a desired number of replicas and a size (in bytes), and then selects three random "tuples" (the number of items in a tuple is #replicas). The first random tuple is "primary," and then we have 2 backup tuples.

Conceptually it looks like this:

{
    us-east-1: [a, b, c, ...],
    us-east-2: [d, e, f, ...],
    us-east-3: [g, h, i, ...],
    ...
}

Where the objects a...N are the full JSON representation of a single storage node.

The choose method can be invoked for both standalone and for StorinfoClient objects. For non-standalone clients, this method requires that the StorinfoClient has successfully performed at least on poll from the Storinfo service. This can be assured by either manually calling the getStorageNodes method or by specifying the pollInterval property during client creation and then waiting for a 'topology' event.

This method takes the argument object and a callback. The properties of the argument object are described below:

mchoose CLI

This module includes a CLI bin/mchoose which provides a scriptable interface for invoking the getStorageNodes and choose methods. The usage is described below:

Models the behavior of the Manta's object placement logic.

Usage:
    mchoose [OPTIONS] COMMAND [ARGS...]
    mchoose help COMMAND

Options:
    -h, --help      Show this help message and exit.
    --version       Print version and exit.

Commands:
    help (?)        Help on a specific sub-command.
    poll            Poll the Storinfo service for storage records and summarizes the results.
    choose          Simulate Manta storage node selection for an object.

Testing

The automated tests do not require access to a Manta deployment. They can be run as follows:

% make test