@cryptexlabs/neural-data-normalizer v1.0.0

Neural data normalizer

This is library convert datasets of human data into arrays of bits understandable for neurons (duh).

Example

Suppose we want a network to know when or when not to water my plants on its own (whatever the units are for now) with this data set.

[
{ "soilhumidity": 500, "airtemp": 32, "airhum": 18, "water": true, "plants": ["tomatoes", "potatoes"] },
{ "soilhumidity": 1050, "airtemp": 40, "airhum": 21, "water": true, "plants": ["potatoes", "asparagus"] },
{ "soilhumidity": 300, "airtemp": 100, "airhum": 90, "water": false, "plants": ["asparagus", "tomatoes"] },
{ "soilhumidity": 950, "airtemp": 103, "airhum": 26, "water": true, "plants": ["asparagus", "asparagus"] },
{ "soilhumidity": 1050, "airtemp": 8, "airhum": 26, "water": true, "plants": ["tomatoes", "tomatoes"] },
{ "soilhumidity": 1050, "airtemp": 56, "airhum": 26, "water": true, "plants": ["potatoes", "french fries"] }
]

In the end, our output is "should we water the plants?": water: true and the rest are our inputs. Let's do this.

const normalizer = new Normalizer(sampleData);

// setting required options and normalize the data
normalizer.setOutputProperties(['water']);
normalizer.normalize();

// find useful information about your data
// to pass to your neural network
const nbrInputs = normalizer.getInputLength();
const nbrOutputs = normalizer.getOutputLength();

const metadata = normalizer.getDatasetMetaData();
const inputs = normalizer.getBinaryInputDataset();
const outputs = normalizer.getBinaryOutputDataset();

console.log(metadata);
console.log(inputs);
console.log(outputs);

There you should have all useful information to give to your network. You know the number if inputs and outputs, you get ~~binarized(?) dataset suitable for neural networks, and event some metadata about your data.

Output:

{ soilhum: { type: 'number', min: 300, max: 1050, distinctValues: null },
  airtemp: { type: 'number', min: 8, max: 103, distinctValues: null },
  airhum: { type: 'number', min: 18, max: 90, distinctValues: null },
  water: { type: 'boolean', min: 0, max: 1, distinctValues: null },
  plants:
   { type: 'array',
     min: null,
     max: null,
     distinctValues: [ 'tomatoes', 'potatoes', 'asparagus', 'french fries' ] } }

[ [ 0.266667, 0.252632, 0, 1, 1, 0, 0 ],
  [ 1, 0.336842, 0.041667, 0, 1, 1, 0 ],
  [ 0, 0.968421, 1, 1, 0, 1, 0 ],
  [ 0.866667, 1, 0.111111, 0, 0, 1, 0 ],
  [ 1, 0, 0.111111, 1, 0, 0, 0 ],
  [ 1, 0.505263, 0.111111, 0, 1, 0, 1 ] ]

[ [ 1 ], [ 1 ], [ 0 ], [ 1 ], [ 1 ], [ 1 ] ]

Why metadata ?

Consider a real example where you actually started to understand what are neural networks and start implementing it. You realize the biggest challenge is data formatting. When you activate Alfred with you data (we always call our network Alfred) you realize you also need to normalize the new data input as well.

So you need to save metadata information that you got earlier (mins, maxes, ets) so that our data normalizer here converts the new inputs to the same scales! (this implies training data MUST contain the min and maxes values at some point).

Then on new unknown input you just have to recall the normalizer one thing: metadata of known values range.

const normalizer = new Normalizer(newData);

normalizer
    .setDatasetMetaData(networkObject.metadata)
    .setOutputProperties(['water']);

const input = normalizer.getBinaryInputDataset()[0];