geo-graph v0.11.0

GeoGraph

A layer built on top of neo4j and postgres to store, relate and query json and geojson

This project is still on beta and any api methods are susceptible to changes

Pre-requisites

• You must have Neo4j installed
• If you want to work with spatial data you must install PostgreSQL too
• Neo4j must be running and the PostgreSQL database must have Postgis extension

Installation

• Lib: npm install geo-graph --save
• Cli: npm install -g geo-graph --save

Tests

npm test

Installing pre-requisites

Via Docker

Use this Docker Compose and run:

docker-compose up -d

And everything will be running.

Important: do not share the ports with your host in production environments

Manual

Neo4j

Download and install neo4j for your platform - Community version is fine

Postgres

If you want to use spatial queries, you need to download and install postgres and postgis

• It's important that you use version 9.5+ of postgres, any previous version won't work
• There are several versions of postgis for each postgres version, be sure to install the compatible one
  • For instance, if you are using ubuntu and installed postgres-9.5, you must install postgis using apt-get install postgres-9.5-postgis.2.3

Configuring the environment

run geo-graph init, this command have the following parameters:

• pg_host: the postgres database host, defaults tolocalhost
• pg_user: the postgres database user, defaults to postgres
• pg_password: the postgres database password, defaults to postgres
• pg_database: the postgres database name, defaults to geograph_geometries
• pg_create_database: if true, will create the database if it does't exist, defaults to false
• neo4j_host: the neo4j database host, defaults to localhost
• neo4j_auth: wether or not you use neo4j auth, defaults to false
• neo4j_user: the neo4j database user, mandatory only if you are using neo4j auth
• neo4j_password: the neo4j database password, mandatory only if you are using neo4j auth

Example: geo-graph init --pg_create_database - will connect to postgres using host: localhost, user: postgres, password: postgres, create the geograph_geometries database if it doesn't exist yet, then create the proper table and indexes. Also, will access neo4j using host: localhost and no auth, then create the indexes proper indexes.

That's it, you're now ready to go

Usage

First, you need a geograph instance

const
    Joi = require('joi'),
    Geograph = require('geo-graph'),
    geograph = new Geograph({
        neo4j: {
            host: 'localhost',
            auth: {
                user: 'neo4j',
                password: 'neo4j'
            }
        },
        validations: {
            'Person': {
                name: Joi.string().required()
            },
            'Music': {
                name: Joi.string().required()
                year: Joi.number().optional()
            }
        }
        pg: {
            host: 'localhost',
            user: 'postgres',
            password: 'postgres',
            database: 'yourdb'
        }
    });

The constructor takes one configuration object as parameter that have the following properties:

• neo4j: mandatory object containing the parameters to connect the neo4j database
• neo4j.host - string, the neo4j bolt address, defaults to localhost
• neo4j.auth - object containg the auth data to connect your neo4j database, mandatory only if neo4j auth is enabled
• neo4j.auth.user - the neo4j database user
• neo4j.auth.password - the neo4j database password
• validations - optional object where each key is the name of the label to validate and the value is a valid joi schema object
• pg: optional knex object/string connection to the postgres database

save(jsons, callback)

Arguments

• jsons: the json or array of jsons to be inserted or updated
  • Geograph will deep search for objects inside the jsons and for every object found, it will perform the following operations:
    • If the object doesn't have a _label property with a valid value, the callback function will be called with an error (a valid label is any case sensitive string starting with a letter that can contain underscores and alphanumeric characters).
    • If the object doesn't contain a uuid property, it will be inserted at the database, otherwise it will update its values.
    • If the object is a geojson feature, then it will be inserted/updated based on it's owner uuid absence/presence
• callback(err, uuid): optional function to be called after all jsons are inserted/updated
  • if you insert a single json, uuid will be a string containing the uuid of the root object
  • if you insert an array of jsons, uuid will be an array of strings where each string is the root uuid of the jsons that you just inserted

Example

geograph.save({
    uuid: '23ceb866-c564-462a-8784-8c0cfc8dbc0a'
    _label: 'Person',
    name: 'Diego de Oliveira',
    age: 24,  
    interests: [{
        _label: 'Movie',
        name: 'Interestellar'
    }],
    nextEvent: {
        uuid: '23ceb866-c564-462a-8784-8c0cfc8dbc0d'
        _label: 'Event'
        city: null,
        name: 'The International',
        location: {
            "type": "Feature",
            "properties": {},
            "geometry": {
                "type": "Point",
                "coordinates": [
                    -122.32246398925981,
                    47.60431120244568
            ]
          }
        }
    }
}, (err, uuid) => {
    if (err) {
        console.log('An error has ocurred', err);
    } else {
        console.log(uuid); // will print 23ceb866-c564-462a-8784-8c0cfc8dbc0a
    }
})

The code above does the following:

• Modify the property name of the object with uuid '23ceb866-c564-462a-8784-8c0cfc8dbc0a' from Diego to Diego de Oliveira
• Add the property age to the object uuid '23ceb866-c564-462a-8784-8c0cfc8dbc0a'
• Add the object {name: 'Interestellar} with label Movie
• Add the Movie as another interest to the Person
• Remove the property city from the obejct with uuid '23ceb866-c564-462a-8784-8c0cfc8dbc0d'
• Change the coordinates of the location of the Event

findById(uuid, callback)

Arguments

• uuid: the uuid of the json that you want to retrieve, it will include all relationships, deep searching them at the database, so, beware when trying to retrieve an object that is associated with too many others. If you want more control, you should use find
• callback(err, json): callback function to be executed after finding the json

Example

geograph.findById('23ceb866-c564-462a-8784-8c0cfc8dbc0a', (err, json) => {
    if (err) {
        console.log('An error has ocurred', err);
    } else {
        console.log(json);
    }
})

find(queryObject, callback)

Arguments

• queryObject: optional object containg the filters to be applied on the search, if ommited it will retrieve every object of the database
  • queryObject.label: string or array of strings, the labels that you want to retrieve
  • queryObject.where: string containing one or multiple property filters separated by boolean operators
    • a property filter have the following pattern: <property_name> <comparison operator> <value>
    • comparison and boolean operators are any supported by neo4j
    • some examples of where are:
      • age > 20: filters all objects that have a key age and the value of that key is a number bigger than 20
      • name starts with "Diego" and age > 20: filters all objects that have a key age wich the value is a number bigger than 20 and a key name wich is a string starting with "Diego"
      • You may also use parenthesis to alter the order in wich the filters are applied
  • queryObject.skip: number of objects to skip after applying the filters
  • queryObject.limit: maximum number of objects to retrieve after applying the filters
  • queryObject.identifier: custom identifier to give to the objects returned by the corresponding filter, useful when you want to compare values between objects inside the database.
  • queryObject.include: array of include objects, where each include represents a relationship to fetch and attach to its owners

    • Each include object can have all the keys of the queryObject, along with:

      • name: mandatory string or array of strings, wich specifies the names of the relationships that you want to fetch
      • mandatory: boolean, tells Geograph to only retrieve the nodes that have the relationship specified on name
• callback(err, json[]): callback function to be executed after finding the objects that match the filters.

Example

geograph.find({
    label: 'Person',
    filter: {
        where: 'name starts with "Diego"',
        identifier: 'p'
    },
    include: [{
        name: 'friends',
        where: 'name = p.name',
        limit: 10,
        include: [{
            name: 'interests',
            label: ['Person', 'Music']
        }]
    }]
}, (err, jsons) => {
    if (err) {
        console.log('An error has ocurred', err);
    } else {
        console.log(jsons);
    }
})

The code above retrieves all objects that have the label Person and name starting with "Diego", including their friends the same name and also the interests of those friends on other Persons or Musics.

findBySpatialQuery(spatialQueryObject, callback)

Arguments

• spatialQueryObject: object containg the filters to be applied on the search, can have all properties of the find query object, plus:
  • spatialQueryObject.geometries: array of strings, where each string is a geometry to look for, some examples are:
    • 'Event.location': fetches all geometries that are located at the location key of objects with Event label.
    • ['Person.address', 'Route.path']: fetches all geometries that are located at the address key of objects with Person label and all geometries that are located at the path key of objects with Route label.
  • spatialQueryObject.filter: a postgres where to be executed and filter the geometries returned by the geometries parammeter.
    • here you have access to a special variable geometry that is the current geometry to be tested
    • any Geograph instance has a st property that is a knex-postgis instance to help you build this string, if you want to.
• callback(err, json[]): callback function to be executed after finding the objects that match the spatial filters.

Example

let rioDeJaneiro = {
        'type': 'Point',
        'coordinates': [
            -43.32115173339844,
             -22.811630707692412
        ]
    };

geograph.findBySpatialQuery({
        nodes: ['Event.location'],
        filter: `ST_Distance(geometry::geography, ST_GeomFromGeoJSON(\'${JSON.stringify(rioDeJaneiro)}\')::geography) >= 100`
    }, (err, results) =>  {
    if (err) {
        console.log('An error has ocurred', err);
    } else {
        console.log(results);
    }
});

The code above returns all objects with the Event label and have its location in a distance of 100 meters or less to the location specified at rioDeJaneiro

deleteNodesById(uuids, callback)

Arguments

• uuids: string or array of strings, where each string is the uuid of the object that you want to delete
• callback(err): callback function to be executed after finding the json

Example

geograph.deleteNodesById('23ceb866-c564-462a-8784-8c0cfc8dbc0a', (err) => {
    if (err) {
        console.log('An error has ocurred', err);
    }
})

deleteNodesByQueryObject(queryObject, callback)

Arguments

• queryObject: optional object having the same fields as the query object used in the find method, if ommited it will delete every object inside the database.
• callback(err): callback function to be executed after deleting the objects

Example

geograph.deleteNodesByQueryObject({
    label: 'Person',
    filter: {
        where: 'name starts with "Diego"',
        identifier: 'p'
    },
    include: [{
        name: 'friends',
        where: 'name = p.name',
        limit: 10,
        include: [{
            name: 'interests',
            label: ['Person', 'Music']
        }]
    }]
}, (err) => {
    if (err) {
        console.log('An error has ocurred', err);
    }
})

The code above will delete all objects that have the label Person and name starting with "Diego", including their friends the same name and also the interests of those friends on other Persons or Musics.

deleteRelationships(relationshipQueryObject, callback)

Arguments

• relationshipQueryObject: object or array of objects, where each object is like a regular json that you would use on update, except that Geograph will ignore any property that is not uuid and if you dot not provide any relationship, an error is thrown
• callback(err): callback function to be executed after finding the json

Example

geograph.deleteRelationships({
    uuid: '23ceb866-c564-462a-8784-8c0cfc8dbc0a', // uuid of the source
    friends: { //name of the relationship to remove
        uuid: '23ceb866-c564-462a-8784-8c0cfc8dbc0b', // uuid of the target
        interests: { // you may also specify more source-relationship-targets
            uuid: '23ceb866-c564-462a-8784-8c0cfc8dbc0b'
        }
    }
}, (err) => console.log(err));

The code above deletes two relationships: '23ceb866-c564-462a-8784-8c0cfc8dbc0a' friends '23ceb866-c564-462a-8784-8c0cfc8dbc0b' and '23ceb866-c564-462a-8784-8c0cfc8dbc0b' interests '23ceb866-c564-462a-8784-8c0cfc8dbc0b'.

That means that the objects with uuid '23ceb866-c564-462a-8784-8c0cfc8dbc0a' and '23ceb866-c564-462a-8784-8c0cfc8dbc0b' are no longer friends and that object with uuid '23ceb866-c564-462a-8784-8c0cfc8dbc0b' no longer have interest on object with uuid '23ceb866-c564-462a-8784-8c0cfc8dbc0b'

Examples

Take a look at the example test files

FAQ

• What are the benefits of using Geograph?

  • Store and relate your jsons as they are without needing to worry with ORM configurations, migrations or schema restrictions
  • Fetch your data using a simple query object and let Geograph build complex cypher and sql for you

• How Geograph stores my data? Every time call save to insert data, Geograph will deep search for json objects inside your json and perform the following steps:

  • if the current object is a json, then store it in neo4j
  • for every key of the current object that holds a geojson value, then convert it to a wkt and save it as a geometry in postgres
  • For every key of the current object (let's call it owner) that holds a object value (let's call it owned), insert the owned object in neo4j, then relate it to its owner.
  • Repeat the 2 steps above to every owned object found.