insights-core v0.0.40

Insights

As our education marketplace grew, we faced a dilemma: we had little visibility into our data.

Questions such as "who is our best customer?", "which target countries are gaining in sales?" or "what are our most popular categories for paid leads?" required tinkering in the Rails console or writing custom stats pages. We lost a lot of developer time over the years.

So we started looking for the best Business Intelligence solution out there.

Unfortunately, they all had their problems. Some required knowledge of SQL. Some made multi-table data exploration a pain. Most cost an arm, a leg and an iPhone. Per month.

The best tool we found was Looker, but at a monthly price equal to our burn rate, it wouldn't work.

So I decided to re-implement the essential parts of Looker as an open source alternative.

Insights is a desktop or self-hosted "SQL-not-required" data analytics and business intelligence tool. Featuring linkable URLs, easy data exploration, automatic joins, graphs, exports, facets (pivots), saveable views, pretty colors and a ridiculously permissive license (MIT).

It's a work in progress and you're brave for checking it out! Cheers!

Play with the demo here.

Installing

To install, make sure you have Node 7.6+ installed and then run:

# desktop (electron) version
npm install -g insights insights-core insights-desktop
insights desktop          # run the desktop/electron version

# server version
npm install -g insights insights-core insights-server
insights createsuperuser  # create an admin user to access the server
insights createsecret     # create a new authentication secret at ~/.insights/secret
insights server           # run the server

Read the deployment guide if you wish to run this on a fresh Ubuntu 16.04 instance.

How does it work

Similar to Looker and their LookML, Insights requires a mapping of your database's structure to function.

This mapping is either autogenerated from your schema and foreign keys, or created with scripts such as the insights_export gem, which exports Rails (ActiveRecord) associations.

Once you have a mapping, you may then edit it to add custom columns (e.g. full_name: first_name || ' ' || last_name), hide existing fields (e.g. encrypted_password) or hide entire models.

(Note: as of this moment, it's only possible to edit the mapping if you generate the .yml file manually. The editor for the autogenerated structures is coming soon!)

How to use

Play with the demo here and try to answer the following questions:

1. Which product has been bought the most? (solution)
2. Sales by country by month (solution) - export it as a PDF as well!
3. Sales by delivery status (solution)
4. Where are your users from (solution)
5. ... by month? (solution)
6. ... only ones with confirmed orders? (solution)

Hint: to count rows, select the id field and then count from the table header.

Coming soon

• Structure editor in the interface
• Manage users in the interface
• PDF and XLSX exports (they worked on the old Rails version, but have to be ported to NodeJS)
• Better graph controls
• Graphs that don't require a time column
• View generated SQL
• Dashboard item grouping/tabbing
• Multiple lines from different sources on one chart in the dashboard
• Polishing

Development

insights is built as a monorepo and uses oao for package management, at least until yarn workspaces become a thing. oao uses yarn internally, so you need to have it installed as well.

If you want to help with development, run these steps:

# 1. fork the repo in github

# 2. clone it
git clone git@github.com:<YOUR_NAME_HERE>/insights.git

# 3. install all dependencies
cd insights
yarn

# 4. run the development environment
yarn start

# 5. open http://localhost:3030/ and hack away

How to publish new versions

Here's the latest workflow to publish a new version. This may change in the future.

# install oao
npm install -g oao

# optional: check if the core builds as needed.
yarn run build:core

# update the versions of the repos under packages/* and the root repo
oao all "npm version patch"
git add .
git commit -m 'update package versions'
npm version patch

# sync package.jsons
oao prepublish

# publish
oao all "npm publish"

# NB! don't run 'npm publish' on its own: there is no need to publish the insights-build package.

At this beta moment you need to do:

cd packages/insights-charts
yarn link
cd ../../packages/insights-core
yarn link insights-charts