Mongolfier NPM

DEPRECATION NOTICE

This project is abandoned.

mongolfier

A Simple MySQL to MongoDB Migration Tool

Montgolfier Balloon

NOTE: not "montgolfier" but "mongolfier". ;)

install and run

local install and run

$ npm install mongolfier
$ node ./node_modules/.bin/mongolfier --help

global install and run

$ npm install -g mongolfier
$ mongolfier --help

simple migration with bulk-copy

To copy all rows from users table on test mysql database into users collection on test mongo db.

$ mongolfier -s mysql://root@localhost/test -o mongodb://localhost/test -b users

NOTE use -d flag to test WITHOUT real insert operations:

$ mongolfier -s mysql://root@localhost/test -o mongodb://localhost/test -b -d users

NOTE use -e flag to remove all existing documents in the collection:

$ mongolfier -s mysql://root@localhost/test -o mongodb://localhost/test -b -e users

complex migration with template(EXPERIMENTAL)

For complex migration, you need to write a config file and template files and use -c option:

$ mongolfier -c config.json

usage

$ mongolfier -help
  Usage: mongolfier [OPTIONS] [COLLECTION_NAME ...]

  Options:

    -h, --help              output usage information
    -V, --version           output the version number
    -c, --config [FILE]     path to config file(js or json)
    -s, --mysql [URL]       mysql connection url
    -o, --mongo [URL]       mongodb connection url
    -e, --empty-collection  make empty collection before migration
    -b, --bulk-copy         do bulk copy if no template available
    -d, --dry-run           do not insert into collection

configuration

config file(json)

NOTE: no comment allowed here. this file should conforms to strict json format.
NOTE: {{ and }} is just a marker. replace it with yours.

{
  "mysql": "mysql://user:password@host:port/database",
  "mongo": "mongodb://user:password@host:port/db",
  "context": {
    "{{custom_attr_key}}": "{{custom_attr_value}}",
    ...
  },
  "before": [
    "{{before_script.js}}",
    ...
  ],
  "after": [
    "{{after_script.js}}",
    ...
  ]
  "collections": [
    "collection{
      "collection": "{{mongo_collection_name}}",
      "template": "{{path_to_collection_template}}",
      "query": "{{mysql_select_query}}"
    },
    {
      "collection": "{{mongo_collection_name}}",
      "template": "{{path_to_collection_template}}",
      "query": [
        "{{mysql_select_query_part1}}",
        "{{mysql_select_query_part2}}",
        ...
      ]
    },
    ...
  ]
}

context is optional. this could be accessed via $ variable across processing all collections.
before is optional. these scripts are evalulated before processing the first collection.
after is optional. these scripts are evalulated after processing the last collection.
template is optional(default: {{collection name}}.json).
query is string or array. array will be joined to a query.

mapping template/script

NOTE: comment allowed here. this file is javascript or something. ;)
NOTE: {{ and }} is just a marker. replace it with yours.

simple mapping template

({
  // use mongodb ObjectID
  "_id": new mongo.ObjectID(),
  // use mysql column values
  "{{mongo_field_name}}": ${{mysql_column_name}},
  ...
})

complex mapping template

var id = new mongo.ObjectID();
...
({
  "_id": id,
  "foo": ($bar).toUpperCase(),
  "sum": $num1 + $num2,
  "now": new Date(),
  ...
})

NOTE on the enclosing braces the result. do not use return keyword.

async mapping template(EXPERIMENTAL)

var d = q.defer();
var id = new mongo.ObjectID();
setTimeout(function () {
  ...
  d.resolve({
    "_id": id,
    "foo": ($bar).toUpperCase(),
    "sum": $num1 + $num2,
    "now": new Date(),
    ...
  })
}, 100);
d.promise;