rethinkdb-regrid v0.8.1

ReGrid

ReGrid is a method of storing large files inside a RethinkDB database.

Features

• Reliable - Files are replicated across the cluster, benefiting from RethinkDB's automatic failover.
• Scalable - Easily store large files in RethinkDB, distributed across the cluster.
• Consistent - Sha256 hashes are calculated when the file is written, and verified when read back out.
• Realtime - Watch the file system for changes and be notified immediately.
• Fast - Supports in-memory caching so your server doesn't fall over when something goes viral.
• Current - Uses RethinkDB's changefeeds to evict cached items. ReGrid doesn't serve stale files.

View the Changelog

The ReGrid spec is an open specification free for anyone to implement and use.

The spec is very out of date

Need help?

Join Slack here, then meet us on the #regrid channel.

Special thanks to Arthur Andrew Medical for sponsoring this project.

Arthur Andrew Medical manufactures products with ingredients that have extensive clinical research for safety and efficacy. We specialize in Enzymes, Probiotics and Antioxidants.

Installation

Supports node v8.0+

npm install --save rethinkdb-regrid

TL;DR

var ReGrid = require('rethinkdb-regrid')

var bucket = ReGrid({db: 'example'})

// initBucket creates tables and indexes if they don't exist, returns a promise.
bucket.initBucket().then(function () {
  // We are now ready to read and write files

  // Watch a filename for changes
  bucket.watchFilename('/videos/myVid.mp4').then(function (cursor) {
    cursor.each((err, changes) => {
      console.log(changes)
    })
  })

  // Open a write stream to ReGrid
  var dbStream = bucket.upload('/videos/bigvid.mp4')

  // Create read stream from file and pipe it to a ReGrid write stream
  fs.createReadStream('./bigvid.mp4').pipe(dbStream)

  dbStream.on('finish', function () {
    // File is now written to the database

    //Read the file and pipe it to a write stream to save the file back out to the file system.
    bucket.downloadFilename('/videos/bigvid.mp4').pipe(fs.createWriteStream('./copy-of-bigvid.mp4'))
  })

})

API Documentation

ReGrid([connectionOptions, options])

Parameters

Options

returns

Bucket instance

Description

Creates a new ReGrid bucket instance.

Example

var ReGrid = require('rethinkdb-regrid')

var bucket = ReGrid({db: 'mydatabase'}, {bucketName: 'mybucket'})

initBucket()

Parameters

none

returns

Promise

Description

Verifies required tables and indexes exist and will create them if missing.

Example

bucket.initBucket().then(function () {
  // bucket is ready for use.....
})

writeFile(options)

Options

returns

Promise

Description

Returns a promise that resolves to the newly written file.

Example

let fileBuffer = fs.readFileSync('./myVid.mp4')

let newFile = await bucket.writeFile({filename: '/videos/myVid.mp4', buffer: fileBuffer})

createWriteStream(options)

Options

returns

WriteStream

Description

Returns a write stream for storing a file in ReGrid.

Example

var writeStream = bucket.createWriteStream({
  filename: '/videos/myVid.mp4',
  chunkSizeBytes: 1024 * 255,
  metadata: {topic: 'cats'}
})

writeStream.on('finish', function () {
  // File is now stored in ReGrid
})

fs.createReadStream('./myVid.mp4').pipe(writeStream)

getFile(options)

Options

How revision numbers work

If there are five versions of a file, the below chart would be the revision numbers

Description

Returns a promise that resolves to the files information.

Example

let file1 = bucket.getFile({id: 'ca608825-15c0-44b5-9bef-3ccabf061bab'})
let file2 = bucket.getFile({filename: 'catVideo.mp4', revision: 2})

readFile(options)

Options

How revision numbers work

If there are five versions of a file, the below chart would be the revision numbers

Description

Returns a promise that resolves to the files information and contents.

Example

let file1 = bucket.readFile({id: 'ca608825-15c0-44b5-9bef-3ccabf061bab'})
let file2 = bucket.readFile({filename: 'catVideo.mp4', revision: 2})

createReadStream(options)

Options

returns

ReadStream

Description

Returns a read stream for reading a file from ReGrid.

Example

var readStream = bucket.createReadStream({id: 'ca608825-15c0-44b5-9bef-3ccabf061bab'})

readStream.pipe(fs.createWriteStream('./mySavedVideo.mp4'))

downloadFilename(filename[, options])

Parameters

Options

How revision numbers work

If there are five versions of a file, the below chart would be the revision numbers

returns

ReadStream

Description

Returns a read stream for reading a file from ReGrid.

Example

var newestVersion = bucket.downloadFilename('/videos/myVid.mp4')

var originalVersion = bucket.downloadFilename('/videos/myVid.mp4', {revision: 0})

newestVersion.pipe(fs.createWriteStream('./latest.mp4'))

originalVersion.pipe(fs.createWriteStream('./original.mp4'))

getFilename(filename[, options])

Parameters

Options

How revision numbers work

If there are five versions of a file, the below chart would be the revision numbers

returns

Promise that resolves to the files information.

Description

Returns information for a file from ReGrid.

Example

bucket.getFilename('/videos/myVid.mp4', {revision: 0}).then(function (file) {
  console.log(file)

  /*
  RETURNS
  {
    "chunkSizeBytes": 261120 ,
    "filename":  "/videos/myVid.mp4" ,
    "finishedAt": Wed Jan 11 2017 20:35:20 GMT+00:00 ,
    "id":  "2578d295-6041-40e6-9235-d02e96c524ca" ,
    "length": 174110 ,
    "sha256":  "298fac8be0bdb1e48fa520dee5f510ef5f91ea0411b7584e8c62ec0cf34ce932" ,
    "startedAt": Wed Jan 11 2017 20:35:20 GMT+00:00 ,
    "status":  "Complete"
  }
  */
})

listRegex(pattern[, options])

Parameters

Options

returns

ReadStream in objectMode. The stream emits objects, not buffers.

Description

Returns a read stream for finding files via regular expression. The stream emits objects as they are found, and can be a long running operation.

You may call toArray() on the returned stream to coerce it to an array. This may fail if the result set is very large. You may optionally call with limit to prevent this.

Example

var videoStream = bucket.listRegex('^/videos/', {limit: 100})

videoStream.on('data', function (video) {
  console.log(video)
})

// OR
bucket.listRegex('^/videos/', {limit: 100}).toArray().then(function (videos) {
  // list the first 100 videos in the `/videos` directory
  console.log(videos)
})

listFilename(filename[, options])

Parameters

Options

returns

ReadStream in objectMode. The stream emits objects, not buffers.

Description

Returns a read stream for finding files by filename. The stream emits objects as they are found, and can be a long running operation. However, this method uses an index and is very efficient.

You may call toArray() on the returned stream to coerce it to an array. This may fail if the result set is very large. You may optionally call with limit to prevent this.

Example

var revisionStream = bucket.listFilename('/videos/editedVideo.mp4')

revisionStream.on('data', function (revision) {
  console.log(revision)
})

// OR
bucket.listFilename('/videos/editedVideo.mp4').toArray().then(function (revisions) {
  // list all revisions of '/videos/editedVideo.mp4'
  console.log(revisions)
})

listMetadata(metadata[, options])

Parameters

Options

returns

ReadStream in objectMode. The stream emits objects, not buffers.

Description

Returns a read stream for finding files by metadata. The stream emits objects as they are found, and can be a long running operation.

You may call toArray() on the returned stream to coerce it to an array. This may fail if the result set is very large. You may optionally call with limit to prevent this.

Example

var catStream = bucket.listMetadata({topic: 'cats'})

catStream.on('data', function (catVideo) {
  console.log(catVideo)
})

// OR
bucket.listMetadata({topic: 'cats'}).toArray().then(function (catVideos) {
  // list all videos under the 'cat' topic.
  console.log(catVideos)
})

watchRegex(pattern)

Parameters

returns

A RethinkDB changeFeed cursor

Description

Allows you to be notified of changes to a file if the filename property matches your regular expression.

Example

bucket.watchRegex('^/videos').then(function (cursor) {
  cursor.each((err, video) => {
    console.log(video)
  })
})

watchFilename(filename)

Parameters

returns

A RethinkDB changeFeed cursor

Description

Allows you to be notified of changes to a file if the filename property === your string.

Example

bucket.watchFilename('/videos/myVid.mp4').then(function (cursor) {
  cursor.each((err, changes) => {
    console.log(changes)
  })
})

watchMetadata(metadata)

Parameters

returns

A RethinkDB changeFeed cursor

Description

Allows you to be notified of changes to a file if the metadata property matches your object.

Example

bucket.watchMetadata({topic: 'cats'}).then(function (cursor) {
  cursor.each((err, catVideoChanges) => {
    console.log(catVideoChanges)
  })
})

deleteId(fileId)

Parameters

returns

Returns a promise that resolves to a boolean, depending on whether the operation was successful.

Description

Marks a file as deleted in ReGrid

Example

bucket.deleteId('ca608825-15c0-44b5-9bef-3ccabf061bab')

renameId(fileId, filename)

Parameters

returns

Returns a promise that resolves to a boolean, depending on whether the operation was successful.

Description

Renames a file in ReGrid

Example

bucket.renameId('ca608825-15c0-44b5-9bef-3ccabf061bab', 'newName.mp4')

getMetadata(fileId)

Parameters

returns

Returns a promise that resolves to the file metadata.

Description

Retrieves the metadata for a given fileId.

Example

bucket.getMetadata('ca608825-15c0-44b5-9bef-3ccabf061bab')

replaceMetadata(fileId, metadata)

Parameters

returns

Returns a promise that resolves to a boolean, depending on whether the operation was successful.

Description

Replaces a file's metadata in ReGrid

Example

bucket.replaceMetadata('ca608825-15c0-44b5-9bef-3ccabf061bab', {topic: 'other cat stuff'})