green-s3 v1.0.4

green-s3

An S3 local agent for AWS Greengrass.

Current version: 1.0.4

Lead Maintainer: Halim Qarroum

Table of contents

• Installation
• Features
• Description
• API
• See also

Install

green add npm://green-s3

Features

• Synchronization of data located on the local filesystem between the edge with AWS S3 on the cloud.
• Able to upload data payload sent through the Expressify API directly to S3.
• Works offline, and syncs the data to S3 when Internet connectivity is available.

Description

The green-s3 application requires a Greengrass local volume to be associated with the application and which is mounted on a given directory on the host local file system on the /storage mountpoint. Once the mountpoint is associated with the application, green-s3 will run a synchronization loop which can be executed at a regular time interval (typically 1-2 times per day).

The synchronization loop will execute different archiving operations before uploading local assets to a configurable S3 bucket on AWS :

1. It will list the assets available on the /storage mountpoint.
2. It will then archive in the zip format any directory available on this mountpoint.
3. Directories and files (which will not be archived) will be uploaded to AWS S3 using an upload pool (currently, a maximum of 10 uploads will run in parrallel at any given point).

4. The upload being fail-safe, meaning that a missed upload will not cause the failure of the other uploads, green-s3 will remove from the local filesystem the files and directories which have been successfully synchronized. Other assets will be kept in the mountpoint and scheduled for upload on the next occurent of the synchronization loop.

Bucket layout

When uploading the local assets to the remote S3 bucket, green-s3 will create a structured hirerarchy in the bucket which will be as follow :

• The first level of containers at the root of the S3 bucket will be associated with the thing name of the Greengrass core which performs a synchronization. This is to allow the segregation of uploads triggered by multiple Greengrass instances.
• The second level of will be constituted of containers associated with the date of the upload (in the ISO8601 format). This allows the synchronization of files with the same name but with a different content to be uploaded multiple time during different synchronization loops without overwriting them in the bucket.

Overall, the content of the bucket after different synchronizations from multiple Greengrass instances will look as follow :

<thing-name-1>
  /2018-06-13T07:47:15+00:00
    /foo.txt
  /2018-06-13T15:31:31+00:00
    /bar.mpg
<thing-name-2>
  /2018-06-13T08:12:11+00:00
    /baz.mp3
  /2018-06-13T17:14:25+00:00
    /qux.raw

API

This application exposes an Expressify API which is accessible from any green application locally, but also from the AWS IoT Core service from the cloud remotely. The available API endpoints are documented and described below.

See also

• The green-cli command-line deployment tool.

• The green-connectivity agent.