S3-ingestor NPM

S3-INGESTOR

This is a simple utility in NodeJS for pushing files from a local file system to S3 on an ongoing basis.

It assumes there is some "mothership" that can accept a periodic "phone home" event as an HTTP(S) request. It will send the contents of s3-ingestor-context.json to the "mothership" and will receive a reply to store in the same place.

The each request to the "mothership" will include the following:

state: This is the state the s3-ingestor thinks it is in -- generally it is whatever the "mothership" last sent in reply to a "phone home" event, but unless the "state" is set to "configured," the s3-ingestor will not upload any files.
version: The version of the s3-ingestor.
info: This contains information about the s3-ingestor. This will always contain "hostname" (which is currently expected to be unique), but will also contain the following on "startup." Simplistically, this will be the following information:
- hostname: os.hostname()
- hosttype: os.type()
- platform: os.platform()
- release: os.release()
- totalmem: os.totalmem()
- network: os.networkInterfaces()
action: This tells the "mothership" what prompted the s3-ingestor to "phone home." These can be:
- startup: Sent when the s3-ingestor is started.
- heartbeat: Sent every "heartbeat_period" seconds as defined in the configuration.
- wakeup: Sent if the s3-ingestor is asked to "wake up" (see below).
- upload: Sent in response to an action replied by the "mothership" that results in files being uploaded.
- error: Sent if an error occurs in response to an action replied by the "mothership"
- If the mothership sends an "action" in reply to a "phone home" event, this action is echoed back unless an error occurs.
result: This changes with each type of action, but generally, it is the status of files uploaded or considered for uploading:
- added: New files uploaded.
- updated: Existing files updated.
- skipped: Files skipped because their file size matches what is already in S3.
- ignored: Files not matching the policy to be uploaded.
- unchanged: Files whose timestamp is the same as the last time it was considered.

The s3-ingestor has a set of default configuration options, but any of these can be overridden in a s3-ingestor.json configuration file:

debug: Turns on/off debug logging (default: false)
api_port: Port used for the "wakekup" HTTP service (default: 4567)
host_service: Protocol used for contacting the "mothership" (default: 'https')
host_dns: DNS name of the "mothersip" -- this must be provided
host_uri: The URI where the "mothership" expects to receive "phone home" events (default: '/ingestor')
host_port: The port on which the "mothership" is listening (default: 443)
heartbeat_period: The frequency in seconds of "hearbeat" events (default: 3600)
aws_keys_file: Where AWS keys are kept and expected to be provided by the "mothership" (default: 's3-ingestor-keys.json')
s3_bucket: Where S3 files will be put -- this must be provided and expected to be provided by the "mothership"
upgrade_command: The command used to upgrade itself when receiving an "upgrade" action from the "mothership" (default: 'npm update s3-ingestor')
policies: An array of policy JSON objects used for deciding what to upload to S3 (default: a single policy object with all defaults)

Policy JSON objects used for uploading files from specific locations with the local file system include the following fields:

input_file_pattern: The local file system pattern for files to consider (default: '*/')
input_remove_prefix: The prefix of the local file system filename to trunctate when making an S3 key (default: none)
output_key_prefix: The prefix to add when creating an S3 key (default: none)
customizer: The name of a Javascript file (minus the '.js') used to find a function to which to pass a filename to construct an key -- if this function returns a "falsey" value, the file will be ignored.

If the mothership sends "aws_keys" in the response, the contents will be stored in s3-ingestor-keys.json.

If the mothership sends "config" in the response, the config will be updated with whatever is included.

The "mothership" current can send the following "actions" to be performed by s3-ingestor:

customizers: New contents for the customizers directory will be pulled from the s3_bucket at S3 key prefix "code/s3-ingestor/customizers/".
report: The current configuration will be sent as the "result" to the "mothership".
upgrade: The s3-ingestor NPM package will be updated.

The s3-ingestor is designed to be monitored by PM2 (https://github.com/Unitech/pm2) or something similar. For example, if the "mothership" sends the upgrade action, s3-ingestor will upgrade itself and then terminate the process, assuming that PM2 will restart it.