coinstats-data-archive v1.0.3

Coinstats Data Archive

About

This library is intended to use when you want to archive the data.

It will put the data in s3 bucket with partition keys so later you can run queries with Atlas Federated Data Lake or AWS Athena

Installation

npm i coinstats-data-archive --save

Usage

const { ArchiveClient } = require('coinstats-data-archive'); // ES5
import { ArchiveClient } from 'coinstats-data-archive'; // ES^

const client = new ArchiveClient({
  // this may be skipped if credentials are in env variable in AWS format
  s3Options: { 
    region: process.env.REGION_S3,
    credentials: {
      accessKeyId: process.env.ACCESS_KEY_ID_S3,
      secretAccessKey: process.env.SECRET_ACCESS_KEY_S3
  },
  bucketName: 'archive-bucket',
  fileNameGenerator: (generatorArgs) => new Date().toISOString() // this is default name generator
  },
});

const user = { id: '123' };
const collection = 'transactions';
const partitionKeys = [user.id];
const data = [{ id: '1', event: 'test' }, { id: '2', event: 'test2' }];

await client.archiveData(collection, data, partitionKeys);
// the object will be saved under s3://archive-bucket/transactions/123/2023-02-01T07:51:38.912Z.json.gz

You can also use silentArchiveData function which will run archive and handle error without throwing it. For it you need these env variables

process.env.S3_ARCHIVE_BUCKET_NAME
process.env.REGION_S3
process.env.ACCESS_KEY_ID_S3,
process.env.SECRET_ACCESS_KEY_S3

If environment is configured to get aws credentials via STS you can only provider bucket name

Methods and Properties

ArchiveClient constructor -> { s3Options: S3ClientConfig (https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-s3/interfaces/s3clientconfig.html), bucketName: string, fileNameGenerator: function(generatorArgs) => string }

archiveData(collection, data, partitionKeys, generatorArgs?) -> collection: string, data: object, partitionKeys: [string], generatorArgs: object (will be passed to fileNameGenerator function