1.0.1 • Published 7 months ago

@suhendra/s3-zip v1.0.1

Weekly downloads
-
License
MIT
Repository
github
Last release
7 months ago

s3-archiver

Archive files in Amazon S3 to zip in parallel

License

License

Sample Command

ACCESS_KEY=<s3-access-key>
SECRET_KEY=<s3-secret>
REGION=<s3-region>
MANIFEST_BUCKET=<s3-bucket>
FILE_BUCKET=<s3-bucket>
MANIFEST="$PWD/manifest.json"
PATTERNS='["requests/2021/07/31"]' 
node script.js

What This Script Does

  1. Download remote AWS S3's manifest.json file, or use local's based on env variable USE_LOCAL_MANIFEST
  2. Download the child manifest files locally, or use local's based on env variable USE_LOCAL_MANIFEST
  3. Iterate through all manifest files to look for matches according to the patterns in env variable PATTERNS
  4. Iterate through patterns, then through it's matches
  5. Download the matches locally then pipe it to zip file
  6. Upload the zip file to S3
  7. Remove the archived keys in S3, then remove the locally downloaded files

Env Variables

VariableDescription
ACCESS_KEYAWS S3 access key
SECRET_KEYAWS S3 secret key
REGIONAWS S3 region
MANIFEST_BUCKETAWS S3 bucket of the manifest files
FILE_BUCKETAWS S3 bucket of the files that will be archived
MANIFESTManifest file name to be downloaded, or looked up locally
USE_LOCAL_MANIFESTIf true, then will use local manifest file without downloading. Default to false
PATTERNSRegex patterns to match the files that will be archived, use JSON syntax, e.g ["a", "b"]
s3archivenodejs
@aws-sdk/client-s3archivercsv-parsercli-progressfszlib
@everything-registry/sub-chunk-880
1.0.1

7 months ago

1.0.0

7 months ago