multer-remote-storage v0.0.6
multer-remote-storage
Use Multer to easily upload files to Cloudinary, AWS S3, or Google Cloud Storage
- Easily switch between storage targets
- Use your own validator function to prevent file uploads in the event other HTTP request body data does not pass validation
- Add a chunk_size to the options to automatically trigger a chunked upload for your file
Contents
Setup
Run this command to install:
npm install multer-remote-storage
Examples
Cloudinary Example
storage.js
import {v2 as Cloudinary} from 'cloudinary';
import { RemoteStorage } from 'multer-remote-storage';
Cloudinary.config({
cloud_name:process.env.CLOUDINARY_NAME,
api_key:process.env.CLOUDINARY_API_KEY,
api_secret:process.env.CLOUDINARY_SECRET,
secure:true,
});
const storage = new RemoteStorage({
client:Cloudinary,
params: {
folder:'Myfolder'
}
});
export { storage }
route.js
import multer from 'multer';
import { storage } from '../utilities/storage.js';
import {filter} from '../utilities/validators/fileValidator.js'; //Multer-related
import { topicValidation } from '../utilities/validators/middleware/validators.js'; //JOI
const router = express.Router({ caseSensitive: false, strict: false });
const parser = multer({storage, fileFilter:filter, limits:{fileSize:1024000}});
router.route('/:username/create')
.post(isLoggedIn,isAuthor,parser.single('topic[file]'),topicValidation,createTopic)
Property | Value |
---|---|
client | Cloudinary namespace |
params? | Same as Cloudinary's upload_stream and upload_chunked_stream |
options? | See Below |
The following data will be appended to req.file
variable | data type | info |
---|---|---|
etag | string | |
filename | string | includes folder, excludes extension |
folder | string | |
height | number | if applicable |
width | number | if applicable |
path | string | public URL |
signature | string | |
size | number | |
timeCreated | string | |
versionId | string |
Google Cloud Storage Example
storage.js
import {join,dirname} from 'path';
import { Storage as Gcs } from '@google-cloud/storage';
import { RemoteStorage } from 'multer-remote-storage';
const __filename = fileURLToPath(import.meta.url);
const gcsStorage = new RemoteStorage({
client:new Gcs({
keyFilename: join(dirname(__filename),'pathToKey.json'),
projectId: 'your-google-storage-projectId',
}),
params:{
bucket:'mybucket'
}
});
export { gcsStorage }
router.js
import multer from 'multer';
import { gcsStorage } from '../utilities/storage.js';
import {filter} from '../utilities/validators/fileValidator.js'; //Multer-related
import { topicValidation } from '../utilities/validators/middleware/validators.js'; //JOI
const router = express.Router({ caseSensitive: false, strict: false });
const gcsParser = multer({storage:gcsStorage, fileFilter:filter, limits:{fileSize:1024000}});
router.route('/:username/create')
.post(isLoggedIn,isAuthor,gcsParser.single('topic[file]'),topicValidation,createTopic)
Property | Value |
---|---|
client | Google Cloud's Storage Class |
params | bucket (required) PLUS options for uploading object to bucket |
options? | See Below |
The following data will be appended to req.file
variable | data type | info |
---|---|---|
bucket | string | |
contentType | string | |
etag | string | |
filename | string | |
path | string | public URL |
size | number | |
storageClass | string | |
timeCreated | string |
AWS S3 Example
storage.js
import { S3Client } from '@aws-sdk/client-s3';
import { RemoteStorage } from 'multer-remote-storage';
const s3Storage = new RemoteStorage({
client: new S3Client({
credentials: {
accessKeyId:process.env.S3_ACCESS_KEY,
secretAccessKey:process.env.S3_SECRET_KEY
},
region:process.env.S3_REGION,
}),
params: {
bucket:'mybucket'
}
});
export { s3Storage }
router.js
import multer from 'multer';
import { s3Storage } from '../utilities/storage.js';
import {filter} from '../utilities/validators/fileValidator.js'; //Multer-related
import { topicValidation } from '../utilities/validators/middleware/validators.js'; //JOI
const router = express.Router({ caseSensitive: false, strict: false });
const s3Parser = multer({storage:s3Storage, fileFilter:filter, limits:{fileSize:1024000}});
router.route('/:username/create')
.post(isLoggedIn,isAuthor,s3Parser.single('topic[file]'),topicValidation,createTopic)
Property | Value |
---|---|
client | S3Client Class |
params | bucket (required) and options for Upload class |
options? | See Below |
The following data will be appended to req.file
variable | data type | info |
---|---|---|
bucket | string | |
contentType | string | |
etag | string | |
filename | string | |
metadata | object | metadata passed in Upload options |
path | string | public URL |
encryption | string | |
versionId | string | undefined if versioning disabled |
Options
All options are optional. Some may only apply to certain clients. See below.
Options | Data Type | Description |
---|---|---|
chunk_size | number | This will trigger a chunked upload for any client |
public_id | string or (req,file,cb) => string | String or function that returns a string that will be used as the filename |
trash | string | Alternative text for trash text. See Validation Note |
validator | (req,file,cb) => boolean | See Validation |
leavePartsOnError | boolean | S3 Only |
queueSize | number | S3 Only |
tags | Tags[] | S3 Only |
Validation
Sometimes when uploading a file, it will be part of a form with other data that will exist on req.body. What if you need to validate that other form data before proceeding to upload the file? That's where this function comes in!
Here is an example:
/**
* You will have access to Express's req.body
*
* You can validate anything you wish and return true/false
* on whether you want the file to upload (true = upload)
*
* True or false will still cause the code to continue to the next
* middleware in your route. Calling cb(err) will result in Multer
* calling next(err) in your app.
*
* This example manually calls JOI's (a validation library for NodeJS)
* validation function to validate the fields on req.body
*
* In this example, you'll still need to call JOI's validator as a
* middleware in your route AFTER multer to actually give the client
* a response with error details
*
* You can create a more complex function that checks on which
* URL the client is visiting to decide which validator to use.
* This prevents you from having to create another RemoteStorage
* object for each type of validator
*/
const handleTopicValidation = (req, file, cb) => {
let output = true;
try {
const {error, value} = topicValidator.validate(req.body);
if (error) output = false;
} catch(err) {
output = false;
}
return output;
}
const topicStorage = new RemoteStorage({
client:Cloudinary,
params: {
folder:'myfolder'
},
options: {
chunk_size:1024 * 256 * 5,
validator: handleTopicValidation
},
});
Big Note on Validator
If you pass false, thus bypassing file upload, the software still has to pipe the readable stream somewhere. As a solution, it will use fs
's createWriteStream
function to create a file to dump the data in. Immediately afterwards, it will call fs
's rm
function to delete that file.
Two notes:
1. Make sure your Node app has writing privileges in its own directory. Otherwise, you might get an Access Denied
error.
2. The default filename created is called trash.txt
. You can use the trash
option to customize the filename so it doesn't mess with any file you may have.
Public Id
The public_id
option allows you to define a string, or a function that returns a string, to deal with naming the file you upload. Using this property may overwrite any similar function in the params
object for the client you are using.
NOTE: Cloudinary does NOT want the file extension in the filename whereas Google Cloud Storage and AWS S3 do.
const handlePublicId = (req,file,cb) => {
/*
This example uses the date object to ensure a unique filename and assumes only
one dot (prior to extension) exists
*/
return `${file.originalname.split('.')[0]}-${Date.now()}.${file.originalname.split('.')[1]}`
}
const s3Storage = new RemoteStorage({
client: new S3Client({
credentials: {
accessKeyId:process.env.S3_ACCESS_KEY,
secretAccessKey:process.env.S3_SECRET_KEY
},
region:process.env.S3_REGION,
}),
params: {
bucket:'mybucket'
},
options: {
chunk_size: 1024 * 1000 * 10,
public_id: handlePublicId
}
});
TypeScript Example
import { RemoteStorage } from 'multer-remote-storage';
import { v2 as Cloudinary } from 'cloudinary';
import { CLOUDINARY_NAME, CLOUDINARY_API_KEY, CLOUDINARY_SECRET } from './config';
import topicValidator from './validators/topicValidator';
//Types
import { Request } from 'express';
import { File, MulterCallback } from 'multer-remote-storage';
const handleTopicValidation = (req: Request, file: File, cb: MulterCallback) => {
let output = true;
try {
const { error, value } = topicValidator.validate(req.body);
if (error) output = false;
} catch (err) {
output = false;
}
return output;
}
const handlePublicId = (req: Request, file: File, cb: MulterCallback) => {
return `${file.originalname.split('.')[0]}-${Date.now()}}`
}
const storage = new RemoteStorage({
client: Cloudinary,
params: {
folder: 'Programminghelp'
},
options: {
public_id: handlePublicId,
validator: handleTopicValidation
},
});
export { Cloudinary, storage };