create-backpack v1.9.11

Intro video: https://www.youtube.com/watch?v=RYBSJeUnZbc

What is Backpack?

Backpack is a REST API starter kit which includes a handful of helpful tools to get you up and running quickly without locking you into anything. After running the init script, you'll start with directory which looks like this:

api/
	devops/
	lib/
	models/
	routes/
	tests/
	server.js
	...and some other stuff

...So that's the first thing. Using Backpack means you start with a sensibly organized Express/Mongoose app. When you want to add a new CRUD collection like books for example, instead of manually creating a new models file and a new routes file and then importing those into server.js, you can just create them in their appropriate directories and they'll be auto-imported. Better yet, just run npm run model Book to automatically generate them, then just edit models/Book.js to define your schema. The goal is to start with a good boilerplate, then hack at it to make it your own.

The 3 core pieces to Backpack

1. The init script npm init backpack which instantly pulls down the Backpack boilerplate and creates a new api/ directory.
2. The Backpack middleware which simplifies things like sending response JSON or errors, and handling JSON web tokens and protected routes.
3. The client-side backpack-fetch NPM module which streamlines and improves fetch requests to your API.

Quick start

1. Create a new directory called myProject and cd into it.
2. Run: npm init backpack.
3. Browse to http://localhost:8100/api

Standardized JSON Response

When designing a REST API, it's important to impose some standards on what the response will look like and then adhere to those standards. For example, what properties can we count on our API response to always include? Using the provided Backpack middleware functions res.sendData() or res.sendError(), will always include at least status, data, and error properties; and when applicable, it will also include token and/or meta (for pagination data). Either data or error will always be populated with an object and the other will be null.

Example data response:

{
  "status": 200,
  "data": [...book1, ...book2, ...],
  "error": null
}

Example error response:

{
  "status": 404,
  "data": null,
  "error": {"message": "Not found"}
}

One of the main benefits of using Backpack is that planning this sort of standardization is already all figured out and implemented sensibly.

Core piece 1: The init script

To add a Backpack REST API to your project, just cd into the root of your app directory and run the NPM init script:

npm init backpack

This will add a new api/ directory at the current location, install dependencies, and start the server. You now have a running Backpack server.

Exploring the api/ directory

The file server.js is where the bulk of the action is. Take a look at that to discover that this is really just a pretty standard Express server which imports some middleware and auto-imports whatever files exist in the routes/ directory. Other than that, there's really nothing special.

The two most important directories in api/ are: models/ and routes. In models/, you'll find User.js and UserAuth.js which contain default Mongoose model schemas for managing users and JWT-based authorization. More on this later. The routes/ directory is where we define our Express routes and controller functions and starts off with a file for users.js. All pretty standard stuff. Even if we were to stop here and didn't use any other features of Backpack, we've already got a running API server with sensible defaults and auto-loading routes in about a minute.

Core piece 2: The middleware

Backpack includes middleware to help us work more efficiently. These middleware and the tools they provide are totally optional. They provide simplified methods for sending a response, easier token management, and a couple other handy tools.

New properties and methods added to Express req and res objects.

When writing controller functions to handle routes in Express, we're used to dealing with the req and res objects. Here's a typical example querying the database with Mongoose:

// GET /people/:id
router.get('/:id', async (req, res, next) => {
  const person = await Person.findById(req.params.id).catch(err => next(err));
  if (person) {
    res.status(200).json(status: 200, data: { person: person },
  }
});

You're certainly welcome to do that in your Backpack routes. But there's an easier way because Backpack middleware adds some nifty properties and methods to both the req and res objects. Probably the two most useful are sendData and sendError.

res.sendData(obj, [meta])

Sends the JSON response along with optional meta data and auto-included token if applicable.

Example: res.sendData( { name: 'Bob', age: 35 }).

This is almost the equivalent of res.status(200).json(status: 200, data: { name: 'Bob', age: 35}, error: null). The difference is that sendData will also include pagination and token properties when appropriate (see below).

res.sendError(status, [message], [details])

Sends an error response with appropriate HTTP status code and the appropriate message (automatically populated according to status code if not supplied explicitly) and optional details.

req.token

If a token was included in the request headers, the token middleware will try to validate it. If the token is valid, it will be decoded and assigned to req.token available for use in controller functions. Otherwise, res.token will be undefined. See "Tokens and Authorization" section below for details.

req.pagination

If the request has a querystring with either _page or _limit properties, those values will be used to create an object like { page: 2, limit: 20 } and set to req.pagination available for use in controller functions. See "Pagination" section below for details.

req.getQuery(Model)

Converts querystring parameters to a Mongoose query. For example, if a request came in with /employees?age=21, then req.GetQuery(Employee) would return { age: 21 }. This can then be passed into Mongoose find() to filter results. Note that the model is passed in so that the key ("age" in this case) can be checked to make sure it's a valid schema key and also to handle keys which are String types differently. For example, /employees?name=bob would be converted to { name: { $regex: 'bob', $options: 'i' } }. This allows for matching partial strings and is case-insensitive.

res.encodedToken

Whenever a new token is created or refreshed, the newly created raw (not decoded) token is assigned here. While this is unlikely to be very useful in controller functions. It's used by the middleware for the response method sendData(). If res.encodedToken is set, it will be included in the response JSON.

res.setToken(obj)

See this in action in routes/users.js. When the user has successfully authenticated with password or validation key or whatever method we decide to use, we need to issue a token. Do that with res.setToken() and pass in an object with whatever data you want to save in the token. Using the defaults provided with Backpack user auth standards, those properties will be _id, and roles .

Core piece 3: The client side module

Rather than using fetch or axios to make requests from the client-side, we can use the companion NPM module backpack-fetch which was designed to work perfectly with Backpack. From backpack-fetch we import the api module which has methods get(), post(), patch(), and delete() for our standard CRUD requests. It also has another method setUrl() which allows us to set the base URL for the API server just once for the whole application (be sure to call setUrl() early in the life cycle of your app).

The standard CRUD methods all accept an endpoint as the first argument and post(), and patch() accept a payload object as the second request.

Example using standard fetch without backpack-fetch:

const baseUrl = 'http://localhost:8100/api/';

async function postProduct(productObj) {
  try {
    const res = await fetch(baseUrl + 'product', {
      method: post,
      body: JSON.stringify(productObj),
      headers: {
        Accept: 'application/json',
        'Content-Type': 'application/json'
      }
    });
    const json = await res.json();
    console.log(json);
  } catch (error) {
    console.error(error);
  }
}

postProduct({ name: 'iPhone', description: "Duh. It's an iPhone.", price: 899 });

This will be a lot easier using the api module from backpack-fetch.

Example with backpack-fetch:

// First: `npm install backpack-fetch`
import api from 'backpack-fetch';
api.setUrl('http://localhost:8100/api');

async function postProduct(productObj) {
  const res = await api.post(productObj).catch(console.error(error));
  res.error ? console.error(res.error) : console.log(res.data);
}

postProduct({ name: 'iPhone', description: "Duh. It's an iPhone.", price: 899 });

The postProduct() function went from 14 lines of code to 2 and does precisely the same thing. Besides being slimmer and sexier, there are other advantages to using the api module:

• Conversion from JSON response to object is automatic
• Accept and Content-Type Headers for JSON payloads added automatically
• Guaranteed Backpack standardized response object (even if the server doesn't respond)
• Handles server timeouts (default is 4 seconds)
• Configurable base URL
• Automatically includes localStorage token as Authorization header for every request
• Response token automatically saved to localStorage

Bonus core piece: The code generator

There's nothing magical about the files in models/ or routes/ and it's completely fine to manually add more models and routes here directly. But there's a faster way:

npm run model <MyModel>

Creates 3 new files:

models / MyModel.js; // A Mongoose model with empty schema
routes / MyModel.js; // An Express routes file with basic CRUD routes
tests / MyModel.http; // HTTP tests for VS Code "HTTP Client" extension

It's important to emphasize that this script only generates these three files. Nothing more. Basic CRUD routes and tests are now ready to go. But the model schema still needs to be edited. So the typical workflow is to run npm run model <MyModel>, then edit the schema for your new model. At some point, you'll likely want to edit the routes file and maybe the tests too – but immediately after running the generator and editing the schema, you've got basic CRUD routes already working. Again, these three files are meant to be hacked. They're' just a starting point.

What's in the tests/ dir?

If you use VS Code and have installed the amazing "Rest Client" extension, then this file is a simple alternative to using something like Postman to test your routes. The code generator above will create this file with a handful of standard CRUD routes which are clickable so you can test API endpoints. And of course you can add more routes or hack it up however you like. Or you can just ignore these files and use Postman or whatever.

Pagination

https://github.com/aravindnc/mongoose-paginate-v2

Backpack uses mongoose-paginate-v2 with a couple config tweaks that make it work with our standardized JSON response. Model files made using the generator script will include this plugin. Then in your controllers, when you want paginated results, just use <Model>.paginate() instead of <Model>.find().

Whereas <Model>.find() will return an array, <Model>.paginate() will return an object like this: { data, meta }.

Example using standard find() method:

const data = Book.find();
res.sendData(data);

Example using paginate() method:

const { data, meta } = Book.paginate({}, { page: 3, limit: 20 });
res.sendData(data, meta);

Notice that we send meta as the second argument to res.sendData().

The meta object contains pagination information like this:

{
  (totalDocs = 547),
    (limit = 20),
    (page = 3),
    (totalPages = 28),
    (hasNextPage = true),
    (nextPage = 4),
    (hasPrevPage = true),
    (prevPage = 2),
    (pagingCounter = 3);
}

As discussed above, the middleware also has a convenience helper for finding pagination parameters passed into querystrings that look like this: /books?_page=5&_limit=20. These should be prefixed with an underscore since it's possible that page or limit could be names of properties you'd want to be able to use. So we reserve these two names prefixed with underscores instead.

When _page and optionally _limit are included in the querystring, the middleware removes these special keys from req.query and instead sets them at req.pagination as page and limit. This makes it very easy to get paginated results like this:

Ex: /books?_page=5&_limit=20

// Middleware has populated this --> req.pagination == { page: 5, limit: 20 }
const { data, meta } = Book.paginate({}, req.pagination);
sendData(data, meta);

The paginate() method accepts a query as the first argument, and an options object as the second. We passed in the options object with properties page and limit since that's what req.pagination will always include.

💡 The limit property is optional and has a default value of 50 (this setting is in server.js).

Tokens and Authorization

If you're not a JSON Webtoken aficionado, the basic idea is that the server is able to validate a token by hashing it against the key it was generated with – without hitting the database. So whatever value exists for JWT_KEY in the .env file is used to both generate tokens and also to validate them. This key was randomly generated during the configuration phase of npm init backpack.

The basic idea

We want to avoid doing a database lookup for every single protected request. For example, an admin user logged into a control panel we've built into the app might make a hundred are so requests in a 10 minute session. Using JSON webtokens means we can issue the token once, then just validate that token on the server side for every subsequent request – until the token has expired or is due for a refresh.

The reason for having both exp and refresh

The problem with issuing a token which grants privileges without any database calls is that there's no way to invalidate the token from the server side without invalidating all tokens. So we want to keep the interval somewhat short in case privileges have been revoked (or downgraded or upgraded) for whatever reason. In other words, if this user's admin privileges have been revoked, the only way her token becomes invalidated is if we check the database and discover that. This is why we "refresh" the token at a set interval (default is 15 minutes).

If the exp timestamp is outdated, the user is actually required to get a new token (usually by logging in again). We don't want to make the user log in every 15 minutes. So behind the scenes, if the token refresh is outdated but the exp is not, we just check the database and then issue a new token with updated values for exp, refresh, and roles.

Auto-populated req.token

If an Authorization header was included with a request AND if the token can be validated, that token will automatically be decoded and set to req.token. Then in your Express routes function, you can easily access things like req.token.roles (to check for admin privileges, for example) or whatever other properties you may be storing in the token.

💡 It's worth pointing out here that these "magical" new properties aren't really magical at all. And it doesn't lock you into any particular way of doing things. The middleware file lib/token.js simply looks for an Authorization header and if there's a valid token there, it decodes it and sets it to req.token. It's available for you to use or ignore. Totally up to you.

Setting a new token with res.setToken()

When a user has authenticated (by clicking an email link or whatever), this method accepts an object with properties to be stored in the token. The token must include an _id property so that the user can be authenticated for token refresh. If you don't supply exp or refresh (usually you won't), those will be populated for you. The default users routes include endpoints for getting a new token with validation key or password. Both of these prebaked routes create a new User instance, then pass the _id and roles properties to res.setToken(). Check out routes/users.js to see this stuff in action.

Auto token refresh

By default, a token will expire after 7 days. When a token expires, the user will be required to get a new token (typically by re-authenticating). In addition to all tokens having an exp property, tokens created with res.setToken() will also have a refresh property with a default of 15 minutes.

💡 Defaults can be changed at the top of lib/token.js

So during a session, instead of checking the user auth against the database for every request, we can just validate the token for every request until the token is due for a refresh. When the current time exceeds the value of refresh, the middleware will lookup the User by the _id in the token and then generate a new token with updated exp, refresh, and roles values.

Auto token inclusion with response

When using res.sendData(), the new tokens will be included as a property of the JSON response object. Whenever a new token is generated either by calling res.setToken() or because the token was refreshed, in either case the new token is set to res.encodedToken. That's the new base-64 encoded token which isn't likely very useful for us to access in the routes. But the fact that it's there means that res.sendData() will include it in the response.

So instead of:

{ "status": 200, "data": {...theData }, "error": null }

The client should instead get something like:

{ "status": 200, "data": {...theData }, "error": null, "token": "123456.123456.123456" }

Putting all that together...

So that may all seem like a lot random bits of token management. But when you put it all together and use it in combination with the api methods provided by backpack-fetch on the frontend, token management becomes completely automated from the point you call res.setToken() during user authentication (which is already done for you in the default routes in routes/users.js). Let's look at how the token is handled during a round-trip request:

• The user hits an endpoint perhaps with a validation code so we know we can now safely authenticate this user.

• Your controller function calls res.setToken() which will create a new token and this value is set to res.encodedToken.

• When you send the response using res.sendData(), the token will be included in the response because res.encodedToken exists.
• If the request was made using one of the api methods provided by backpack-fetch, the token property of the JSON response will be set to the browser localStorage.
• On the next request the user makes, if there's a token found in localStorage, it will be included in the next request as an "Authorization" header.
• Back on the server side, any request that includes an Authorization header will try to validate that token value. If it validates AND isn't expired AND isnt due for a refresh, the token is decoded and set to res.token for easy access in your routes.
• If the token is due for a refresh, the token middleware finds that user by the token _id and sets new values for exp, refresh, and roles and saves the new token to res.encodedToken and the decoded version to res.token. Which means it will be included in res.sendData(), and assuming you're using backpack-fetch on the client side, will automatically update the token in localStorage.

Token guarantees

Using the token middleware with sendData and backpack-fetch on the client side. You can count on these things to always be true:

1. On the server side, If you create a new token or a token has automatically refreshed, it will be included in the JSON response using sendData.
2. Using backpack-fetch on the client side, if there's a token included in the JSON response, it will always be saved to localStorage.
3. Also using backpack-fetch, if there's a token in localStorage, it will always be included in the request headers.
4. On the server side, whenever an Authorization header exists, the included token is validated and refreshed if due for refresh, then included with the JSON response (see Step 1).

So this effectively gives us automated rolling sessions with JSON webtokens.

Protected routes

In lib/auth.js which can be modified like anything else, there are two middleware functions available to be used in protected routes: requireAuth and requireAdmin. For any routes you want restricted to a logged in user (with a valid token), you could do this in your routes file:

// Example for GET /secretThing/:id
router.get('/:id', requireAuth, async (req, res, next) => { // NOTICE THE SECOND ARG
  const thing = await secretThing.findById(req.params.id).catch(err => next(err));
  thing ? res.sendData(thing) : res.sendError(404);
}

That second argument requireAuth simply checks for the presence of res.token. Remember, for every request, the token middleware always attempts to validate any token found in the headers and then sets it res.token only if it's present and valid. So determining whether a user has a valid token is extremely easy. If there's a res.token, the user has a valid token.

Instead of using the middleware above, we could have just added a line like this as the first line inside the function block:

if (!res.token) return res.sendError(401);

If you open up lib/auth.js, you'll see that's exactly what it does. In fact, let's take a look at that right now:

In /lib/auth.js

requireAuth(req, res, next) {
  // Validate token.
  if (!res.token) return res.sendError(401);
  next();
},

Pretty dang simple. Either adding that line as shown above or inserting the middleware into your route function as shown at the top of this section, either approach has the same exact effect. The middleware way is a little more terse which is nice for a full page of protected routes functions – keeps things DRY.

The other auth function available by default is requireAdmin. This is basically the same but takes the additional step of checking that the token.roles array includes the value 'admin'. Of course, it's trivial to add other auth functions here as your app may require.

Additional planned features

File uploads

Although this is not baked in yet, there's nothing stopping you from adding this functionality in your routes. The plan is to add baked in functionality using "Multer" at some point.

Logging

Currently only logging to console is supported (using Morgan). Should be pretty easy to configure Morgan in server.js to log to files. Not to be a broken record but all this stuff is just a starting point. You aren't locked into anything. Hack at it!