@mondomob/gae-js-firestore-backups v9.0.0

GAE JS FIRESTORE BACKUPS

Module to perform scheduled Firestore exports in your app.

Backups can be configured to perform full or partial exports of Firestore into Google Cloud Storage.

As well as regular backups you can configure automatic export and import into BigQuery. This is a full import/export and any existing data in BigQuery will be deleted for the requested kinds.

Firestore exports are long-running operations. This module keeps track of operations it starts in a backup-operations firestore collection. After an operation is started it will queue tasks to update the status of the operation until it is complete. Once complete the backup can be auto-imported into BigQuery.

Installation

npm install @mondomob/gae-js-firestore-backups

Quick Start

Create queue to handle backup tasks

By default this library uses the task queue named backup-queue, so create that if it doesn't exist. Alternatively add configuration for the queue you wish to use.

e.g. add to queue.yaml (or add equivalent to terraform, etc)

queue:
  - name: backup-queue
    mode: push
    rate: 1/s
    retry_parameters:
      task_retry_limit: 20
      min_backoff_seconds: 120
      max_doublings: 5

Create bucket to store backup data

By default this library will use a bucket called firestore-backup-[PROJECT_ID], so create that if it doesn't exist. This must be a regional bucket in the same region as your firestore instance. Alternatively you can add configuration for the bucket you wish to use.

Add IAM Roles

Authorise the App Engine Default Service account to perform datastore exports In Google Cloud Console IAM & Admin add the following IAM role:

• Datastore -> Cloud Datastore Import Export Admin

Add cron and task routes to your app

e.g. at the root application

  app.use("/crons", gaeJsCron, firestoreBackupCronRoutes());
  app.use("/tasks", gaeJsTask, firestoreBackupTaskRoutes());

or as part of a sub-application/sub-router

  const cronsController = Router();
  cronsController.use(firestoreBackupCronRoutes())
  cronsController.get("/some-other-cron", asyncHandler(async () => {}))

  app.use("/crons", gaeJsCron, cronsController);

  const tasksController = Router();
  tasksController.use(firestoreBackupTaskRoutes())
  tasksController.post("/some-other-task", asyncHandler(async () => {}))

  app.use("/tasks", gaeJsTask, tasksController);

Add Cron schedule

Add cron schedules to your cron.yaml for the backup jobs you wish to run. e.g.

cron:
  - description: "Firestore export"
    url: /crons/backups/firestore?type=EXPORT&name=FullBackup
    schedule: every day 03:00
    timezone: Australia/NSW

  - description: "Firestore export and load to BigQuery"
    url: /crons/backups/firestore?type=EXPORT_TO_BIGQUERY&name=ExportToBigQuery&targetDataset=backup_data&collectionIds=demo-items
    schedule: every day 03:00
    timezone: Australia/NSW

Endpoint options

Default endpoint: GET /crons/backups/firestore

Query params:

Edit configuration (if required)

No configuration is required if you are happy with the default conventions.

The following options are available under the firestoreBackup namespace.

e.g.

{
  "projectId": "my-project-dev",
  "firestoreBackup": {
    "bucket": "my-backup-bucket",
    "queue": "admin-queue",
    "taskPrefix": "/tasks/admin",
    "taskService": "backup-service",
    "timeZone": "Australia/Sydney"
  }
}