gmail-sync-service v1.0.2

Gmail Sync Service

Features:

• Efficiently syncs Google Gmail inboxes with your own MongoDB database.

• Supports updates via Gmail API server push notifications, using the recommended Google Cloud Pub/Sub and Gmail watch setup explained here.

• Syncs attachments based on content id (hash). Never fetches the same attachment twice.

Usage:

Step 0: Setup a Google Cloud Platform Project.

Go here, setup a new project and create and download OAuth client ID credentials.

Step 1: Create an .env config file and fill it in as follows:

# Google Oauth client ID credentials
GSS_GOOGLE_CREDENTIALS=../google-gmail-sync-credentials.json
# Google Pub/Sub topic to subscribe to for Gmail watch updates.
GSS_GOOGLE_TOPIC_NAME=projects/taskworld/topics/gmail-webhook
# MongoDB connection URL.
GSS_MONGO_URL=mongodb://localhost/db_dev
# Where to store the user token (used mainly while testing).
GSS_TOKEN_CACHE_FILE=/tmp/user_token.json
# Server host
GSS_HOST=localhost
# Server port
GSS_PORT=10003
# Server TLS certificate
GSS_CERT=../keys/cert.crt
# Server TLS certificate private key
GSS_CERT_KEY=../keys/cert.key

Step 2: Test the command-line tool.

# Install Node dependencies.
$ yarn install

# Run the tool without flags, which prints it’s usage.
$ ./cli.js

    Usage:
    ./cli.js
      --auth          Get Google Auth URL.
      --code          Exchange code for access & refresh tokens.

      --messages      Fetch Gmail messages related to a given email address.
        --email       Token owner’s email address.
        --userId      Token owner’s user id in your system.
        --watch       Google Cloud Pub/Sub topic to publish changes to,
                      e.g. projects/taskworld/topics/gmail-webhook
                      NOTE: This topic must already exist.
        [--force]     Force refetching of messages (optional).
        [--only]      Fetch only messages that were received from or sent to this email address (optional).

      --server        Run a server that listens to incoming Gmail webhook calls.
                      This uses the Gmail watch API and Google Cloud Pub/Sub to watch the token owner’s inbox for
                      changes, outlined here:
                      https://developers.google.com/gmail/api/guides/push
                      https://cloud.google.com/pubsub/docs/push

Step 3: Authenticate a Google user.

$ ./cli.js --auth
Follow this auth URL:
https://accounts.google.com/o/oauth2/auth?access_type=offline&scope=....

# Copy the code from the redirected URL and feed it back to ./cli.js
$ ./cli.js --code <CODE FROM REDIRECT URL>

You should now have a token in /tmp/user_token.json which is automatically used in subsequent calls to ./cli.js.

Step 4: Fetch messages.

$ ./cli.js \
  --messages \
  --email  ace@base.se \  # Token owner’s email address.
  --userId USER1 \        # Token owner’s user id, i.e. a reference to a user id in your system.

[GSS] Connected to MongoDB server: mongodb://localhost/db_dev (database: db_dev)
[GSS] Kept 8 / 8 messages.
[GSS] 1. Fetching message 15d6d836d7420001 ..
[GSS] 2. Fetching message 15d6821649350002 ..
[GSS] 3. Fetching message 15d67c3811be0003 ..
[GSS] 4. Fetching message 15d67970ef890004 ..
[GSS] 5. Fetching message 15d64d2699cc0005 ..
[GSS] 6. Fetching message 15d5f9b7ca420006 ..
[GSS] 7. Fetching message 15d502e957370007 ..
[GSS] 8. Fetching message 15d502048ee30008 ..
[GSS] Saved attachment: "Screen Shot 2017-07-17 at 5.37.30 PM.png" (22123 bytes) to /tmp/e987c208149f17e35c83fad59c80f7a63fe94c7bcc5ebce363c5ce01f019e558.png
[GSS] Saved attachment: "Screen Shot 2017-07-20 at 5.46.22 PM.png" (93685 bytes) to /tmp/29a993df364c1455134e9ebb7c0975af0580977aef835a1b994d9e2845d4c452.png
[GSS] Saved attachment: "Screen Shot 2017-07-05 at 23.02.19.png" (48765 bytes) to /tmp/19403eae291ad4b1138d781d08ecd2350ed4d6ed336485e1cd69f86bb88256f4.png
[GSS] Downloaded 3 / 15 attachments (identical: 12).
[GSS] Saved 8 messages.

Step 5: Fetch updates since the last fetched message.

$ ./cli.js --messages --email ace@base.se --userId USER1 --update

[GSS] Connected to MongoDB server: mongodb://localhost/db_dev (database: db_dev)
[GSS] Found 0 new / added messages since history id 1888.

Step 6: Setup a watch for this user, i.e. the token owner.

$ ./cli.js --messages --email ace@base.se --userId USER1 --watch

[GSS] Connected to MongoDB server: mongodb://localhost/db_dev (database: db_dev)
[GSS] Setting up Gmail watch for ace@base.se (id: USER1). Publishing to topic projects/taskworld-crm-174408/topics/gmail-webhook
[GSS] Saving Gmail watch for ace@base.se (id: USER1) expiring on Mon Jul 31 2017 08:40:36 GMT+0700 (+07)

Step 7: Start a local server to listen for Google notifications (nice while testing).

# Terminal window 1
# Open a SSH tunnel to the remote host which Google will
# be pushing it’s notifications to.
$ ./server-tunnel.sh my-remote-host.nginx.com

# Terminal window 2
$ ./cli.js --server

# Or if you want to run a custom update function when you get a notification
$ ./cli.js --server --onUpdate ./my-custom.update-function.js

# Terminal window 3
$ ./post-test.sh

Database

Running a fetch produces the following collections in MongoDB:

Collection: gss_messages

[
  {
      "_id" : "15d6821649000001",
      "userId" : "USER1",
      "threadId" : "15d502048e000001",
      "labelIds" : [
          "UNREAD",
          "IMPORTANT",
          "CATEGORY_PERSONAL",
          "INBOX"
      ],
      "snippet" : "Have ya paid your dues, Jack? Yessir, the check is in the mail.",
      "historyId" : "1830",
      "date" : "1500690593000",
      "fromName" : "Anri Digholm",
      "fromEmail" : "anri@taskworld.com",
      "toName" : "Jack Burton",
      "toEmail" : "jburton@bigtroubleinlittle.china",
      "parts" : [
          {
              "partId" : "0.0",
              "mimeType" : "text/plain",
              "body" : "Have ya paid your dues, Jack? Yessir, the check is in the mail.",
              "size" : 63
          },
          {
              "userId" : "USER1",
              "messageId" : "15d6821649000001",
              "partId" : "1",
              "mimeType" : "image/png",
              "filename" : "Screen Shot 2017-07-17 at 5.37.30 PM.png",
              "size" : 22123,
              "attachmentId" : "downloaded",
              "contentId" : "15d501fec2f9a635ecc1",
              "localFile" : "/tmp/e987c208149f17e35c83fad59c80f7a63fe94c7bcc5ebce363c5ce01f019e558.png"
          },
      ]
  }
]