@concord-consortium/token-service

Token Service

The Token Service provides a way for browser based applications to get credentials to access cloud services. This is done by maintaining a database of tools and resources. The tools represent the browser based application. The resources represent the parts of the cloud services that a particular user or group should have access to. A tool can request temporary credentials for a resource on behalf of a user.

Users are authenticated with the token service using a Firebase JWT which can be obtained from the Concord portal.

The Token Service is similar to some features provided by AWS Cognito. A key difference is that users can create new resources themselves in the token service. This allows them to create places in the cloud services to work or store files which only that user can access. This also means we need to be careful what features of the cloud services we expose so users cannot intentionally or unintentionally incur large costs, or abuse the services.

Currently the Token Service supports two types of resources:

• AWS S3 Folders
• AWS Athena Workgroups

Client (client subdirectory)

A javascript client is published in the @concord-consortium/token-service package that handles talking to the token service. Written in TypeScript, it emits type definitions so that it can be imported with types in another TypeScript package.

The client constructor has the following options defined in client/src/client.ts:

type EnvironmentName = "dev" | "staging" | "production"

export interface TokenServiceClientOptions {
  jwt: string;
  env?: EnvironmentName;
  serviceUrl?: string;
}

The jwt option is required and the serviceUrl and env options are optional. If the env option is not supplied it is determined via the window location. If the serviceUrl option is not supplied it falls back to using the service url defined for the env and if that is not found defaults to the production service. The serviceUrl option can be overridden via the token-service-url query parameter.

Example App (example-app subdirectory)

https://token-service.concord.org/example-app/index.html

This app is designed to show how to use Token Service (TokenServiceClient) and it's meant to be used by developers to work on the new Token Service features or to provide recipes showing how to use its API. Note that errors are not handled, so in the real applications, you should consider adding that. It's not done here on purpose to keep the code smaller and less opinionated.

Service API (functions subdirectory)

This is the actual token service code. It implements a REST API to interact with resources. If possible, your browser based app should use the client library above to work with the API instead of directly using the API.

Administration

Creating new S3ResourceTools

To Create a new S3 Resource tool, create a resourceSettings document in FireStore for each environment you work in, eg: staging:resourceSettings The entry should look like this:

  bucket: "token-service-files",
  folder: "example-app",   // good idea to match `tool` below.
  region: "us-east-1",
  tool: "example-app",
  type: "s3Folder",
  allowedAccessRuleTypes: ["user", "context"], // optionally "readWriteToken", check AccessRuleType type for all allowed values
  domain: "https://cloudfront.domain.com", // optionally domain that will be used to construct public URL, usually a cloudfront domain
  domainIncludesFolder: true // optional, but it needs to be set when the domain points to S3 bucket folder, not just the root. It prevents token-service from appending folder to the path and URL again.

Creating authenticated (shared) resources

A resource with an authenticated access rule grants temporary AWS credentials to any user with a valid portal JWT, so any portal user can write to the resource's S3 folder. These resources are never created through the API — POST /resources and PATCH /resources/:id reject authenticated rules unconditionally. Instead, an admin creates the resource document directly in Firestore (e.g. in staging:resources), choosing the document id so it maps to the desired S3 path ({folder}/{id}/), and includes the rule in accessRules:

  type: "s3Folder",
  tool: "example-app",
  name: "shared-folder",
  description: "World-writable shared folder",
  accessRules: [
    { "type": "authenticated" }
  ]

Note that authenticated is special: unlike user, context and readWriteToken — where adding the type to a tool's allowedAccessRuleTypes is what enables API creation of resources with that rule — authenticated is never created through the API, so it does not belong in allowedAccessRuleTypes and listing it there has no effect.

Creating new AthenaWorkgroupTools

To Create a new Athena Workgroup Resource tool, create a resourceSettings document in FireStore for each environment you work in, eg: staging:resourceSettings The entry should look like this:

  bucket: "token-service-files",
  folder: "example-app",
  region: "us-east-1",
  tool: "example-app",
  type: "athenaWorkgroup",
  allowedAccessRuleTypes: ["user"],
  // AWS account where Athena instance is located
  account: "123456"

Configuration

The Service API uses Firebase parameterized configuration. Secrets are stored in Google Cloud Secret Manager and bound to the Cloud Function at deploy time. Non-secret configuration uses environment variables.

Secrets (set via firebase functions:secrets:set <NAME>):

• ADMIN_PUBLIC_KEY — RSA public key to validate JWT tokens (multiline PEM — pipe from file: firebase functions:secrets:set ADMIN_PUBLIC_KEY < key.pem)
• AWS_KEY — AWS IAM user access key
• AWS_SECRET — AWS IAM user secret key
• AWS_ROLE_ARN — AWS IAM Role ARN with a trust relationship to the IAM user above. This role defines the permissions the token service has in AWS. The token service can only provide credentials to clients that grant permissions which are a subset of this Role's permissions.

Environment variables (set in functions/.env for local dev, functions/.env.<projectId> for deployed environments):

• AWS_DURATION — time in seconds that temporary credentials are valid (e.g., 3600)

Note: never set FIRESTORE_EMULATOR_HOST on deployed functions — besides breaking Firestore access, it re-enables the local-development-only test token shortcut.

Development Setup

TokenServiceClient

1. cd client (all the commands below should be executed in client/ dir)
2. Run npm i
3. Run npm run build

It will create lib/ dir with client JS files.

Example App

This app is using TokenServiceClient. It is linked in the example-app/package.json using relative path ../client. It lets you develop the client and the example app together. But it means that the client needs to be built first (follow steps described in Development Setup > TokenServiceClient). Also, TokenServiceClient needs to be rebuilt each time there are some changes that you would like to use in the example app. Its build process can also be run directly from example-app dir using npm run client:build.

To start local server with example app:

1. cd example-app (all the commands below should be executed in example-app/ dir)
2. Install dependencies: npm i
3. Start local server: npm start

Firebase functions

Basic Firebase setup (done once)

1. cd functions (all the commands below should be executed in functions/ dir)
2. Run npx firebase login
3. Run npx firebase use staging to use https://console.firebase.google.com/project/token-service-staging Firebase project (to use production version: firebase use default).
4. Create functions/.secret.local with your development secrets:

   ADMIN_PUBLIC_KEY=<public key value>
   AWS_KEY=<AWS api key>
   AWS_SECRET=<AWS api secret key>
   AWS_ROLE_ARN=<AWS Role ARN>

5. Copy the env template (from within functions/ dir):

   cp .env.example .env

6. Set up admin credentials:

• Open https://console.firebase.google.com/project/token-service-staging/settings/serviceaccounts/adminsdk
• Click "Generate New Private Key" and download JSON file
• Save it as functions/key.json — this is gitignored, never commit or share this file

Firebase Functions server

1. cd functions (all the commands below should be executed in functions/ dir)
2. Run npm i to load all the dependencies
3. Run npm start to start all servers

Testing

1. cd functions (all the commands below should be executed in functions/ dir)
2. npm run test:with-emulator starts Firestore emulator and run all the tests
3. Alternatively, you can start emulator first using: npm run firestore-emulator and then use npm test or npm run test:watch. This is better for development, as emulator doesn't have to start and shutdown each time.