@davegarvey/plugin-catalog-backend-module-tyk v0.1.10

The Tyk Backstage entity provider imports Tyk API definitions and components into the Backstage catalog directly from Tyk Dashboards.

Getting Started

To use the Tyk Backstage entity provider, you will need an active Tyk installation with a valid Tyk Dashboard API token.

1. Package Installation

To install the package, run this command from the Backstage root directory:

yarn --cwd packages/backend add @davegarvey/plugin-catalog-backend-module-tyk

2. Module Configuration

To configure the Tyk entity provider, add a tyk section to the root of the Backstage app-config.yaml file.

This is an example configuration:

tyk:
  globalOptions:
    router:
      enabled: true
    scheduler:
      enabled: true
      frequency: 5
    importCategoriesAsTags: true
  dashboards:
    - host: http://localhost:3000
      token: ${TYKDASHBOARDAPITOKEN}
      name: development
      defaults:
        owner: group:default/guests
        system: system:default/tyk
        lifecycle: development

Note: It's possible to set configuration values using environment variables. See the use of ${TYKDASHBOARDAPITOKEN} in the above example.

Configuration Description

3. Plugin Configuration

Now that the entity provider is installed and configured, the final step is to configure the Backstage catalog to use the Tyk entity provider module. The process for this differs, depending on whether you are using the current or legacy architecture. Use of the current architecture approach is encouraged from Backstage v1.18.0 onwards.

Current Architecture

Follow this approach to configure the plugin for Backstage deployments using the current architecture (from Backstage v1.18.0).

Add this line to the Backstage packages/backend/src/index.ts file:

backend.add(import('@davegarvey/plugin-catalog-backend-module-tyk/alpha'));

The line can be added anywhere in the file between the lines const backend = createBackend(); and backend.start();, for example:

import { createBackend } from '@backstage/backend-defaults';

const backend = createBackend();

backend.add(import('@backstage/plugin-app-backend/alpha'));
backend.add(import('@backstage/plugin-proxy-backend/alpha'));
backend.add(import('@backstage/plugin-scaffolder-backend/alpha'));
backend.add(import('@backstage/plugin-techdocs-backend/alpha'));
// Tyk entity provider
backend.add(import('@davegarvey/plugin-catalog-backend-module-tyk/alpha'));

// auth plugin
backend.add(import('@backstage/plugin-auth-backend'));
// See https://backstage.io/docs/backend-system/building-backends/migrating#the-auth-plugin
backend.add(import('@backstage/plugin-auth-backend-module-guest-provider'));
// See https://github.com/backstage/backstage/blob/master/docs/auth/guest/provider.md

// catalog plugin
backend.add(import('@backstage/plugin-catalog-backend/alpha'));
backend.add(
  import('@backstage/plugin-catalog-backend-module-scaffolder-entity-model'),
);

// permission plugin
backend.add(import('@backstage/plugin-permission-backend/alpha'));
backend.add(
  import('@backstage/plugin-permission-backend-module-allow-all-policy'),
);

// search plugin
backend.add(import('@backstage/plugin-search-backend/alpha'));
backend.add(import('@backstage/plugin-search-backend-module-catalog/alpha'));
backend.add(import('@backstage/plugin-search-backend-module-techdocs/alpha'));

backend.start();

Legacy Architecture

Follow this approach to configure the plugin for Backstage deployments using the legacy architecture (prior to Backstage v1.18.0).

Several edits are required to the core backend catalog plugin file packages/backend/src/plugins/catalog.ts.

Follow the step-by-step process below. A fully edited example is available at the end of this section.

Step 1: Add the Import

Add this line to import the entity provider into the catalog plugin:

import { TykEntityProvider } from '@davegarvey/plugin-catalog-backend-module-tyk';

Put the line near the top, with the other imports.

Step 2: Create the Entity Providers

Add these lines to create the entity providers and add them to the catalog builder:

const tykEPs = TykEntityProvider.fromConfig({ config:env.config, logger:env.logger, scheduler: env.scheduler });
builder.addEntityProvider(tykEPs);

Put the lines after const builder: CatalogBuilder = CatalogBuilder.create(env); but before const {processingEngine, router} = await builder.build();.

Step 3: Create Routes (Optional)

This step is only necessary if the router functionality is enabled i.e. tyk.globalOptions.router.enabled is set to true.

In this case, add these lines to register the routes:

await Promise.all(tykEPs.map(async (ep) => {
  await ep.registerRoutes(router);
}));

Put the lines after await processingEngine.start(); but before return router;.

Note that this only covers the Backstage side of the process - see the Dynamic Data Import section below for information on the Tyk side.

Full Example

This example shows a fully edited packages/backend/src/plugins/catalog.ts file, with the three steps marked with comments Step 1, Step 2 and Step 3:

import {CatalogBuilder} from '@backstage/plugin-catalog-backend';
import {ScaffolderEntitiesProcessor} from '@backstage/plugin-scaffolder-backend';
import {Router} from 'express';
import {PluginEnvironment} from '../types';
// Step 1
import { TykEntityProvider } from '@davegarvey/plugin-catalog-backend-module-tyk';

export default async function createPlugin(
  env: PluginEnvironment,
): Promise<Router> {
  const builder: CatalogBuilder = CatalogBuilder.create(env);
  builder.addProcessor(new ScaffolderEntitiesProcessor());

  // Step 2
  const tykEPs = TykEntityProvider.fromConfig({ config:env.config, logger:env.logger, scheduler: env.scheduler });
  builder.addEntityProvider(tykEPs);  

  const {processingEngine, router} = await builder.build();
  await processingEngine.start();

  // Step 3
  await Promise.all(tykEPs.map(async (ep) => {
    await ep.registerRoutes(router);
  }));

  return router;
}

4. Validate Functionality

If the entity provider module is successfully installed and configured, you will see entries in the Backstage backend application logs related to initialisation and entity import.

Initialisation

On startup, the entity provider writes to the log to confirm that it has been initialised:

2024-04-08T09:08:44.125Z catalog info Tyk entity provider initialized for development Dashboard

Entity Import

On data import, the entity provider writes to the log to specify how many entities were imported and where they were imported from:

2024-04-08T09:08:45.315Z catalog info Importing 44 Tyk entities from development Dashboard entityProvider=tyk-entity-provider-development

Multi-Dashboard Configuration

It's possible to target multiple Tyk Dashboards in the entity provider configuration. To do this, specify multiple dashboards in the tyk.dashboards section of the Backstage configuration.

For example, this configuration defines two dashboards, development and production:

tyk:
  dashboards:
    - name: development
      host: http://tyk-dashboard.dev:3000
      token: ${TYKDASHBOARDAPITOKENDEV}
      defaults:
        owner: group:default/guests
        system: system:default/tyk
        lifecycle: development
    - name: production
      host: http://tyk-dashboard.prod:3000
      token: ${TYKDASHBOARDAPITOKENPROD}
      defaults:
        owner: group:default/guests
        system: system:default/tyk
        lifecycle: production

Note: For brevity, globalOptions is omitted from the above configuration.

Backstage Default Data

Some Backstage entity fields are not naturally part of Tyk's data set. Therefore, it's necessary to specify default values, so that entity data can be correctly assigned during the import process.

Default values are provided in the defaults part of each Tyk dashboard configuration. The values for owner, system and lifecycle must be defined, so that they can be applied as defaults to all entities imported from that dashboard.

Overriding Default Data

The default values can be overridden on a per-entity basis by providing the equivalent data in the Tyk objects being imported. In Tyk, use the API Definition config_data field to specify the data as a JSON object. The fields must be inside a root backstage object, for example:

"config_data": {
  "backstage": {
    "lifecycle": "production",
    "owner": "group:default/developers",
    "system": "system:default/tyk-development-environment"
  }
},

The entity provider will check for the presence of this data when importing the API definition, and will override the default values accordingly.

It's not necessary to specify and override all three fields - it's possible to provide just one or two.

Dynamic Data Import

Dynamic data import allows Backstage entity data to be updated quickly after it is changed in the Tyk Dashboard. This is an improvement on the schedule-based approach.

When the router option is enabled in the entity provider config, endpoints are set up in Backstage that enable the data import process to be triggered remotely by the Dashboard. To do this, the Dashboard sends a webhook request to Backstage when it detects a data change, which triggers the data import process.

Endpoint Paths

The Backstage endpoints are based on the name of the Dashboard in the Backstage configuration, for example:

/api/catalog/tyk/development/sync

Here development is the name given to the Dashboard in the Backstage configuration. Since the name is unique, each dashboard configuration is assigned its own endpoint. The name is the only part of the path to change, the rest remains the same across all dashboard configurations.

Tyk Dashboard Organisation Configuration

To configure the Tyk Dashboard to make the webhook request, the Tyk organisation object needs to be provided with the Backstage URL.

To do this, update your Tyk organisation JSON object via the Dashboard Admin API. In the organisation JSON, add an api_event object to the event_options section. For example:

{
  "api_event": {
    "webhook": "http://my-backstage-backend:7007/api/catalog/tyk/development/sync",
    "email": "",
    "redis": false
  }
}

Make sure that: 1. The webhook URL resolves to the Backstage backend deployment from the Tyk Dashboard. 2. The webhook URL path uses the dashboard configuration name specified in the Backstage app-config.yaml - the example value development is based on the example above.

Logging

Troubleshooting

If the entity provider encounters a problem it will log warnings and errors in the Backstage backend application log.

To increase the logging verbosity, set the log level to debug. For example, using yarn:

LOG_LEVEL=debug yarn start-backend

Setting LOG_LEVEL to debug won't display additional warning or error messages, as these are normally always displayed. Nevertheless, the additional debug information may be useful for troubleshooting.

Sequence Diagrams

Entity Provider Initialisation

How the Backstage catalog initialises Tyk entity providers:

sequenceDiagram
    participant ca as Catalog
    participant co as Backstage Configuration
    participant ep as Tyk Entity Provider
    participant td as Tyk Dashboard
    ca->>co: Read entity provider configuration
    co-->>ca: Entity provider configuration
    loop Create each entity provider defined in configuration
        ca->>ep: Entity provider configuration
        ep-->>ca: Entity provider
        ca->>ca: Add entity provider to processing engine
    end
    loop Initialise each entity provider
        ca->>ep: Scheduler and router
        ep->>ep: Setup schedule and routes
        ep->>td: Get Tyk data
        Note over ep: Initial synchronisation
        td-->>ep: Tyk data
        ep->>ep: Convert Tyk data into entities
        ep-)ca: Tyk entities
    end

Data Import Process

How the Tyk entity provider imports data from a Tyk dashboard into the Backstage catalog:

sequenceDiagram
    participant ep as Tyk Entity Provider
    participant td as Tyk Dashboard
    participant ca as Catalog
    ep->>ep: Generate dashboard entity based on provided config
    ep->>td: Get API data
    td-->>ep: API data
    loop Process APIs defined in API data
        ep->>ep: Convert API data into entity
    end
    ep->>td: Get system data
    td-->>ep: System data
    loop Process gateways defined in system data
        ep->>td: Get gateway data
        td-->>ep: Gateway data
        ep->>ep: Convert gateway data into entity
    end
    ep-->>ep: Generate relationships between API and gateway entities based on tags
    ep-)ca: Tyk entities

Operation of Schedule-Based Data Import

How the Backstage scheduler triggers the Tyk entity provider:

sequenceDiagram
    participant ts as Task Scheduler
    participant ep as Entity Provider
    participant td as Tyk Dashboard
    participant ca as Catalog
    ts->>ts: Interval event occurs, based on interval 
    ts-)ep: Trigger synchronisation
    ep->>td: Get Tyk data
    td-->>ep: Tyk data
    ep->>ep: Convert Tyk data into entities
    ep-)ca: Tyk entities

Operation of Router-Based Data Import

How the Backstage router triggers the Tyk entity provider:

sequenceDiagram
    participant td as Tyk Dashboard
    participant ro as Router
    participant ep as Entity Provider
    participant ca as Catalog
    td->>ro: Event payload, triggered by data change in dashboard
    ro-)ep: Trigger synchronisation
    ep->>td: Get Tyk data
    td-->>ep: Tyk data
    ep->>ep: Convert Tyk data into entities
    ep-)ca: Tyk entities
    ro-->>td: Status code