@spotify/backstage-plugin-soundcheck-metrics-backend

Soundcheck Metrics Backend

A Backstage backend module that exports Soundcheck usage metrics to BigQuery for analytics and monitoring.

Overview

This module extends the Soundcheck plugin to automatically collect and export aggregate metrics on a configurable schedule. It provides visibility into Soundcheck adoption, check pass rates, track completion, and group-level performance across your Backstage environments.

Key Capabilities:

• Scheduled metrics collection and export to BigQuery
• Group-based metric aggregation using Soundcheck's GroupHierarchy
• Track-level pass/fail metrics (entities completing entire tracks)
• Environment context tracking (installation ID, customer detection)
• Robust error handling with per-row retry logic

Architecture

This is a backend module (not a standalone plugin) that extends the Soundcheck plugin:

• Module ID: soundcheck.metrics
• Shares the Soundcheck database (backstage_plugin_soundcheck)
• Integrates with Soundcheck's GroupHierarchyService
• Exports data to Google BigQuery via the @google-cloud/bigquery library

Installation

1. Install the Package

cd packages/backend
yarn add @spotify/backstage-plugin-soundcheck-metrics-backend

2. Register the Module

Add the module to your backend in packages/backend/src/index.ts:

import { soundcheckMetricsModule } from '@spotify/backstage-plugin-soundcheck-metrics-backend';

// Register with your backend
backend.add(soundcheckMetricsModule);

Important: The Soundcheck plugin must be installed and registered before this module.

3. Configure the Module

Add configuration to app-config.yaml:

soundcheckMetrics:
  enabled: true
  schedule: '0 2 * * *' # Daily at 2 AM UTC

  # Group types to aggregate metrics for
  groupTypesToAggregate:
    - squad
    - misison
    - product area
    - studio

  # BigQuery configuration
  bigquery:
    project: my-gcp-project
    dataset: backstage_metrics
    table: soundcheck_usage
    credentials: ${BIGQUERY_CREDENTIALS} # Optional: use ADC if omitted

Configuration Reference

Core Settings

Field	Type	Required	Default	Description
enabled	boolean	Yes	-	Enable/disable metrics export
schedule	string	No	'0 2 * * *'	Cron expression for export schedule
groupTypesToAggregate	string[]	No	[]	Group types to create separate metric rows for

BigQuery Settings

Field	Type	Required	Description
bigquery.project	string	Yes	GCP project ID containing the BigQuery dataset
bigquery.dataset	string	Yes	BigQuery dataset name
bigquery.table	string	Yes	BigQuery table name
bigquery.credentials	string	No	Service account JSON (omit to use Application Default Credentials)

Schedule Format

The schedule field uses standard cron format: minute hour day month weekday

Examples:

• '0 2 * * *' - Daily at 2 AM UTC (default)
• '0 */4 * * *' - Every 4 hours
• '*/15 * * * *' - Every 15 minutes
• '0 0 * * 0' - Weekly on Sunday at midnight
• '0 0 1 * *' - Monthly on the 1st at midnight

Task Timeout: The export task has a 10-minute timeout. If collection or export exceeds this limit, the task will be terminated and retried on the next scheduled run.

Group-Based Aggregation

Configure groupTypesToAggregate to export separate metric rows for each group of specified types:

soundcheckMetrics:
  groupTypesToAggregate:
    - squad # Metrics for each squad
    - mission # Metrics for each mission
    - studio # Metrics for each studio

How it works:

1. Queries GroupHierarchyService for all root groups of the specified types
2. For each group, collects metrics filtered to entities owned by that group (including descendant groups)
3. Exports a separate BigQuery row per group with:
   • group_name: Name extracted from the group entity ref (e.g., "toast-infra")
   • group_level: Group type (e.g., "squad", "mission")
   • Check and campaign counts filtered to entities owned by the group
   • Global counts for total campaigns, tracks, and checkers (not filtered by group)

If omitted: No metrics will be exported (the module requires at least one group type to be configured).

Authentication

Three methods for BigQuery authentication:

Option 1: Application Default Credentials (Recommended for GCP)

When running in GCP (Cloud Run, GKE), omit the credentials field:

soundcheckMetrics:
  bigquery:
    project: my-project
    dataset: metrics
    table: soundcheck_usage

The module will automatically use the service account attached to the workload.

Option 2: Service Account JSON (Recommended for Non-GCP)

Provide service account credentials as a JSON string:

soundcheckMetrics:
  bigquery:
    credentials: ${BIGQUERY_CREDENTIALS}

Set the environment variable:

export BIGQUERY_CREDENTIALS='{"type":"service_account","project_id":"...","private_key":"..."}'

Option 3: Credentials File (Development Only)

Reference a local credentials file:

soundcheckMetrics:
  bigquery:
    credentials: '${file:./credentials.json}'

BigQuery Setup

1. Create the Table

Run this SQL in the BigQuery Console or via bq CLI:

CREATE TABLE `my-project.backstage_metrics.soundcheck_usage` (
  -- Environment identification
  environment_name STRING,
  installation_id STRING,
  organization_name STRING,
  customer BOOL,

  -- Group identification
  group_name STRING,
  group_level STRING,

  -- Global counts (not filtered by group)
  total_campaigns INT64,
  total_tracks INT64,
  total_checkers INT64,

  -- Group-scoped metrics (filtered to entities owned by the group)
  group_total_checks INT64,
  group_passed_checks INT64,
  group_failed_checks INT64,
  group_active_campaigns INT64,
  group_archived_campaigns INT64,
  tracks_passed_by_entities_of_group INT64,
  tracks_failed_by_entities_of_group INT64,

  -- Timestamp
  snapshot_timestamp TIMESTAMP
)
PARTITION BY DATE(snapshot_timestamp)
CLUSTER BY environment_name, customer, group_level
OPTIONS(
  description="Soundcheck usage metrics exported from Backstage instances"
);

Schema Details

Environment Context:

• environment_name: Hostname prefix from app.baseUrl (e.g., "spc-acme-prod")
• installation_id: Value from K_SERVICE environment variable (or 'UNKNOWN')
• organization_name: From organization.name config
• customer: true if environment name starts with spc- (Spotify customer pattern)

Group Context:

• group_name: Group entity name (e.g., "toast-infra", "platform")
• group_level: Group type (e.g., "squad", "mission", "studio")

Global Metrics (not filtered by group):

• total_campaigns: Total campaigns in the Soundcheck database
• total_tracks: Total tracks/programs in the database
• total_checkers: Total checker definitions in the database

Group-Scoped Metrics (filtered to entities owned by the group):

• group_total_checks: Total check results for entities owned by this group
• group_passed_checks: Passed check results for group entities
• group_failed_checks: Failed check results for group entities
• group_active_campaigns: Non-archived campaigns for group entities
• group_archived_campaigns: Archived campaigns for group entities
• tracks_passed_by_entities_of_group: Track instances where a group entity passed all checks
• tracks_failed_by_entities_of_group: Track instances where a group entity failed any check

Timestamp:

• snapshot_timestamp: UTC timestamp when metrics were collected (truncated to date boundary)

Partitioning & Clustering

• Partitioned by date: Reduces query costs by scanning only relevant dates
• Clustered by: environment_name, customer, group_level for optimal query performance

2. Create a Service Account

# Create service account
gcloud iam service-accounts create soundcheck-metrics-exporter \
    --display-name="Soundcheck Metrics Exporter" \
    --project=my-project

# Grant BigQuery Data Editor role
gcloud projects add-iam-policy-binding my-project \
    --member="serviceAccount:soundcheck-metrics-exporter@my-project.iam.gserviceaccount.com" \
    --role="roles/bigquery.dataEditor"

# Create key (for non-GCP environments)
gcloud iam service-accounts keys create credentials.json \
    --iam-account=soundcheck-metrics-exporter@my-project.iam.gserviceaccount.com

Required Permissions:

• bigquery.tables.updateData (insert rows)
• bigquery.tables.get (verify table exists)

The roles/bigquery.dataEditor role includes both permissions.

3. Test the Configuration

Restart your Backstage backend and check logs for:

[SoundcheckMetricsBackend] - BigQuery exporter initialized { project: 'my-project', dataset: 'backstage_metrics', table: 'soundcheck_usage' }
[SoundcheckMetricsBackend] - Soundcheck metrics export module initialized! { schedule: '0 2 * * *' }

Metrics Reference

Track-Level Metrics

Track metrics measure how many track instances (entity + track combinations) resulted in all checks passing or any check failing.

Calculation Logic:

For each track in soundcheck_program:

1. Get all check IDs associated with the track from soundcheck_program_check_description
2. For each entity owned by the group:
   • Query check_result for results matching the entity and track's check IDs
   • Passed: Entity has results for ALL checks in the track, and ALL results are state='passed'
   • Failed: Entity has at least one result with state='failed' for the track's checks

Example:

Track "Security Baseline" has 3 checks: tls-check, vuln-scan, secrets-scan

Entity	TLS Check	Vuln Scan	Secrets Scan	Result
service-a	passed	passed	passed	Track passed (1)
service-b	passed	failed	passed	Track failed (1)
service-c	passed	(no result)	passed	Not counted (incomplete)

Result: tracks_passed_by_entities_of_group = 1, tracks_failed_by_entities_of_group = 1

Metrics Summary Table

Metric	Scope	Source	Description
total_campaigns	Global	soundcheck_campaign	Total campaigns across all entities
total_tracks	Global	soundcheck_program	Total tracks/programs defined
total_checkers	Global	checker	Total checker definitions
group_total_checks	Group	check_result filtered by group entities	Total check results for group entities
group_passed_checks	Group	check_result WHERE state='passed'	Passed checks for group entities
group_failed_checks	Group	check_result WHERE state='failed'	Failed checks for group entities
group_active_campaigns	Group	soundcheck_campaign WHERE archived=false	Active campaigns for group entities
group_archived_campaigns	Group	soundcheck_campaign WHERE archived=true	Archived campaigns for group entities
tracks_passed_by_entities_of_group	Group	Calculated from check_result + soundcheck_program_check_description	Track instances fully passed by group entities
tracks_failed_by_entities_of_group	Group	Calculated from check_result + soundcheck_program_check_description	Track instances failed by group entities

Example Queries

Latest Metrics Per Environment

SELECT
  environment_name,
  group_name,
  group_level,
  group_total_checks,
  group_passed_checks,
  group_failed_checks,
  ROUND(SAFE_DIVIDE(group_passed_checks, group_total_checks) * 100, 2) AS pass_rate_pct,
  tracks_passed_by_entities_of_group,
  tracks_failed_by_entities_of_group,
  snapshot_timestamp
FROM `my-project.backstage_metrics.soundcheck_usage`
WHERE DATE(snapshot_timestamp) = CURRENT_DATE()
ORDER BY environment_name, group_name;

Pass Rate Trend Over Time

SELECT
  DATE(snapshot_timestamp) AS date,
  environment_name,
  group_name,
  SUM(group_passed_checks) AS total_passed,
  SUM(group_failed_checks) AS total_failed,
  ROUND(SAFE_DIVIDE(SUM(group_passed_checks), SUM(group_total_checks)) * 100, 2) AS pass_rate_pct
FROM `my-project.backstage_metrics.soundcheck_usage`
WHERE snapshot_timestamp >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY)
GROUP BY date, environment_name, group_name
ORDER BY date DESC, environment_name, group_name;

Customer vs Internal Comparison

SELECT
  customer,
  COUNT(DISTINCT environment_name) AS num_environments,
  COUNT(DISTINCT group_name) AS num_groups,
  AVG(group_total_checks) AS avg_checks_per_group,
  AVG(SAFE_DIVIDE(group_passed_checks, group_total_checks)) AS avg_pass_rate,
  SUM(tracks_passed_by_entities_of_group) AS total_tracks_passed,
  SUM(tracks_failed_by_entities_of_group) AS total_tracks_failed
FROM `my-project.backstage_metrics.soundcheck_usage`
WHERE DATE(snapshot_timestamp) = CURRENT_DATE()
GROUP BY customer;

Top Performing Groups

SELECT
  group_level,
  group_name,
  environment_name,
  group_total_checks,
  group_passed_checks,
  ROUND(SAFE_DIVIDE(group_passed_checks, group_total_checks) * 100, 2) AS pass_rate_pct,
  tracks_passed_by_entities_of_group,
  tracks_failed_by_entities_of_group
FROM `my-project.backstage_metrics.soundcheck_usage`
WHERE DATE(snapshot_timestamp) = CURRENT_DATE()
  AND group_total_checks > 0
ORDER BY pass_rate_pct DESC
LIMIT 20;

Track Completion Rates

SELECT
  group_level,
  group_name,
  tracks_passed_by_entities_of_group,
  tracks_failed_by_entities_of_group,
  tracks_passed_by_entities_of_group + tracks_failed_by_entities_of_group AS total_track_instances,
  ROUND(
    SAFE_DIVIDE(
      tracks_passed_by_entities_of_group,
      tracks_passed_by_entities_of_group + tracks_failed_by_entities_of_group
    ) * 100,
    2
  ) AS track_completion_rate_pct
FROM `my-project.backstage_metrics.soundcheck_usage`
WHERE DATE(snapshot_timestamp) = CURRENT_DATE()
  AND (tracks_passed_by_entities_of_group + tracks_failed_by_entities_of_group) > 0
ORDER BY track_completion_rate_pct DESC;

Monitoring & Logs

All log messages are prefixed with [SoundcheckMetricsBackend] - for easy filtering.

Successful Export

[SoundcheckMetricsBackend] - Soundcheck metrics export module initialized! { schedule: '0 2 * * *' }
[SoundcheckMetricsBackend] - Starting scheduled Soundcheck metrics export
[SoundcheckMetricsBackend] - Collecting Soundcheck metrics...
[SoundcheckMetricsBackend] - Environment context retrieved { environmentName: 'spc-acme-prod', ... }
[SoundcheckMetricsBackend] - Collecting metrics for group types: squad, product area, studio, mission
[SoundcheckMetricsBackend] - Collecting metrics for group: platform-squad (type: squad)
[SoundcheckMetricsBackend] - Exporting metrics to BigQuery { environment: 'spc-acme-prod', ... }
[SoundcheckMetricsBackend] - Successfully exported metrics to BigQuery
[SoundcheckMetricsBackend] - Metrics collected successfully for 12 groups
[SoundcheckMetricsBackend] - Completed scheduled Soundcheck metrics export: 12 succeeded, 0 failed

Common Log Patterns

Module initialization:

[SoundcheckMetricsBackend] - BigQuery exporter initialized { project: '...', dataset: '...', table: '...' }
[SoundcheckMetricsBackend] - Soundcheck metrics export module initialized! { schedule: '...' }

Per-group collection:

[SoundcheckMetricsBackend] - Collecting metrics for group: toast-infra (type: squad)
[SoundcheckMetricsBackend] - Found 47 entities owned by toast-infra

Export summary:

[SoundcheckMetricsBackend] - Exporting 15 metrics row(s) to BigQuery
[SoundcheckMetricsBackend] - Completed scheduled Soundcheck metrics export: 15 succeeded, 0 failed

Troubleshooting

Module Not Initializing

Symptom: No initialization logs appear

Possible Causes:

1. soundcheckMetrics.enabled is not set to true
2. Configuration is missing or invalid
3. Module not registered in packages/backend/src/index.ts

Check:

[SoundcheckMetricsBackend] - Soundcheck metrics export is disabled.

Solution: Verify config and ensure enabled: true

No Metrics Exported

Symptom: Export completes but 0 rows inserted

Possible Causes:

1. groupTypesToAggregate is empty or contains invalid group types
2. No groups exist for the specified types
3. All groups failed to collect metrics

Check logs for:

[SoundcheckMetricsBackend] - No groups found for types: squad, product area, studio, mission

Solution:

• Verify groupTypesToAggregate contains valid group types
• Confirm groups exist in Soundcheck's GroupHierarchy
• Check that GroupHierarchyService is properly configured

BigQuery Insert Failures

Symptom: Errors during export

Error: no such field: tracks_passed_by_entities_of_group

Solution: The BigQuery table schema doesn't match the exported data structure. Run the ALTER TABLE commands:

ALTER TABLE `my-project.backstage_metrics.soundcheck_usage`
ADD COLUMN tracks_passed_by_entities_of_group INT64;

ALTER TABLE `my-project.backstage_metrics.soundcheck_usage`
ADD COLUMN tracks_failed_by_entities_of_group INT64;

Error: Not found: Table my-project:backstage_metrics.soundcheck_usage

Solution: Create the BigQuery table using the SQL in the "Create the Table" section.

Error: Permission 'bigquery.tables.updateData' denied

Solution: Grant the service account the BigQuery Data Editor role:

gcloud projects add-iam-policy-binding my-project \
    --member="serviceAccount:soundcheck-metrics-exporter@my-project.iam.gserviceaccount.com" \
    --role="roles/bigquery.dataEditor"

Missing Soundcheck Tables

Symptom: Warnings about table queries failing

Logs:

[SoundcheckMetricsBackend] - Failed to query table soundcheck_campaign
[SoundcheckMetricsBackend] - Failed to load track-check mappings

Behavior: The module uses defensive error handling. Missing tables result in 0 values for the corresponding metrics, but the export continues.

Solution:

• Verify Soundcheck plugin is installed and initialized
• Check that Soundcheck has created its database tables
• This is expected if Soundcheck is not fully configured in the environment

Authentication Errors

Error: Could not load the default credentials

Solution: Provide credentials via one of three methods:

1. Set soundcheckMetrics.bigquery.credentials with service account JSON
2. Use Application Default Credentials (when running in GCP)
3. Set GOOGLE_APPLICATION_CREDENTIALS environment variable to a credentials file path

Error: invalid_grant: Invalid JWT Signature

Solution: The service account key may be corrupted or incorrectly formatted. Regenerate the key:

gcloud iam service-accounts keys create new-credentials.json \
    --iam-account=soundcheck-metrics-exporter@my-project.iam.gserviceaccount.com

Partial Export Failures

Symptom: Some groups succeed, others fail

Logs:

[SoundcheckMetricsBackend] - Failed to export metrics for group platform-squad { error: ... }
[SoundcheckMetricsBackend] - Completed scheduled Soundcheck metrics export: 10 succeeded, 2 failed

Behavior: The module uses per-row error handling. Individual group failures don't prevent other groups from being exported.

Solution: Check the error details in the logs to diagnose the specific group failure (e.g., entity ownership query failure, BigQuery validation error).

Development

Running Locally

1. Set up local BigQuery credentials:

export BIGQUERY_CREDENTIALS=$(cat credentials.json)

1. Configure app-config.local.yaml:

soundcheckMetrics:
  enabled: true
  schedule: '*/5 * * * *' # Every 5 minutes for testing
  groupTypesToAggregate:
    - squad
  bigquery:
    project: my-dev-project
    dataset: dev_metrics
    table: soundcheck_usage
    credentials: ${BIGQUERY_CREDENTIALS}

1. Start the backend:

yarn workspace backend start

Testing Metrics Collection

To test without waiting for the scheduled run, modify the schedule to run frequently:

schedule: '*/1 * * * *' # Every minute

Or trigger manually by restarting the backend (the initial export runs immediately).

Debugging

Enable debug logs:

backend:
  logger:
    level: debug

Filter for module logs:

yarn workspace backend start | grep '\[SoundcheckMetricsBackend\]'

Adding New Metrics

1. Update the SoundcheckMetrics interface in src/types.ts
2. Update the metric collection logic in src/service/MetricsCollector.ts
3. Update the BigQuery table schema (ALTER TABLE or recreate)
4. Update this README with the new metric documentation

License

Copyright 2024 Spotify AB. All rights reserved.