@tunnelhub/sdk NPM

TunnelHub SDK

TunnelHub SDK is a robust TypeScript library for implementing automated integrations with logging and tracing capabilities on the TunnelHub platform.

Overview

The SDK provides a foundation for building reliable data integrations with features including:

Delta detection and synchronization
Batch processing capabilities
Comprehensive logging and monitoring
Error handling and retry mechanisms
AWS infrastructure integration

Installation

npm install @tunnelhub/sdk

Core Concepts

Integration Flows

The SDK provides three main types of integration flows:

Delta Integration Flow (DeltaIntegrationFlow)
- Tracks changes between source and target systems
- Handles insert, update, and delete operations
- Maintains state between executions
- Best for synchronization scenarios
Batch Delta Integration Flow (BatchDeltaIntegrationFlow)
- Extends Delta Integration Flow
- Processes items in configurable batch sizes
- Optimized for large datasets
- Supports bulk operations
No Delta Integration Flow (NoDeltaIntegrationFlow)
- Simple one-way data transfer
- No state tracking between executions
- Available in single and batch variants
- Ideal for one-time or streaming data transfers

Key Components

Automation Logs

The SDK automatically handles logging through the AutomationLog class, capturing:

Operation type (INSERT, UPDATE, DELETE, NODELTA, TRANSFER)
Status (SUCCESS, FAIL, NEUTRAL)
Timestamps
Detailed error messages
Operation payloads

Delta Tracking

AutomationDelta manages state between executions:

Stores previous execution state
Enables change detection
Persists in both DynamoDB and S3
Handles large datasets efficiently

Parameters & Configuration

Environment-specific configurations
Custom parameter management
Secure credential storage
System connection details

Usage Examples

Creating a Delta Integration

class MyDeltaIntegration extends DeltaIntegrationFlow<MyDataType> {
    protected async loadSourceSystemData(): Promise<MyDataType[]> {
        // Implement source system data loading
    }

    protected async loadTargetSystemData(): Promise<MyDataType[]> {
        // Implement target system data loading
    }

    protected async insertAction(item: MyDataType): Promise<IntegrationMessageReturn> {
        // Implement insert logic
    }

    protected async updateAction(oldItem: MyDataType, newItem: MyDataType): Promise<IntegrationMessageReturn> {
        // Implement update logic
    }

    protected async deleteAction(item: MyDataType): Promise<IntegrationMessageReturn> {
        // Implement delete logic
    }

    protected defineMetadata(): Array<Metadata> {
        return [
            {
                fieldName: 'id',
                fieldLabel: 'ID',
                fieldType: 'TEXT'
            }
            // Add more metadata fields
        ];
    }
}

Creating a Batch Integration

class MyBatchIntegration extends BatchDeltaIntegrationFlow<MyDataType> {
    constructor(event: ProcessorPayload, context?: LambdaContext) {
        super(event, ['id'], ['name', 'value'], context);
        this.packageSize = 100; // Set batch size
    }

    protected async batchInsertAction(items: MyDataType[]): Promise<IntegrationMessageReturnBatch[]> {
        // Implement batch insert logic
    }

    protected async batchUpdateAction(oldItems: MyDataType[], newItems: MyDataType[]): Promise<IntegrationMessageReturnBatch[]> {
        // Implement batch update logic
    }

    protected async batchDeleteAction(items: MyDataType[]): Promise<IntegrationMessageReturnBatch[]> {
        // Implement batch delete logic
    }
}

Additional Features

Data Store

Conversion table management
Sequence generation
System configuration storage

API Integration

Built-in middleware for API Gateway
Request/response logging
Error handling

AWS Integration

DynamoDB integration
S3 storage
Lambda context handling
ECS task tracking

Best Practices

Error Handling
- Implement proper try-catch blocks
- Use appropriate error statuses
- Provide meaningful error messages
Performance
- Use batch operations for large datasets
- Implement proper indexing in database queries
- Monitor memory usage
Logging
- Include relevant context in logs
- Use appropriate log levels
- Monitor execution statistics
Delta Management
- Choose appropriate key fields
- Implement proper change detection
- Handle data consistency