@tunnelhub/sdk v2.6.1
TunnelHub SDK
TunnelHub SDK is a robust TypeScript library for implementing automated integrations with logging and tracing capabilities on the TunnelHub platform.
Overview
The SDK provides a foundation for building reliable data integrations with features including:
- Delta detection and synchronization
- Batch processing capabilities
- Comprehensive logging and monitoring
- Error handling and retry mechanisms
- AWS infrastructure integration
Installation
npm install @tunnelhub/sdk
Core Concepts
Integration Flows
The SDK provides three main types of integration flows:
Delta Integration Flow (
DeltaIntegrationFlow
)- Tracks changes between source and target systems
- Handles insert, update, and delete operations
- Maintains state between executions
- Best for synchronization scenarios
Batch Delta Integration Flow (
BatchDeltaIntegrationFlow
)- Extends Delta Integration Flow
- Processes items in configurable batch sizes
- Optimized for large datasets
- Supports bulk operations
No Delta Integration Flow (
NoDeltaIntegrationFlow
)- Simple one-way data transfer
- No state tracking between executions
- Available in single and batch variants
- Ideal for one-time or streaming data transfers
Key Components
Automation Logs
The SDK automatically handles logging through the AutomationLog
class, capturing:
- Operation type (INSERT, UPDATE, DELETE, NODELTA, TRANSFER)
- Status (SUCCESS, FAIL, NEUTRAL)
- Timestamps
- Detailed error messages
- Operation payloads
Delta Tracking
AutomationDelta
manages state between executions:
- Stores previous execution state
- Enables change detection
- Persists in both DynamoDB and S3
- Handles large datasets efficiently
Parameters & Configuration
- Environment-specific configurations
- Custom parameter management
- Secure credential storage
- System connection details
Usage Examples
Creating a Delta Integration
class MyDeltaIntegration extends DeltaIntegrationFlow<MyDataType> {
protected async loadSourceSystemData(): Promise<MyDataType[]> {
// Implement source system data loading
}
protected async loadTargetSystemData(): Promise<MyDataType[]> {
// Implement target system data loading
}
protected async insertAction(item: MyDataType): Promise<IntegrationMessageReturn> {
// Implement insert logic
}
protected async updateAction(oldItem: MyDataType, newItem: MyDataType): Promise<IntegrationMessageReturn> {
// Implement update logic
}
protected async deleteAction(item: MyDataType): Promise<IntegrationMessageReturn> {
// Implement delete logic
}
protected defineMetadata(): Array<Metadata> {
return [
{
fieldName: 'id',
fieldLabel: 'ID',
fieldType: 'TEXT'
}
// Add more metadata fields
];
}
}
Creating a Batch Integration
class MyBatchIntegration extends BatchDeltaIntegrationFlow<MyDataType> {
constructor(event: ProcessorPayload, context?: LambdaContext) {
super(event, ['id'], ['name', 'value'], context);
this.packageSize = 100; // Set batch size
}
protected async batchInsertAction(items: MyDataType[]): Promise<IntegrationMessageReturnBatch[]> {
// Implement batch insert logic
}
protected async batchUpdateAction(oldItems: MyDataType[], newItems: MyDataType[]): Promise<IntegrationMessageReturnBatch[]> {
// Implement batch update logic
}
protected async batchDeleteAction(items: MyDataType[]): Promise<IntegrationMessageReturnBatch[]> {
// Implement batch delete logic
}
}
Additional Features
Data Store
- Conversion table management
- Sequence generation
- System configuration storage
API Integration
- Built-in middleware for API Gateway
- Request/response logging
- Error handling
AWS Integration
- DynamoDB integration
- S3 storage
- Lambda context handling
- ECS task tracking
Best Practices
Error Handling
- Implement proper try-catch blocks
- Use appropriate error statuses
- Provide meaningful error messages
Performance
- Use batch operations for large datasets
- Implement proper indexing in database queries
- Monitor memory usage
Logging
- Include relevant context in logs
- Use appropriate log levels
- Monitor execution statistics
Delta Management
- Choose appropriate key fields
- Implement proper change detection
- Handle data consistency
Environment Variables
The following environment variables are required only when using the API integration features:
TH_TENANT_ID
: Tenant identifierTH_ENVIRONMENT_ID
: Environment identifierTH_EXPIRATION_PERIOD
: Log retention period in days
Contributing
Please read our Contributing Guidelines for details on submitting pull requests.
License
This project is licensed under the MIT License - see the LICENSE file for details.
Support
For detailed documentation, visit https://docs.tunnelhub.io
For support queries, contact support@tunnelhub.io
7 months ago
7 months ago
8 months ago
8 months ago
8 months ago
9 months ago
9 months ago
9 months ago
9 months ago
9 months ago
9 months ago
9 months ago
9 months ago
9 months ago
9 months ago
9 months ago
9 months ago
9 months ago
9 months ago
9 months ago
9 months ago
9 months ago
9 months ago
9 months ago
9 months ago
9 months ago
9 months ago
9 months ago
10 months ago
10 months ago
10 months ago
10 months ago
10 months ago
10 months ago
10 months ago
10 months ago
10 months ago
10 months ago
10 months ago
10 months ago
10 months ago
10 months ago
10 months ago
10 months ago
10 months ago
10 months ago
10 months ago
10 months ago
10 months ago
10 months ago
10 months ago
10 months ago
10 months ago
10 months ago
10 months ago
10 months ago
10 months ago
10 months ago
10 months ago
10 months ago
10 months ago
10 months ago
10 months ago
10 months ago
10 months ago
10 months ago
10 months ago
10 months ago
10 months ago
10 months ago
10 months ago
10 months ago
10 months ago
10 months ago
10 months ago
10 months ago
10 months ago
10 months ago
10 months ago
10 months ago
1 year ago
1 year ago
1 year ago
1 year ago
1 year ago
1 year ago
2 years ago
2 years ago
2 years ago
2 years ago
2 years ago
2 years ago
2 years ago
2 years ago
2 years ago
2 years ago
2 years ago
2 years ago
2 years ago
2 years ago
2 years ago
2 years ago
2 years ago
2 years ago
2 years ago
2 years ago
2 years ago
2 years ago
2 years ago
2 years ago
2 years ago
2 years ago
2 years ago
2 years ago
2 years ago
2 years ago
2 years ago
2 years ago
2 years ago
2 years ago
2 years ago
2 years ago
2 years ago
2 years ago
2 years ago
2 years ago
2 years ago
2 years ago
2 years ago
3 years ago
3 years ago