2.2.0 • Published 3 months ago

@x12i/xronox-storage-s3

Licence

MIT

Version

2.2.0

Deps

Size

101 kB

Vulns

Weekly

Summary Dependency Versions

@x12i/xronox-storage-s3

S3-compatible storage adapter for Xronox ecosystem - supports AWS S3, DigitalOcean Spaces, MinIO, Cloudflare R2, and other S3-compatible services.

Env-Ready Component (ERC 2.0)

This component supports zero-config initialization via environment variables using @x12i/env.

Quick Start (Zero-Config Mode)

# 1. Install the package
npm install @x12i/xronox-storage-s3 @aws-sdk/client-s3 @aws-sdk/s3-request-presigner

# 2. Copy .env.example to .env (auto-generated in package root)
cp node_modules/@x12i/xronox-storage-s3/.env.example .env

# 3. Fill in required values in .env
# S3_ENDPOINT=https://nyc3.digitaloceanspaces.com
# S3_REGION=nyc3
# S3_ACCESS_KEY_ID=your_access_key
# S3_SECRET_ACCESS_KEY=your_secret_key

# 4. Use with zero config!
import { S3Storage } from '@x12i/xronox-storage-s3';
const storage = new S3Storage(); // Auto-discovers from process.env

// Use it!
await storage.putJSON('my-bucket', 'data/item.json', { id: '123', name: 'Test' });
const data = await storage.getJSON('my-bucket', 'data/item.json');

Advanced Mode (Programmatic Configuration)

import { S3Storage, S3Client } from '@x12i/xronox-storage-s3';
import { S3Client as AWSS3Client } from '@aws-sdk/client-s3';

// Option 1: Explicit config
const storage = new S3Storage({
  config: {
    endpoint: 'https://custom-endpoint.com',
    region: 'us-east-1',
    accessKeyId: 'key',
    secretAccessKey: 'secret'
  }
});

// Option 2: Explicit S3Client
const s3Client = new AWSS3Client({
  endpoint: 'https://custom-endpoint.com',
  region: 'us-east-1',
  credentials: {
    accessKeyId: 'key',
    secretAccessKey: 'secret'
  }
});
const storage = new S3Storage({ s3Client });

Environment Variables

See .env.example (auto-generated in package root) for the complete list of required and optional variables with descriptions.

Required Variables:

S3_ENDPOINT - S3-compatible endpoint URL
S3_REGION - AWS region or provider region
S3_ACCESS_KEY_ID - S3 access key ID
S3_SECRET_ACCESS_KEY - S3 secret access key

Optional Variables:

S3_FORCE_PATH_STYLE - Force path-style URLs (auto-detected if not provided)
S3_BUCKET - Default bucket name

ERC 2.0 Compliance

Auto-discovers configuration from environment variables
Type-safe with automatic coercion and validation
All dependency requirements documented
Transitive requirements automatically merged

Dependencies:

@x12i/env (Configuration engine)
@aws-sdk/client-s3 (peer dependency, non-ERC - must be installed separately)
@aws-sdk/s3-request-presigner (peer dependency, non-ERC - must be installed separately)

Multi-Provider Support - Works with AWS S3, DigitalOcean Spaces, MinIO, Cloudflare R2
JSON & Binary Storage - Automatic serialization for JSON, raw buffer support
Data Integrity - SHA-256 verification for all uploads
Presigned URLs - Generate time-limited access URLs
Stream Handling - Safe conversion of streams to buffers
Error Handling - Clear, actionable error messages
Structured Logging - Built-in logging via @x12i/logxer
Configuration Management - nxconfig integration with ENV token resolution
TypeScript - Full type safety and IntelliSense support

Installation

npm install @x12i/xronox-storage-s3 @aws-sdk/client-s3 @aws-sdk/s3-request-presigner

Automatic Provider Detection

This package automatically detects S3-compatible providers and sets optimal configuration:

import { createS3Client, detectProvider } from '@x12i/xronox-storage-s3';

// Detect provider information
const provider = detectProvider('https://nyc3.digitaloceanspaces.com');
console.log(provider.provider); // 'digitalocean'
console.log(provider.forcePathStyle); // true

Supported Providers:

AWS S3 - Virtual-hosted-style URLs
DigitalOcean Spaces - Automatic path-style detection
MinIO - Automatic path-style detection
Cloudflare R2 - Automatic configuration

Usage Examples

Using ERC-Compliant S3Storage Class (Recommended)

import { S3Storage } from '@x12i/xronox-storage-s3';

// Zero-config initialization (reads from process.env)
const storage = new S3Storage();

// Store JSON data
await storage.putJSON('my-bucket', 'data/item.json', { id: '123', name: 'Test' });

// Retrieve JSON data
const data = await storage.getJSON('my-bucket', 'data/item.json');

// Store raw binary data
await storage.putRaw('my-bucket', 'file.bin', Buffer.from('binary data'));

// Get raw data
const buffer = await storage.getRaw('my-bucket', 'file.bin');

// Check if object exists
const exists = await storage.head('my-bucket', 'data/item.json');
console.log(exists.exists); // true/false

// List objects
const list = await storage.list('my-bucket', 'data/');
console.log(list.keys); // ['data/item.json', 'data/other.json']

// Generate presigned URL
const url = await storage.presignGet('my-bucket', 'data/item.json', 3600);

Using S3 Operations Directly

import { S3Client } from '@aws-sdk/client-s3';
import { putJSON, getJSON } from '@x12i/xronox-storage-s3';

// Create S3 client
const s3 = new S3Client({
  endpoint: 'https://nyc3.digitaloceanspaces.com',
  region: 'nyc3',
  credentials: {
    accessKeyId: process.env.SPACE_ACCESS_KEY!,
    secretAccessKey: process.env.SPACE_SECRET_KEY!
  },
  forcePathStyle: false
});

// Store JSON data
const data = { id: '123', name: 'Test Item' };
const result = await putJSON(s3, 'my-bucket', 'data/item.json', data);
console.log('Stored:', result.sha256, 'Size:', result.size);

// Retrieve JSON data
const retrieved = await getJSON(s3, 'my-bucket', 'data/item.json');
console.log('Retrieved:', retrieved);

Using with nxconfig

import { autoLoadConfig } from '@nx-intelligence/nxconfig';
import { S3Client } from '@aws-sdk/client-s3';
import { putJSON, getJSON } from '@x12i/xronox-storage-s3';

// Load configuration
const { config } = await autoLoadConfig();
const s3Config = config.spacesConnections.default;

// Create S3 client from config
const s3 = new S3Client({
  endpoint: s3Config.endpoint,
  region: s3Config.region,
  credentials: {
    accessKeyId: s3Config.accessKey,
    secretAccessKey: s3Config.secretKey
  },
  forcePathStyle: s3Config.forcePathStyle
});

// Use S3 operations
await putRecords(s3, 'bucket', 'key.json', { data: 'value' });

Configuration File (xronox-config.json)

{
  "xronox": {
    "spacesConnections": {
      "default": {
        "endpoint": "ENV.SPACE_ENDPOINT",
        "region": "ENV.SPACE_REGION",
        "accessKey": "ENV.SPACE_ACCESS_KEY",
        "secretKey": "ENV.SPACE_SECRET_KEY",
        "forcePathStyle": false
      }
    }
  }
}

Environment Variables (.env)

S3_ENDPOINT=https://nyc3.digitaloceanspaces.com
S3_REGION=nyc3
S3_ACCESS_KEY=your_access_key
S3_SECRET_KEY=your_secret_key

# Optional: Configure logging
XRONOX_STORAGE_LOG_LEVEL=info
XRONOX_STORAGE_LOG_FORMAT=json

const result = await putRecords(s3, 'bucket', 'path/file.json', {
  id: '123',
  data: 'value'
});
// Returns: { size: number, sha256: string, etag?: string }

getRecords(s3, bucket, key)

Retrieve records data from S3.

const data = await getRecords(s3, 'bucket', 'path/file.json');
// Returns: Record<string, unknown>

Raw Data Operations

putRaw(s3, bucket, key, body, contentType?)

Store raw binary data in S3.

const buffer = Buffer.from('raw data');
const result = await putRaw(s3, 'bucket', 'path/file.bin', buffer, 'application/octet-stream');
// Returns: { size: number, sha256: string, etag?: string }

get(s3, bucket, key)

Retrieve raw data as Buffer.

const buffer = await get(s3, 'bucket', 'path/file.bin');
// Returns: Buffer

Metadata Operations

head(s3, bucket, key)

Get object metadata without downloading.

const metadata = await head(s3, 'bucket', 'path/file.json');
// Returns: { contentLength?: number, contentType?: string, lastModified?: Date, etag?: string, metadata?: Record<string, string> }

exists(s3, bucket, key)

Check if an object exists.

const fileExists = await exists(s3, 'bucket', 'path/file.json');
// Returns: boolean

List Operations

list(s3, bucket, prefix, options?)

List objects with a prefix (async generator).

for await (const key of list(s3, 'bucket', 'path/')) {
  console.log('Found:', key);
}
// Yields: string (object keys)

Management Operations

del(s3, bucket, key)

Delete an object.

await del(s3, 'bucket', 'path/file.json');

copy(s3, bucket, sourceKey, destKey)

Copy an object within the same bucket.

await copy(s3, 'bucket', 'source.json', 'destination.json');

URL Operations

presignGet(s3, bucket, key, ttlSeconds?)

Generate presigned URL for direct access.

const url = await presignGet(s3, 'bucket', 'path/file.json', 3600);
// Returns: string (presigned URL, default: 1 hour expiration)

Provider Configuration Examples

AWS S3

const s3 = new S3Client({
  region: 'us-east-1',
  credentials: {
    accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!
  }
});

DigitalOcean Spaces

import { createS3Client } from '@x12i/xronox-storage-s3';

// Automatic provider detection - forcePathStyle is automatically set to true
const s3 = createS3Client({
  endpoint: 'https://nyc3.digitaloceanspaces.com',
  region: 'nyc3',
  credentials: {
    accessKeyId: process.env.DO_SPACES_KEY!,
    secretAccessKey: process.env.DO_SPACES_SECRET!
  }
  // forcePathStyle automatically detected and set to true
});

// Or explicitly set it
const s3 = new S3Client({
  endpoint: 'https://nyc3.digitaloceanspaces.com',
  region: 'nyc3',
  credentials: {
    accessKeyId: process.env.DO_SPACES_KEY!,
    secretAccessKey: process.env.DO_SPACES_SECRET!
  },
  forcePathStyle: true // Required for DigitalOcean Spaces compatibility
});

MinIO

import { createS3Client } from '@x12i/xronox-storage-s3';

// Automatic provider detection - forcePathStyle automatically detected
const s3 = createS3Client({
  endpoint: 'http://localhost:9000',
  region: 'us-east-1',
  credentials: {
    accessKeyId: process.env.MINIO_ACCESS_KEY!,
    secretAccessKey: process.env.MINIO_SECRET_KEY!
  }
  // forcePathStyle automatically detected and set to true
});

Cloudflare R2

const s3 = new S3Client({
  endpoint: `https://${accountId}.r2.cloudflarestorage.com`,
  region: 'auto',
  credentials: {
    accessKeyId: process.env.R2_ACCESS_KEY_ID!,
    secretAccessKey: process.env.R2_SECRET_ACCESS_KEY!
  }
});

Error Handling

All operations throw descriptive errors:

try {
  await getJSON(s3, 'bucket', 'missing.json');
} catch (error) {
  // Error: Object 'missing.json' not found in bucket 'bucket'.
  console.error(error.message);
}

Common errors:

NoSuchBucket - Bucket doesn't exist
NoSuchKey - Object not found
AccessDenied - Insufficient permissions
InvalidBucketName - Invalid bucket name format

Logging

Built-in structured logging via @x12i/logxer:

# Configure log level
export XRONOX_STORAGE_LOG_LEVEL=debug  # debug|info|warn|error

# Configure log format
export XRONOX_STORAGE_LOG_FORMAT=json  # text|json

# Enable file logging
export XRONOX_STORAGE_LOG_TO_FILE=true
export XRONOX_STORAGE_LOG_FILE=/var/log/xronox-storage.log

Example log output (JSON format):

{
  "timestamp": "2025-10-12T10:30:00.000Z",
  "package": "XRONOX_STORAGE_S3",
  "level": "INFO",
  "message": "putJSON completed successfully",
  "data": {
    "bucket": "my-bucket",
    "key": "data/item.json",
    "size": 1234,
    "sha256": "abc123...",
    "duration": 45
  }
}

Testing

This package includes comprehensive real-world testing in the peer demos package following Poiesis Rule 25 principles - no mocks, no JEST, only real S3 connections.

Running Demos

Demos for this package are maintained in the peer demos repository:

# Clone the demos repository
git clone https://github.com/xronoces/xronox-demos.git

# Navigate to storage-s3 demos
cd xronox-demos/packages/xronox-storage-s3

# Run demos
npm run demo:all

Demo Coverage

The demos package includes:

S3 Operations (10 demos) - Direct S3 operations
Adapter Operations (9 demos) - StorageAdapter interface
Integration Workflows (3 demos) - Complete scenarios

Evidence Capture

Every demo captures real I/O evidence:

io/input/ - What was sent to S3
io/output/ - What was received from S3
io/debug/demo.log - Complete execution log

See the demos repository for detailed testing documentation.

Development

# Install dependencies
npm install

# Build
npm run build

# Watch mode
npm run dev

# Clean
npm run clean

Package Boundaries

This Package Does

S3 operations (PUT, GET, DELETE, HEAD, LIST, COPY)
JSON serialization/deserialization
Stream to buffer conversion
SHA-256 hash computation
Presigned URL generation
Multi-provider S3 compatibility

This Package Does NOT

Bucket management (create/delete buckets)
Permission management (IAM policies)
Data transformation (compression, encryption)
Caching strategies
Multi-part uploads
Custom retry logic

Dependencies

nx-config2 (^2.5.0) - Configuration management and ERC 2.0 compliance
logs-gateway (^1.4.0) - Structured logging
@aws-sdk/client-s3 (peer, ^3.450.0) - AWS S3 SDK (non-ERC, external dependency)
@aws-sdk/s3-request-presigner (peer, ^3.450.0) - Presigned URLs (non-ERC, external dependency)

License

MIT

Contributing

This package follows Poiesis Development Methodology:

One expertise per component
Clear boundaries
Real tests with I/O evidence
Structured logging via logs-gateway

See docs/context.md for package context and boundaries.

Support

For issues and questions:

GitHub Issues: Report a bug
Documentation: See docs/ folder

Changelog

1.0.0 (2025-10-12)

Initial release
Full S3-compatible storage operations
Multi-provider support (AWS, DigitalOcean, MinIO, Cloudflare R2)
SHA-256 data integrity verification
Structured logging via logs-gateway
nxconfig integration

Keywords

xronox storage s3 aws spaces digitalocean minio cloudflare-r2 object-storage