nx-remote-json

remote-json

A unified TypeScript SDK for reading, writing, and mirroring JSON objects across multiple backends: local filesystem (disk), S3-compatible object storage, and MongoDB.

Package Information

• Package: nx-remote-json
• Version: 1.0.2
• Runtime: Node.js 18+
• Language: TypeScript 5+

Features

• Unified API: Single interface for three different storage backends
• Type-safe: Full TypeScript support with comprehensive type definitions
• Flexible routing: Map logical routers to backend-specific locations
• Mirroring: Sync JSON objects between backends
• Atomic operations: Safe file writes and upsert operations

Installation

npm install nx-remote-json

Requirements

• Node.js 18+
• TypeScript 5+

Quick Start

import { 
  createRemoteJsonClient,
  type RemoteJsonConfig 
} from "nx-remote-json";

const config = {
  disk: {
    rootLocation: "/data/remote-json",
    defaultExtension: ".json",
    mapping: [
      { jsonRouter: "settings", folder: "settings" },
      { jsonRouter: "dashboards", folder: "dashboards" }
    ]
  }
};

const client = createRemoteJsonClient(config);

// Read
const settings = await client.read({
  mode: "disk",
  jsonRouter: "settings",
  jsonKey: "user-123"
});

// Write
await client.write({
  mode: "disk",
  jsonRouter: "settings",
  jsonKey: "user-123",
  payload: { theme: "dark", language: "en" }
});

Configuration

Disk Backend

{
  disk: {
    rootLocation: "/data/remote-json",  // Base directory
    defaultExtension: ".json",          // Optional, defaults to ".json"
    mapping: [
      { jsonRouter: "settings", folder: "settings" },
      { jsonRouter: "dashboards", folder: "dashboards" }
    ]
  }
}

Path resolution: rootLocation/folder/jsonKey.json

Storage Backend (S3-compatible)

{
  storage: {
    region: "eu-central-1",
    endpoint: "https://s3.example.com",
    forcePathStyle: true,
    accessKey: "ACCESS_KEY",
    secretKey: "SECRET_KEY",
    bucket: "my-remote-json",
    defaultExtension: ".json",
    mapping: [
      { jsonRouter: "settings", folder: "settings" },
      { jsonRouter: "dashboards", folder: "dashboards" }
    ]
  }
}

Object key resolution: folder/jsonKey.json

Database Backend (MongoDB)

{
  database: {
    mongoUri: "mongodb://user:pass@localhost:27017",
    mongoDb: "remote_json",
    mapping: [
      { jsonRouter: "settings", collectionName: "settings" },
      { jsonRouter: "dashboards", collectionName: "dashboards" }
    ],
    queryFilters: [
      {
        field: "router",
        operator: "eq",
        valueSource: "literal",
        literalValue: "settings"
      },
      {
        field: "key",
        operator: "eq",
        valueSource: "jsonKey",
        transform: "none"
      }
    ]
  }
}

Filter construction: Filters are built from queryFilters templates and combined with $and. At least one filter must use valueSource: "jsonKey".

Write behavior: Filter fields are automatically injected into documents during writes, ensuring the same jsonKey can always re-find the document.

Type Exports

The package exports the following TypeScript types:

import type {
  RemoteJsonConfig,        // Main configuration type
  RemoteJsonMode,          // "disk" | "storage" | "database"
  RemoteJsonClient,        // Client interface
  ReadRequest,             // Read operation request
  WriteRequest,            // Write operation request
  MirrorRequest,           // Mirror operation request
  MirrorResult,            // Mirror operation result
  DiskBackendConfig,       // Disk backend configuration
  StorageBackendConfig,    // Storage backend configuration
  DatabaseBackendConfig,   // Database backend configuration
  RemoteJsonError,         // Error class
  RemoteJsonErrorCode,     // Error code type
} from "nx-remote-json";

Important: The main configuration type is RemoteJsonConfig, not RemoteJsonClientConfig.

API Reference

read<T>(req: ReadRequest): Promise<T | null>

Reads a JSON object from the specified backend.

const data = await client.read<MyType>({
  mode: "disk",
  jsonRouter: "settings",
  jsonKey: "user-123"
});

if (data === null) {
  console.log("Not found");
}

write<T>(req: WriteRequest<T>): Promise<void>

Writes a JSON object to the specified backend.

await client.write({
  mode: "storage",
  jsonRouter: "dashboards",
  jsonKey: "team-A",
  payload: {
    title: "Team A Dashboard",
    widgets: []
  }
});

mirror(req: MirrorRequest): Promise<MirrorResult>

Mirrors JSON objects from a master backend to target backends.

// Mirror one object
const result = await client.mirror({
  masterMode: "disk",
  jsonRouter: "settings",
  jsonKey: "user-123",
  targetModes: ["storage", "database"],
  overwriteExisting: true
});

// Mirror all objects in a router
const result = await client.mirror({
  masterMode: "disk",
  jsonRouter: "settings",
  targetModes: ["storage"]
});

// Mirror all routers
const result = await client.mirror({
  masterMode: "disk",
  targetModes: ["storage", "database"]
});

Mirror options:

• masterMode: Source backend (required)
• jsonRouter: Optional, limits to specific router
• jsonKey: Optional, limits to specific key
• targetModes: Optional, defaults to all configured modes except master
• overwriteExisting: Optional, defaults to true

Result structure:

{
  items: [
    {
      jsonRouter: "settings",
      jsonKey: "user-123",
      sourceMode: "disk",
      targetMode: "storage",
      status: "created" | "updated" | "skipped" | "failed",
      error?: string
    }
  ]
}

Error Handling

All methods throw RemoteJsonError with the following error codes:

• CONFIG_ERROR: Configuration is invalid
• MAPPING_NOT_FOUND: No mapping found for the specified jsonRouter
• BACKEND_NOT_CONFIGURED: Requested backend is not configured
• SERIALIZATION_ERROR: Failed to serialize JSON payload
• DESERIALIZATION_ERROR: Failed to parse JSON response
• BACKEND_IO_ERROR: Backend I/O operation failed

import { RemoteJsonError } from "nx-remote-json";

try {
  await client.write({ ... });
} catch (err) {
  if (err instanceof RemoteJsonError) {
    console.error(`Error code: ${err.code}`);
    console.error(`Message: ${err.message}`);
    if (err.cause) {
      console.error(`Cause:`, err.cause);
    }
  }
}

Examples

Multi-backend Configuration

const config = {
  disk: {
    rootLocation: "./data",
    mapping: [
      { jsonRouter: "settings", folder: "settings" }
    ]
  },
  storage: {
    region: "us-east-1",
    endpoint: "https://s3.amazonaws.com",
    forcePathStyle: false,
    accessKey: process.env.AWS_ACCESS_KEY_ID!,
    secretKey: process.env.AWS_SECRET_ACCESS_KEY!,
    bucket: "my-bucket",
    mapping: [
      { jsonRouter: "settings", folder: "settings" }
    ]
  },
  database: {
    mongoUri: process.env.MONGODB_URI!,
    mongoDb: "app_db",
    mapping: [
      { jsonRouter: "settings", collectionName: "settings" }
    ],
    queryFilters: [
      {
        field: "key",
        operator: "eq",
        valueSource: "jsonKey"
      }
    ]
  }
};

const client = createRemoteJsonClient(config);

MongoDB with ObjectId

{
  database: {
    // ... other config
    queryFilters: [
      {
        field: "_id",
        operator: "eq",
        valueSource: "jsonKey",
        transform: "objectId"  // Converts jsonKey to ObjectId
      }
    ]
  }
}

Mirroring with Skip Existing

// Only create new objects, skip existing ones
const result = await client.mirror({
  masterMode: "disk",
  jsonRouter: "settings",
  overwriteExisting: false  // Skip if already exists
});

const created = result.items.filter(i => i.status === "created");
const skipped = result.items.filter(i => i.status === "skipped");

License

MIT