8.0.3 • Published 2d ago

@travetto/model

Licence

MIT

Version

8.0.3

Deps

Vulns

Weekly

Stars

Summary Dependency Versions

Data Modeling Support

Datastore abstraction for core operations.

Install: @travetto/model

npm install @travetto/model

# or

yarn add @travetto/model

This module provides a set of contracts/interfaces to data model persistence, modification and retrieval. This module builds heavily upon the Schema, which is used for data model validation.

A Simple Model

A model can be simply defined by usage of the @Model decorator, which opts it into the Schema contracts, as well as making it available to the ModelRegistryIndex.

Code: Basic Structure

import { Model } from '@travetto/model';

@Model()
export class SampleModel {
  id: string;
  name: string;
  age: number;
}

Once the model is defined, it can be leveraged with any of the services that implement the various model storage contracts. These contracts allow for persisting and fetching of the associated model object.

Contracts

The module is mainly composed of contracts. The contracts define the expected interface for various model patterns. The primary contracts are Basic, CRUD, Expiry, Blob and Bulk.

Basic

All Data Modeling Support implementations, must honor the Basic contract to be able to participate in the model ecosystem. This contract represents the bare minimum for a model service.

Code: Basic Contract

export interface ModelBasicSupport<C = unknown> {
  /**
   * Id Source
   */
  idSource: ModelIdSource;

  /**
   * Get underlying client
   */
  get client(): C;

  /**
   * Get by Id
   * @param id The identifier of the document to retrieve
   * @throws {NotFoundError} When an item is not found
   */
  get<T extends ModelType>(cls: Class<T>, id: string): Promise<T>;

  /**
   * Create new item
   * @param item The document to create
   * @throws {ExistsError} When an item with the provided id already exists
   */
  create<T extends ModelType>(cls: Class<T>, item: OptionalId<T>): Promise<T>;

  /**
   * Delete an item
   * @param id The id of the document to delete
   * @throws {NotFoundError} When an item is not found
   */
  delete<T extends ModelType>(cls: Class<T>, id: string): Promise<void>;
}

CRUD

The CRUD contract, builds upon the basic contract, and is built around the idea of simple data retrieval and storage, to create a foundation for other services that need only basic support. The model extension in Authentication, is an example of a module that only needs create, read and delete, and so any implementation of Data Modeling Support that honors this contract, can be used with the Authentication model extension.

Code: Crud Contract

export interface ModelCrudSupport extends ModelBasicSupport {
  /**
   * Update an item
   * @param item The document to update.
   * @throws {NotFoundError} When an item is not found
   */
  update<T extends ModelType>(cls: Class<T>, item: T): Promise<T>;

  /**
   * Create or update an item
   * @param item The document to upsert
   * @param view The schema view to validate against
   */
  upsert<T extends ModelType>(cls: Class<T>, item: OptionalId<T>): Promise<T>;

  /**
   * Update partial, respecting only top level keys.
   *
   * When invoking this method, any top level keys that are null/undefined are treated as removals/deletes.  Any properties
   * that point to sub objects/arrays are treated as wholesale replacements.
   *
   * @param id The document identifier to update
   * @param item The document to partially update.
   * @param view The schema view to validate against
   * @throws {NotFoundError} When an item is not found
   */
  updatePartial<T extends ModelType>(cls: Class<T>, item: Partial<T> & { id: string }, view?: string): Promise<T>;

  /**
   * List all items of a collection, results returned in batches of items.
   *
   * Note: Batch size hint can be used to optimize batch size, but is not guaranteed.
   *
   * @param cls The class to list
   * @param options Options for listing
   */
  list<T extends ModelType>(cls: Class<T>, options?: ModelListOptions): AsyncIterable<T[]>;
}

The list operation returns batches of model records as an async stream. It also accepts listing options such as limit to cap how many records are produced, alongside other runtime controls such as abort signals and batch size hints.

Expiry

Certain implementations will also provide support for automatic Expiry of data at runtime. This is extremely useful for temporary data as, and is used in the Caching module for expiring data accordingly.

Code: Expiry Contract

export interface ModelExpirySupport extends ModelCrudSupport {
  /**
   * Delete all expired by class
   *
   * @returns Returns the number of documents expired
   */
  deleteExpired<T extends ModelType>(cls: Class<T>): Promise<number>;
}

Blob

Some implementations also allow for the ability to read/write binary data as Blob. Given that all implementations can store Base64 encoded data, the key differentiator here, is native support for streaming data, as well as being able to store binary data of significant sizes.

Code: Blob Contract

export interface ModelBlobSupport {
  /**
   * Upsert blob to storage
   * @param location The location of the blob
   * @param input The actual blob to write
   * @param metadata Additional metadata to store with the blob
   * @param overwrite Should we replace content if already found, defaults to true
   */
  upsertBlob(location: string, input: BinaryType, metadata?: BinaryMetadata, overwrite?: boolean): Promise<void>;

  /**
   * Get blob from storage
   * @param location The location of the blob
   */
  getBlob(location: string, range?: ByteRange): Promise<Blob>;

  /**
   * Get metadata for blob
   * @param location The location of the blob
   */
  getBlobMetadata(location: string): Promise<BinaryMetadata>;

  /**
   * Delete blob by location
   * @param location The location of the blob
   */
  deleteBlob(location: string): Promise<void>;

  /**
   * Update blob metadata
   * @param location The location of the blob
   * @param metadata The metadata to update
   */
  updateBlobMetadata(location: string, metadata: BinaryMetadata): Promise<void>;

  /**
   * Produces an externally usable URL for sharing limited read access to a specific resource.
   * If expiresIn is explicitly set to false, returns a direct/public URL.
   *
   * @param location The asset location to read from
   * @param expiresIn Expiry or false for public/direct URL
   */
  getBlobReadUrl?(location: string, expiresIn?: TimeSpan | false): Promise<string>;

  /**
   * Produces an externally usable URL for sharing allowing direct write access
   *
   * @param location The asset location to write to
   * @param metadata The metadata to associate with the final asset
   * @param expiresIn Expiry
   */
  getBlobWriteUrl?(location: string, metadata: BinaryMetadata, expiresIn?: TimeSpan): Promise<string>;
}

Bulk

Finally, there is support for Bulk operations. This is not to simply imply issuing many commands at in parallel, but implementation support for an atomic/bulk operation. This should allow for higher throughput on data ingest, and potentially for atomic support on transactions.

Code: Bulk Contract

export interface ModelBulkSupport extends ModelCrudSupport {
  processBulk<T extends ModelType>(cls: Class<T>, operations: BulkOperation<T>[]): Promise<BulkResponse>;
}

Declaration

Models are declared via the @Model decorator, which allows the system to know that this is a class that is compatible with the module. The only requirement for a model is the ModelType

Code: ModelType

export interface ModelType {
  /**
   * Unique identifier.
   *
   * If not provided, will be computed on create
   */
  id: string;
}

The id is the only required field for a model, as this is a hard requirement on naming and type. This may make using existing data models impossible if types other than strings are required. Additionally, the type field, is intended to record the base model type, but can be remapped. This is important to support polymorphism, not only in Data Modeling Support, but also in Schema.

Implementations

Service	Basic	CRUD	Indexed	Expiry	Blob	Bulk
DynamoDB Model Support	X	X	X	X
Elasticsearch Model Source	X	X	X	X		X
Firestore Model Support	X	X	X
MongoDB Model Support	X	X	X	X	X	X
Redis Model Support	X	X	X	X
S3 Model Support	X	X		X	X
SQL Model Service	X	X	X	X		X
Memory Model Support	X	X	X	X	X	X
File Model Support	X	X		X	X	X

Custom Model Service

In addition to the provided contracts, the module also provides common utilities and shared test suites. The common utilities are useful for repetitive functionality, that is unable to be shared due to not relying upon inheritance (this was an intentional design decision). This allows for all the Data Modeling Support implementations to completely own the functionality and also to be able to provide additional/unique functionality that goes beyond the interface. Memory Model Support serves as a great example of what a full featured implementation can look like.

To enforce that these contracts are honored, the module provides shared test suites to allow for custom implementations to ensure they are adhering to the contract's expected behavior.

Code: Memory Service Test Configuration

import { DependencyRegistryIndex } from '@travetto/di';
import { type Class, castTo, classConstruct, RuntimeError } from '@travetto/runtime';

import type { ModelType } from '../../src/types/model.ts';
import { ModelBulkUtil } from '../../src/util/bulk.ts';
import { ModelCrudUtil } from '../../src/util/crud.ts';
import { ModelSuite } from './suite.ts';

type ServiceClass = { serviceClass: { new (): unknown } };

@ModelSuite()
export abstract class BaseModelSuite<T> {
  static ifNot(pred: (svc: unknown) => boolean): (x: unknown) => Promise<boolean> {
    return async (x: unknown) => !pred(classConstruct(castTo<ServiceClass>(x).serviceClass));
  }

  serviceClass: Class<T>;
  configClass: Class;

  async getSize<U extends ModelType>(cls: Class<U>): Promise<number> {
    const svc = await this.service;
    if (ModelCrudUtil.isSupported(svc)) {
      let i = 0;
      for await (const batch of svc.list(cls)) {
        i += batch.length;
      }
      return i;
    } else {
      throw new RuntimeError(`Size is not supported for this service: ${this.serviceClass.name}`);
    }
  }

  async saveAll<M extends ModelType>(cls: Class<M>, items: M[]): Promise<number> {
    const svc = await this.service;
    if (ModelBulkUtil.isSupported(svc)) {
      const result = await svc.processBulk(
        cls,
        items.map(x => ({ insert: x }))
      );
      return result.counts.insert;
    } else if (ModelCrudUtil.isSupported(svc)) {
      const out: Promise<M>[] = [];
      for (const el of items) {
        out.push(svc.create(cls, el));
      }
      await Promise.all(out);
      return out.length;
    } else {
      throw new Error('Service does not support crud operations');
    }
  }

  get service(): Promise<T> {
    return DependencyRegistryIndex.getInstance(this.serviceClass);
  }

  async toArray<U>(src: AsyncIterable<U | U[]> | AsyncGenerator<U | U[]>): Promise<U[]> {
    const out: (U | U[])[] = [];
    for await (const el of src) {
      out.push(el);
    }
    return castTo(out.flat());
  }
}

CLI - model:export

The module provides the ability to generate an export of the model structure from all the various @Models within the application. This is useful for being able to generate the appropriate files to manually create the data schemas in production.

Terminal: Help for model:export

$ trv model:export --help

TypeError: Cannot redefine property: toJSON
    at Object.defineProperty (<anonymous>)
    at file://<workspace-root>/module/runtime/src/json.ts:17:8
    at ModuleJob.run (node:internal/modules/esm/module_job:569:25)
    at async node:internal/modules/esm/loader:650:26
    at async $Runtime.importFrom (<workspace-root>/module/runtime/src/context.ts:136:22)
    at async $Registry.#init (<workspace-root>/module/registry/src/registry.ts:88:9)
    at async ModelExportCommand.help (./support/base-command.ts:38:5)
    at async HelpUtil.getExtendedHelpMessage (<workspace-root>/module/cli/src/help.ts:121:30)
    at async HelpUtil.renderCommandHelp (<workspace-root>/module/cli/src/help.ts:170:11)
    at async ExecutionManager.getExecutionCommand (<workspace-root>/module/cli/src/execute.ts:49:22)
    at async ExecutionManager.run (<workspace-root>/module/cli/src/execute.ts:67:23)

TypeError: Cannot redefine property: toJSON
    at Object.defineProperty (<anonymous>)
    at file://<workspace-root>/module/runtime/src/json.ts:17:8
    at ModuleJob.run (node:internal/modules/esm/module_job:569:25)
    at async node:internal/modules/esm/loader:650:26
    at async $Runtime.importFrom (<workspace-root>/module/runtime/src/context.ts:136:22)
    at async $Registry.#init (<workspace-root>/module/registry/src/registry.ts:88:9)
    at async ModelExportCommand.help (./support/base-command.ts:38:5)
    at async HelpUtil.getExtendedHelpMessage (<workspace-root>/module/cli/src/help.ts:121:30)
    at async HelpUtil.renderCommandHelp (<workspace-root>/module/cli/src/help.ts:170:11)
    at async ExecutionManager.getExecutionCommand (<workspace-root>/module/cli/src/execute.ts:49:22)
    at async ExecutionManager.run (<workspace-root>/module/cli/src/execute.ts:67:23)

CLI - model:install

The module provides the ability to install all the various @Models within the application given the current configuration being targeted. This is useful for being able to prepare the datastore manually.

Terminal: Help for model:install

$ trv model:install --help

TypeError: Cannot redefine property: toJSON
    at Object.defineProperty (<anonymous>)
    at file://<workspace-root>/module/runtime/src/json.ts:17:8
    at ModuleJob.run (node:internal/modules/esm/module_job:569:25)
    at async node:internal/modules/esm/loader:650:26
    at async $Runtime.importFrom (<workspace-root>/module/runtime/src/context.ts:136:22)
    at async $Registry.#init (<workspace-root>/module/registry/src/registry.ts:88:9)
    at async ModelInstallCommand.help (./support/base-command.ts:38:5)
    at async HelpUtil.getExtendedHelpMessage (<workspace-root>/module/cli/src/help.ts:121:30)
    at async HelpUtil.renderCommandHelp (<workspace-root>/module/cli/src/help.ts:170:11)
    at async ExecutionManager.getExecutionCommand (<workspace-root>/module/cli/src/execute.ts:49:22)
    at async ExecutionManager.run (<workspace-root>/module/cli/src/execute.ts:67:23)

TypeError: Cannot redefine property: toJSON
    at Object.defineProperty (<anonymous>)
    at file://<workspace-root>/module/runtime/src/json.ts:17:8
    at ModuleJob.run (node:internal/modules/esm/module_job:569:25)
    at async node:internal/modules/esm/loader:650:26
    at async $Runtime.importFrom (<workspace-root>/module/runtime/src/context.ts:136:22)
    at async $Registry.#init (<workspace-root>/module/registry/src/registry.ts:88:9)
    at async ModelInstallCommand.help (./support/base-command.ts:38:5)
    at async HelpUtil.getExtendedHelpMessage (<workspace-root>/module/cli/src/help.ts:121:30)
    at async HelpUtil.renderCommandHelp (<workspace-root>/module/cli/src/help.ts:170:11)
    at async ExecutionManager.getExecutionCommand (<workspace-root>/module/cli/src/execute.ts:49:22)
    at async ExecutionManager.run (<workspace-root>/module/cli/src/execute.ts:67:23)

Keywords

datastore decorators schema travetto typescript