@civic/storage NPM

storage

The storage adapter system allows different pieces of functionality to be composable together to make storing complex data easier.

Usage

A simple example, using local storage:

import { LocalStorageAdapter } from "@civic/storage";

const localStorage = new LocalStorageAdapter();
await localStorage.init();

await localStorage.set("contentKey", "All the content you want to store");
const stringContent = await localStorage.get("contentKey");

More complex example using local storage and Schema validation

import { LocalStorageAdapter, CompositeAdapter, ValidationSchemaAdapter } from "@civic/storage";

// uses jsonschema validation
const validationAdapter = new ValidationSchemaAdapter(
  await import("schema.json"),
  new JsonSerializerAdapter(new LocalStorageAdapter())
);
await validationAdapter.init();

await validationAdapter.set("contentKey", {
  "message": "All the content you want to store"
});
const objectContent = await validationAdapter.get("contentKey");

Storage Adapter design

The storage adapter was designed so you can easily extend it and plug your adapters from data manipulation to storage driver. Even though the adapter is responsible for normalization of data across multiple storage locations, there are some gaps in the current implementation. For example, the list of wallets is not de-duplicated if they are stored in both ipfs and the DID document. These improvements would be made with ticket CM-1857 as starting point.

Interface

interface StorageAdapter<DocumentType, ReceiptType> {
  init(): Promise<void>;
  set(contentId: string, content: DocumentType): Promise<ReceiptType>;
  get(contentId: string): Promise<DocumentType | null>;
  has(contentId: string): Promise<boolean>;
}

Blank diagram - Page 1 (5)

StorageDriver

Storage drivers are the final Adapter in the chain, It communicates to third parties to store or retrieve data. examples of drivers are IPFS, LocalStorage.

StorageTransformer

Storage transformers encode or change the contents of strings, they are used for encryption or base64 encoding/decoding.

StorageSerialiser

The Storage serializers encode an object into a string or decode a string into an object. examples of this would be JSON decoding/encoding.

StorageObjectActor

Storage object actors act on objects before or after they’ve been encoded/decoded, they validate objects, log out details, or cache objects to compare when writing or reading. Storage object actors should also be able to wrap storage composers.

StorageComposer

Object composers can contain routes between object adapters, they can fragment a larger document using JsonPaths, or multiplex the same documents to different adapters.

CivicMeAdapter

The CivicMeAdapter provides the adapter used on Civic.me frontend. It includes:

data encryption using Lexi.
Attach adapters to the DID.

The storageAdapter used on the CivicMeAdapter are created through the storageAdapterFactory. If the adapter setting is encrypted, the storageAdapter will be inside an EncryptionAdapter using Lexi for encription.

Blank diagram - Page 1 (3)

Example

const storageAdapter = new CivicMeAdapter(
  didDocument,
  {
    localProfile: {
      didAware: false,
      encrypted: true,
      locationType: LocationTypes.LOCAL_STORAGE,
    },
    privateRemoteProfile: {
      didAware: {
        fragmentId: "privateRemoteProfile",
        urlPrefix: "ipfs://",
      },
      encrypted: true,
      locationType: LocationTypes.IPFS,
    },
  },
  {
    "$.localProfile": "localProfile",
    "$.privateRemoteProfile": "privateRemoteProfile",
  },
  {
    did: didDocument.id,
    signMessage,
  }
);

await storageAdapter.init();

API

constructor

constructor(
    didDocument: DIDDocument,
    adapterSettings: Record<string, AdapterFactoryInitData>,
    jsonPaths: Record<string, string>,
    globalEncryptionSettings: EncryptionAdapterSettings | null = null
)

didDocument: The did document object.

adapterSettings: The settings for the adapters that will be used on the store. Each adapter has its identifier and own settings.

jsonPaths: The paths used to index the data.

globalEncryptionSettings: Optional encryption settings.

Adapters have a key, the key is used to identify individual adapter instances. In the case of adapters that are attached to the DID, the key is used as the fragment ID, for local storage the key is the content item ID.

When loading the adapters an index is created that stores the adapters against the keys.

DID attached adapters are loaded into a list, when they are used a transaction is generated when an update is made.

get

  get(contentPath: string): Promise<Record<string, unknown> | null>

contentPath: The path of the content to be retrieved.

When getting the content all the content is initially fetched, the fetching is done by passing the key into each instance of the adapter. By doing so the adapter can know where to fetch the content. For DID attached adapters the service endpoint is first checked and the content location is resolved against the DID fragment ID. All the content is fetched and merged into one document.

set

  set(
    contentPath: string,
    content: Record<string, unknown>
  ): Promise<{
    results: Record<string, string>;
    didDocument: DIDDocument;
    transactionInstructions: ((
      connection: Connection
    ) => Promise<TransactionInstruction>)[];
  }> {

contentPath: The path where the content will be set.

content: The data object.

When set is called, the stored JSON paths are ordered by JSON path part length, the part of the document is copied out into a other documents linked adapters, that part of the document is deleted from the clone document. the split of documents are passed to their adapters set method, the returned content id is then mapped to the key of the adapter for further use, ex. used for generating transactions for DID attached adapters.

SchemaMigrationAdapter

The schema migration adapter provides means to migrate from pre-defined schema versions through a migration function. This adapter guarantees the data retrieved and storage is always using the most recent schema.

Example

  new SchemaMigrationAdapter(
      [v1Schema as Schema, v2Schema as Schema],
      (_oldVersion, oldData) => {
        const typedOldData = oldData as unknown as CreatorProfile;
        return {
          localProfile: {
            name: {
              value:
                typedOldData?.localProfile?.name?.value ??
                typedOldData?.localProfile?.name ??
                "",
              status: ProfilePropertySyncStatus.Private,
            },
            image: {
              value:
                typedOldData?.localProfile?.image?.value ??
                typedOldData?.localProfile?.image ??
                "",
              status: ProfilePropertySyncStatus.Private,
            },
          },
          privateRemoteProfile: {
            name: {
              value: typedOldData?.privateRemoteProfile?.name ?? "",
              status: ProfilePropertySyncStatus.Private,
            },
            image: {
              value: typedOldData?.privateRemoteProfile?.image ?? "",
              status: ProfilePropertySyncStatus.Private,
            },
          },
        };
      },
      civicMeStorageAdapter
    )

Types

AdapterFactoryInitData `type`

export type AdapterFactoryInitData = {
  didAware:
    | false
    | {
        fragmentId: string;
        urlPrefix: string;
      };
  encrypted: boolean | EncryptionAdapterSettings;
  locationType: LocationTypes;
};

EncryptionAdapterSettings `type`

export type EncryptionAdapterSettings = {
  // the method used to sign the message
  signMessage(messageToSign: Uint8Array): Promise<Uint8Array>;
  // the DID
  did: string;
  // a string used to signing the message. if empty, a random string will be
  // used
  publicSigningString?: string;
};