@kontent-ai-consulting/migration-toolkit v0.1.8

Kontent.ai Migration Toolkit

The purpose of this tool is to facilitate migration to Kontent.ai environment using a developer friendly abstraction layer where developers focus on data transformation from external systems rather then spending time on writing customized import scripts.

!TIP
This library provides the tools for developers to prepare the data for migration in designated format (json, csv) and import functionality (either in code or via CLI) to bring this over to your Kontent.ai environment.The tool takes care of preparing content items, language variants, moving items through workflow, publishing, archiving, uploading assets, retry policy and more.

This library can only be used in node.js. Use in Browsers is not supported.

Getting started

Use --help anytime to get information about available commands and their options.

npx @kontent-ai-consulting/migration-toolkit --help

# you can also install the package globally, or locally
npm i @kontent-ai-consulting/migration-toolkit -g

# with the package installed, you can call the tool as follows
kontent-ai-migration-toolkit --help

Migrating data from external / 3rd party systems

In order to import data into your Kontent.ai environment you first need to prepare the data you want to import and convert them into appropriate format (see more at https://github.com/Kontent-ai-consulting/kontent-ai-migration-toolkit?tab=readme-ov-file#migration-models).

You have 2 options to prepare your data. Either you use ExportToolkit to fetch your data from external system and store them on File system which you then use to import data. Or you can use ImportToolkit directly without storing anything on File system.

Option 1 - Use ExportToolkit

1. Prepare the desired structure (content types, taxonomies...) in your Kontent.ai environment
2. Write a custom adapter to download / fetch data from your system and convert it to appropriate formats defined by this library & your content model
3. Use ExportToolkit with your adapter which creates files on File system
4. Import the generated export files into a target environment using ImportToolkit.importFromFilesAsync in code or via CLI

An example script can be found here

Option 2 - Direct import

1. Prepare the desired structure (content types, taxonomies...) in your Kontent.ai environment
2. Write a code to download / fetch data from your system and convert it to appropriate formats defined by this library & your content model
3. Use ImportToolkit.importAsync function to directly import the data into your Kontent.ai environment

An example script can be found here

Import

!CAUTION
We do not recommended importing into a production environment directly (without proper testing). Instead you should first create a testing environment and run the script there to make sure everything works as you intended to.

!NOTE
When importing it is essential that used Content types, Taxonomies and Workflows are consistent with the ones defined in target environment. Any inconsistency in data such as referencing inexistent taxonomy term, incorrect element type and other problems will cause import to fail.

How are content items & language variants imported?

The Migration Toolkit creates content items that are not present in target project. If the content item exists in target project (based on item's codename) the item will be updated instead. The workflow of imported language variant will be set according to workflow_step and workflow fields.

You can run kontent-ai-migration-toolkit many times over without being worried that identical content items will be created multiple times. Same goes for assets if you use assetExternalId when migrating assets over.

How are assets imported?

If asset exists in target project, the asset upload will be skipped and not uploaded at all. If asset doesn't exist, the asset from the zip folder will be uploaded. It is highly recommended that you set assetExternalId when migrating assets as that will prevent duplicate assets from being created if you run the migration multiple times.

Import Configuration

Import CLI samples

# Import from zip:
kontent-ai-migration-toolkit --action=import --apiKey=xxx --environmentId=xxx --format=json --itemsFilename=items.zip --assetsFilename=assets.zip

# Import from json file:
kontent-ai-migration-toolkit --action=import --apiKey=xxx --environmentId=xxx --format=json --itemsFilename=items.json

# Import from csv file:
kontent-ai-migration-toolkit --action=import --apiKey=xxx --environmentId=xxx --format=csv --itemsFilename=items.csv

Importing in code

Examples of importing in code can be found at:

1. Import from json
2. Import from csv
3. Import from zip
4. Direct import without files

Migration models

This tools uses simplified models to make migration simpler and more developer friendly. This is useful especially when migrating from external systems because data structure is almost certainly very different to models used within Kontent.ai APIs.

Model definitions

Models are defined at https://github.com/Kontent-ai-consulting/kontent-ai-migration-toolkit/blob/main/lib/core/migration-models.ts

export type MigrationElementType =
    | 'text'
    | 'rich_text'
    | 'number'
    | 'multiple_choice'
    | 'date_time'
    | 'asset'
    | 'modular_content'
    | 'taxonomy'
    | 'url_slug'
    | 'guidelines'
    | 'snippet'
    | 'custom'
    | 'subpages';

export interface IMigrationElement {
    value: string | undefined | string[];
    type: MigrationElementType;
    codename: string;
}

export interface IMigrationItem {
    system: {
        codename: string;
        name: string;
        language: string;
        type: string;
        collection: string;

        // workflow_step & workflow is required for Content items, but optional for Component items
        workflow_step?: string;
        workflow?: string;
    };
    elements: IMigrationElement[];
}

export interface IMigrationReference {
    id?: string;
    codename?: string;
    external_id?: string;
}

export interface IMigrationAsset {
    _zipFilename: string;
    binaryData: Buffer | Blob | undefined;
    filename: string;
    title: string;

    assetId?: string;
    assetExternalId?: string;
    codename?: string;
    collection?: IMigrationReference;
    folder?: IMigrationReference;
}

Model examples

A single record of IMigrationItem type in json format may look like this:

{
    "system": {
        "codename": "_the_dark_knight_rises",
        "collection": "default",
        "language": "en",
        "name": " The Dark Knight Rises",
        "type": "movie",
        "workflow_step": "published",
        "workflow": "default"
    },
    "elements": {
        "title": "The Dark Knight Rises",
        "plot": "<p>Eight years after the Joker's reign of anarchy, the Dark Knight, with the help of the enigmatic Selina, is forced from his imposed exile to save Gotham City, now on the edge of total annihilation, from the brutal guerrilla terrorist Bane.</p>",
        "released": "2012-07-20T00:00:00Z",
        "length": 164,
        "poster": [
            "https://assets-eu-01.kc-usercontent.com:443/cdbf5823-cbec-010d-f4c3-0411eee31c0e/787c8c83-16b4-40b4-878e-b90b6d42a4ef/the_dark_knight_rises.jpg"
        ],
        "category": ["sci_fi", "action"],
        "stars": ["christian_bale", "anne_hathaway", "tom_hardy"],
        "seoname": "the-dark-knight-rises",
        "releasecategory": []
    }
}

You may find sample export (.zip) with both items & assets at https://github.com/Kontent-ai-consulting/kontent-ai-migration-toolkit/tree/main/samples/export-data

Export items & assets from Kontent.ai

There is a built-in kontentAi adapter that can be used to export content items & assets from Kontent.ai environments. However, when migration from 3rd party system you typically only use the import capabilities of this repository.

Export from Kontent.ai Configuration

Export CLI samples

# Export from Kontent.ai environment as json
kontent-ai-migration-toolkit --action=export --adapter=kontentAi --environmentId=xxx --format=json

# Export from Kontent.ai environment as csv
kontent-ai-migration-toolkit --action=export --adapter=kontentAi --environmentId=xxx --format=csv

Exporting in code

You may customize what items get exported by using the customItemsExport option when exporting in code. This option allows you to export exactly the items you need, however be aware that the exported items may reference other items that you might not export and subsequent re-import may fail because of the fact that there are missing items.

Examples:

1. Export from Kontent.ai environment
2. Custom export with files
3. Custom export and direct import

CLI help

To see available commands use:

kontent-ai-migration-toolkit --help

Use with config file

Create a json configuration file in the folder where you are attempting to run script. (e.g. export-config.json)

{
    "environmentId": "x",
    "secureApiKey": "y",
    "adapter": "kontentAi",
    "isSecure": true,
    "isPreview": false,
    "action": "export",
    "baseUrl": null,
    "format": "json" // or 'csv'
}

To execute your action run:

kontent-ai-migration-toolkit --config=export-config.json

Code samples for import & export

See https://github.com/Kontent-ai-consulting/kontent-ai-migration-toolkit/tree/main/samples for examples of how to run this library in code rather then via command line.

Output / Input formats

This library provides csv and json formats out of the box. However, you can create your own format by implementing IFormatService and supplying that to import / export functions. This is useful if you need to extend the existing format, change how it's processing or just support new formats such as xliff, xlxs, xml or other.

Following is a list of built-in format services:

Limitations

• Language variants in Scheduled workflow step exported with built-in Kontent.ai export adapter are not imported with this state as the information about scheduled time is missing in API response.

FAQ

I'm getting Header overflow exception

The Node.js limits the maximum header size of HTTP requests. In some cases it may be required for you to increase this limitation to be able to successfully fetch data from Kontent.ai. You can do so by using the max-http-header-size option (https://nodejs.org/api/cli.html#--max-http-header-sizesize)

Example script call:

node --max-http-header-size 150000 %USERPROFILE%\AppData\Roaming\npm\node_modules\kontent-ai-migration-toolkit\dist\cjs\lib\node\cli\app --action=export --apiKey=<key> --environmentId=<environmentId>