fusion-demo-data v1.0.1

Fusion Demo Data

Generator of static files to ingest into fusion for demoing and E2E tests trello board

Table of Contents

• Fusion Demo Data
  • Table of Contents
    • What is this?
    • How?
    • Why?
  • How to contribute
    • Running locally
    • The SRC Folder
    • The Build Folder
  • Tools
    • Mapping a 3D model.
    • Mapping a PDF.
    • Randomisation

What is this?

The tenant fusion is the demo tenant for cognite. It is also often used for E2E tests. Application developers need to be in control of the data that exists inside of this tenant. These developers also need to collaborate to ensure no one is stepping on any toes.

This repository contains the scripts to generate a tenants data from the ground up. The data can be used to demo all applications.

How?

The /src folder contains the scripts for generation. yarn build uses this folder to generate a folder system view of the data that will be ingested into the tenant. yarn send can then be used to remove all data from a tenant, and ingest the /build folder.

Why?

• To restrict all direct access to the tenant, to prevent undesired demo/E2E blips.
• To have a full destroy/build script, that one can run easily after finishing a demo (that anyone can do)
• Enable BOTH python and javascript to be used.

How to contribute

As an application developer, the goal is to adjust the results of yarn build to output data into the /build folder in the format you desire.

Running locally

Running yarn build will run your /src folder and generate the /build folder. This might be all you need. However if you want to test the publishing, you'll need a test tenant to play with (and an API key for that tenant). You can ask the #demo-tenant channel to get help with this if you don't have one already.

Once you have an API key, you can either run yarn publish --project {PROJECT} --apikey {APIKEY} --cluster {CLUSTER} or you can add these variables to your terminal env as COGNITE_DEMO_DATA_PROJECT, COGNITE_DEMO_DATA_API_KEY, COGNITE_DEMO_DATA_CLUSTER. Then you can run yarn publish.

The SRC Folder

The SRC folder is seperated into logical categories per use case.

For example, there is a generic assets folder where assets, and other standard items across all use cases are created. There is then a folder for each app - e.g. maintain, bestday. More shared folders can be added in the future.

Each folder is structured the same, and provides two methods of adjusting the build folder.

Via scripts The migration folder contains an ordered set of scripts to run - either using typescript or python. The resources folder contains any additional static JSON files or otherwise that might be needed. The combination of these two can be used to programatically generate a large number of resources, such as assets or work orders. When doing this method, keep in mind the required format of the build folder (see below)

Via static uploads The static folder contains files and folders that will simply be copied over to the build folder. Useful for items that don't necessarily need scripts to generate, such as files, 3d models, labels etc.

The Build Folder

The publish script requires the build folder to look a certain way. Each folder has its own logical representation.

1. Labels Each label has its own file, containing the JSON to generate the label build/labels/{label_external_id}.json

2. Dataset Each dataset has its own file, containing the JSON to generate the dataset Note: Datasets cannot be deleted. build/datasets/{dataset_external_id}.json

3. Assets The asset folder hierarchy matches the hierarchy that it will be ingested in. The index.ts file contains the details. We also generate the events here via the events.ts folder (potential future change) build/assets/{root_asset_external_id}/index.json build/assets/{root_asset_external_id}/{child_asset_external_id}/index.json

4. Timeseries & Datapoints The timeseries folder has timeseries in seperate folders. The folder name is the external ID of the asset. Each timeseries has its own file. The timeseries also contains a datapoints object. build/timeseries/{asset_external_id}/{timeseries_external_id.json

5. Events Events are grouped in a folder named after the use case for the events. (e.g. bestday_deferments, workorders) build/timeseries/{category}/{event_external_id.json

6. Relationships Each relationship has its own file, containing the JSON to generate the relationship build/relationship/{relationship_external_id}.json

7. RAW Each folder is its own database. Each file underneath the folder represents a table. The names of the folders match the names of the database/table. Contents of the file are the rows to be generated. build/raw/{database_name}/{table_name}.json

8. 3D Each 3D model has its own file. The name of the file is the name of the model. Note: Publishing the model must be done manually via Fusion. It takes too long to process the model. build/threeD/file.fbx

9. Files Each file has its own folder. Contents should be a index.json file containing file details, and ONE additional file. build/files/{file_external_id}/index.json (& /file.pdf)

10. Functions Each function has its own .zip file. build/functions/{function_external_id}.zip

11. 3D Mapping Each 3D model has its own file containing its nodeID:assetID mappings (See tools on how to make this easier) build/threeDMappings/{model_external_id}.json

12. File Mapping Each file as its own file containing annotation events. (See tools on how to make this easier) build/fileMappings/{file_external_id}.zip

Note: Publishing models must currently be done manually via fusion.

Tools

Certain aspects of ingesting data can be difficult. Some tools and processes have been created to make this substantially easier.

Mapping a 3D model.

To map a 3D model, we recommend the use of the discovery app. For this to work, you should run the publish script against a test tenant you have setup. (Alternatively, make a PR, then use the PR tenant)

1. Go to discovery.cogniteapp.com/lervik-industries/ (use your test tenant)
2. Map your 3D model to the assets.
3. Add the location of your 3D models in the SRC folder to the get3DMappings.ts script.
4. run yarn get-3d-mappings

For each model the script detects in the src folder, it will output a file in assets/static/threeDMappings. It will contain an additional assetExternalId, which the publish script will then read and transform into the correct ID.

NOTE: To run this script, you'll need an API key setup as described in the Running locally section

Mapping a PDF.

To map a PDF, we recommend the use of the fusion app. For this to work, you should run the publish script against a test tenant you have setup. (Alternatively, make a PR, then use the PR tenant)

1. Go to fusion.cognite.com/lervik-industries (use your test tenant) and navigate to the file you want to map
2. Map your file using the tools provided.
3. Add the location of your files in the SRC folder to the get2DMappings.ts script.
4. run yarn get-2d-mappings

For each file the script detects in the src folder, it will output a file in assets/static/fileMappings. It will contain all the annotation events. The metadata will be altered to replace the ID of the asset and file with the external IDs of each, which will then be replaced during publishing with the correct ID.

NOTE: During publishing, this will also connect the file to the relevant asset. NOTE: To run this script, you'll need an API key setup as described in the Running locally section

Randomisation

There are a few functions to assist with randomness randomChoice - Accepts an array, outputs a random item in that array randomRange - Accepts two numbers, outputs a random number between the two. Third parameter determines whether to round the number or not noise - Adds a small amount of noice to data.