# fusion-demo-data

> Generator of static files to ingest into fusion for demoing and E2E tests [trello board](https://trello.com/b/TrUt0BXF/cognite-demo-data)

Latest version **1.0.1** (published 2021-05-10) · MIT license · 0 weekly downloads

## Install

```sh
npm install fusion-demo-data
pnpm add fusion-demo-data
yarn add fusion-demo-data
bun add fusion-demo-data
```

## Health

**Score 15/100 (F)** — status: abandoned.

Positive: no vulnerabilities.

Warnings: low downloads; no types; no esm support; large bundle.

Negative: abandoned; low maintenance score.

## Facts

| | |
|---|---|
| Version | 1.0.1 |
| Published | 2021-05-10 |
| First published | 2021-05-10 |
| Weekly downloads | 0 |
| License | MIT |
| TypeScript types | none |
| Module format | CommonJS |
| Dependencies | 14 |
| Unpacked size | 27 MB |
| Known vulnerabilities | 0 (+3 in 1 direct dependencies) |
| Install scripts | no |
| Author | Dan Levings |
| Maintainers | paulo-menezes-cognite |

## Links

- npm: https://www.npmjs.com/package/fusion-demo-data
- Repository: git@github.com:cognitedata/fusion-demo-data
- npm.io page: https://npm.io/package/fusion-demo-data

## Dependencies (14)

- [ncp](https://npm.io/package/ncp.md) ^2.0.0
- [walk](https://npm.io/package/walk.md) ^2.3.14
- [dayjs](https://npm.io/package/dayjs.md) ^1.10.4
- [tslib](https://npm.io/package/tslib.md) ^2.1.0
- [lodash](https://npm.io/package/lodash.md) ^4.17.21
- [adm-zip](https://npm.io/package/adm-zip.md) ^0.5.5
- [ts-node](https://npm.io/package/ts-node.md) ^9.1.1
- [bluebird](https://npm.io/package/bluebird.md) ^3.7.2
- [commander](https://npm.io/package/commander.md) ^7.2.0
- [mime-types](https://npm.io/package/mime-types.md) ^2.1.30
- [typescript](https://npm.io/package/typescript.md) ^4.2.3
- [@cognite/sdk](https://npm.io/package/@cognite/sdk.md) ^3.5.0
- [python-shell](https://npm.io/package/python-shell.md) ^3.0.0
- [firebase-admin](https://npm.io/package/firebase-admin.md) ^9.7.0

## Recent versions

- 1.0.1 (latest) — 2021-05-10
- 1.0.0 — 2021-05-10

## README

# Fusion Demo Data

Generator of static files to ingest into fusion for demoing and E2E tests
[trello board](https://trello.com/b/TrUt0BXF/cognite-demo-data)

# Table of Contents

- [Fusion Demo Data](#fusion-demo-data)
  - [Table of Contents](#table-of-contents)
    - [What is this?](#what-is-this)
    - [How?](#how)
    - [Why?](#why)
  - [How to contribute](#how-to-contribute)
    - [Running locally](#running-locally)
    - [The SRC Folder](#the-src-folder)
    - [The Build Folder](#the-build-folder)
  - [Tools](#tools)
    - [Mapping a 3D model.](#mapping-a-3d-model)
    - [Mapping a PDF.](#mapping-a-pdf)
    - [Randomisation](#randomisation)

## What is this?

The tenant `fusion` is the demo tenant for cognite. It is also often used for E2E tests. Application developers need to be in control of the data that exists inside of this tenant. These developers also need to collaborate to ensure no one is stepping on any toes.

This repository contains the scripts to generate a tenants data from the ground up. The data can be used to demo all applications.

## How?

The `/src` folder contains the scripts for generation.
`yarn build` uses this folder to generate a folder system view of the data that will be ingested into the tenant.
`yarn send` can then be used to remove all data from a tenant, and ingest the `/build` folder.

## Why?

- To restrict all direct access to the tenant, to prevent undesired demo/E2E blips.
- To have a full destroy/build script, that one can run easily after finishing a demo (that anyone can do)
- Enable BOTH python and javascript to be used.

# How to contribute

As an application developer, the goal is to adjust the results of `yarn build` to output data into the `/build` folder in the format you desire.

## Running locally

Running `yarn build` will run your `/src` folder and generate the `/build` folder. This might be all you need.
However if you want to test the publishing, you'll need a test tenant to play with (and an API key for that tenant). You can ask the #demo-tenant channel to get help with this if you don't have one already.

Once you have an API key, you can either run `yarn publish --project {PROJECT} --apikey {APIKEY} --cluster {CLUSTER}` or you can add these variables to your terminal env as `COGNITE_DEMO_DATA_PROJECT`, `COGNITE_DEMO_DATA_API_KEY`, `COGNITE_DEMO_DATA_CLUSTER`. Then you can run `yarn publish`.

## The SRC Folder

The SRC folder is seperated into logical categories per use case.

For example, there is a generic `assets` folder where assets, and other standard items across all use cases are created.
There is then a folder for each app - e.g. `maintain`, `bestday`. More shared folders can be added in the future.

Each folder is structured the same, and provides two methods of adjusting the build folder.

**Via scripts**
The **migration folder** contains an ordered set of scripts to run - either using typescript or python.
The **resources folder** contains any additional static JSON files or otherwise that might be needed.
The combination of these two can be used to programatically generate a large number of resources, such as assets or work orders.
When doing this method, keep in mind the required format of the build folder (see below)

**Via static uploads**
The **static folder** contains files and folders that will simply be copied over to the build folder. Useful for items that don't necessarily need scripts to generate, such as files, 3d models, labels etc.

## The Build Folder

The publish script requires the build folder to look a certain way. Each folder has its own logical representation.

1. **Labels**
   Each label has its own file, containing the JSON to generate the label
   `build/labels/{label_external_id}.json`

2. **Dataset**
   Each dataset has its own file, containing the JSON to generate the dataset
   Note: Datasets cannot be deleted.
   `build/datasets/{dataset_external_id}.json`

3. **Assets**
   The asset folder hierarchy matches the hierarchy that it will be ingested in. The `index.ts` file contains the details. We also generate the events here via the `events.ts` folder (potential future change)
   `build/assets/{root_asset_external_id}/index.json`
   `build/assets/{root_asset_external_id}/{child_asset_external_id}/index.json`

4. **Timeseries & Datapoints**
   The timeseries folder has timeseries in seperate folders. The folder name is the external ID of the asset. Each timeseries has its own file. The timeseries also contains a `datapoints` object.
   `build/timeseries/{asset_external_id}/{timeseries_external_id.json`

5. **Events**
   Events are grouped in a folder named after the use case for the events. (e.g. bestday_deferments, workorders)
   `build/timeseries/{category}/{event_external_id.json`

6. **Relationships**
   Each relationship has its own file, containing the JSON to generate the relationship
   `build/relationship/{relationship_external_id}.json`

7. **RAW**
   Each folder is its own database. Each file underneath the folder represents a table. The names of the folders match the names of the database/table. Contents of the file are the rows to be generated.
   `build/raw/{database_name}/{table_name}.json`

8. **3D**
   Each 3D model has its own file. The name of the file is the name of the model.
   Note: Publishing the model must be done manually via Fusion. It takes too long to process the model.
   `build/threeD/file.fbx`

9. **Files**
   Each file has its own folder. Contents should be a index.json file containing file details, and ONE additional file.
   `build/files/{file_external_id}/index.json (& /file.pdf)`

10. **Functions**
    Each function has its own .zip file.
    `build/functions/{function_external_id}.zip`

11. **3D Mapping**
    Each 3D model has its own file containing its nodeID:assetID mappings (See tools on how to make this easier)
    `build/threeDMappings/{model_external_id}.json`

12. **File Mapping**
    Each file as its own file containing annotation events. (See tools on how to make this easier)
    `build/fileMappings/{file_external_id}.zip`

Note: **Publishing models** must currently be done manually via fusion.

# Tools

Certain aspects of ingesting data can be difficult. Some tools and processes have been created to make this substantially easier.

## Mapping a 3D model.

To map a 3D model, we recommend the use of the `discovery` app. For this to work, you should run the publish script against a test tenant you have setup. (Alternatively, make a PR, then use the PR tenant)

1. Go to `discovery.cogniteapp.com/lervik-industries/` (use your test tenant)
2. Map your 3D model to the assets.
3. Add the location of your 3D models in the SRC folder to the `get3DMappings.ts` script.
4. run `yarn get-3d-mappings`

For each model the script detects in the src folder, it will output a file in `assets/static/threeDMappings`. It will contain an additional `assetExternalId`, which the publish script will then read and transform into the correct ID.

**NOTE: To run this script, you'll need an API key setup as described in the [Running locally](#running-locally) section**

## Mapping a PDF.

To map a PDF, we recommend the use of the `fusion` app. For this to work, you should run the publish script against a test tenant you have setup. (Alternatively, make a PR, then use the PR tenant)

1. Go to `fusion.cognite.com/lervik-industries` (use your test tenant) and navigate to the file you want to map
2. Map your file using the tools provided.
3. Add the location of your files in the SRC folder to the `get2DMappings.ts` script.
4. run `yarn get-2d-mappings`

For each file the script detects in the src folder, it will output a file in `assets/static/fileMappings`. It will contain all the annotation events. The metadata will be altered to replace the ID of the asset and file with the external IDs of each, which will then be replaced during publishing with the correct ID.

**NOTE: During publishing, this will also connect the file to the relevant asset.**
**NOTE: To run this script, you'll need an API key setup as described in the [Running locally](#running-locally) section**

## Randomisation

There are a few functions to assist with randomness
**randomChoice** - Accepts an array, outputs a random item in that array
**randomRange** - Accepts two numbers, outputs a random number between the two. Third parameter determines whether to round the number or not
**noise** - Adds a small amount of noice to data.

---
_Source: https://npm.io/package/fusion-demo-data · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._