latona-datacatalog v0.1.5

Latona - Data Catalog

Latona addon for generating model-based human-readable data catalog pages (md, html, Confluence Cloud)

• Disclaimer
• Usage
  • Quick start
  • Addon options
  • Upload pages to Confluence Cloud
• Contributing
• License

Disclaimer

This is NOT a fully-fledged data catalog implementation like Apache Atlas. This is just an addon to Latona code generator. This addon renders a set of pages to simplify digesting metadata in a user-friendly way. Nothing more than that.

We used this tool on a number occasions, usually when it is too early to start thinking about heavy data cataloging tools, but presenting some collected metadata is beneficial.

Usage

Quick start

1. Install Latona - see latona documentation
2. Scaffold model and project - see latona documentation
3. Add latona-datacatalog addon to package.json:

npm i latona-datacatalog

4. Add latona-datacatalog to your latona project. To use the default configuration add this snippet to the addons array:

{
  "moduleName": "latona-datacatalog",
  "options": {
    "renderFormat": "md"
  }
}

5. Add tables to your model. See our test model for examples. Data catalog expects tables array to be at the root of the model.

6. Add extra collections for other catalogs. Default configuration will look for collections.businessProcesses for business process catalog, and for collections.applications for applications catalog. Nothing bad will happen if these are missing - respective catalogs will be empty in this case.

7. Add relationship types to your model. They should land in collections.relationships by default. Each relationship type should be structured as follows (titles are used to group relationship links at the catalog item's page):

{
  "id": "string",
  "title_forward": "string",
  "title_backward": "string"
}

8. Add relationship records. The default place for them is at collections.relationshipTypes. typeId should be one of defined at the previous steps, source is the source (upstream) object's slug, target is the target (downstream) object's slug. Typical relationship looks like this:

{
  "typeId": "fk",
  "source": "[orders].[dbo].[FactOrderItem]",
  "target": "[orders].[dbo].[DimClient]"
}

9. Add custom markdown content to the ./content folder (use item's slug as a file name).

10. Render as described in the Latona's guide.

Addon options

Please refer for more information to these source files:

• Default configuration
• Options schema
• Test model

Upload pages to Confluence Cloud

The latona-datacatalog package also includes dc2confcloud utility which enables automated upload of pages to your Confluence Cloud space.

1. Install the package globally to use this tool:

npm i -g latona-datacatalog

2. Create the configuration json:

{
  "confluence": {
    "baseUrl": "<your_account>.atlassian.net",
    "spaceKey": "<your_space_key>"
  },
  "contentFolder": "./path/to/generated/conf/output",
  "notesSuffix": " - Notes", // see addon options
  "pageTree": "___page_tree___.json", // see addon options
  // specify parent pages for each "root" catalog page
  "parents": [
    {
      "slug": ".datacatalog",
      "confPageId": 123456789 // you need to create this page manually
    },
    {
      "slug": ".datacatalog-notes",
      "confPageId": 123456789 // you need to create this page manually
    },
    {
      "slug": ".bpcatalog",
      "confPageId": 123456789 // you need to create this page manually
    },
    {
      "slug": ".bpcatalog-notes",
      "confPageId": 123456789 // you need to create this page manually
    },
    {
      "slug": ".appcatalog",
      "confPageId": 123456789 // you need to create this page manually
    },
    {
      "slug": ".appcatalog-notes",
      "confPageId": 123456789 // you need to create this page manually
    }
  ]
}

3. Run the tool:

dc2confcloud -u <your_user_name> -t <your_access_token> -c <./path/to/dc2confcloud.json>

Usage details:

Usage: dc2confcloud [options]

Options:
  -u, --user <user>      Confluence user name (email)
  -t, --token <token>    Confluence access token
  -c, --config <config>  path to dc2confcloud's configuration json (default: "./dc2confcloud.json")
  -h, --help             display help for command

Contributing

Please read our Latona's contribution guidelines for details on our development approach, and the process for submitting pull requests to us. All contributors should comply with our Latona's Code of Conduct

License

Latona is copyright (c) 2019-present DataArt (www.dataart.com) and all contributors and licensed under the Apache License, Version 2.0. See the LICENSE file for more details.