0.1.5 • Published 5 years agoCLI

latona-datacatalog

Licence

Apache 2.0

Version

0.1.5

Deps

Size

99 kB

Vulns

Weekly

Summary Dependency Versions

Latona - Data Catalog

Latona addon for generating model-based human-readable data catalog pages (md, html, Confluence Cloud)

Disclaimer
Usage
Contributing
License

This is NOT a fully-fledged data catalog implementation like Apache Atlas. This is just an addon to Latona code generator. This addon renders a set of pages to simplify digesting metadata in a user-friendly way. Nothing more than that.

We used this tool on a number occasions, usually when it is too early to start thinking about heavy data cataloging tools, but presenting some collected metadata is beneficial.

Usage

Quick start

Install Latona - see latona documentation
Scaffold model and project - see latona documentation
Add latona-datacatalog addon to package.json:

npm i latona-datacatalog

Add latona-datacatalog to your latona project. To use the default configuration add this snippet to the addons array:

{
  "moduleName": "latona-datacatalog",
  "options": {
    "renderFormat": "md"
  }
}

Add tables to your model. See our test model for examples. Data catalog expects tables array to be at the root of the model.
Add extra collections for other catalogs. Default configuration will look for collections.businessProcesses for business process catalog, and for collections.applications for applications catalog. Nothing bad will happen if these are missing - respective catalogs will be empty in this case.
Add relationship types to your model. They should land in collections.relationships by default. Each relationship type should be structured as follows (titles are used to group relationship links at the catalog item's page):

{
  "id": "string",
  "title_forward": "string",
  "title_backward": "string"
}

Add relationship records. The default place for them is at collections.relationshipTypes. typeId should be one of defined at the previous steps, source is the source (upstream) object's slug, target is the target (downstream) object's slug. Typical relationship looks like this:

{
  "typeId": "fk",
  "source": "[orders].[dbo].[FactOrderItem]",
  "target": "[orders].[dbo].[DimClient]"
}

Add custom markdown content to the ./content folder (use item's slug as a file name).
Render as described in the Latona's guide.

Addon options

Please refer for more information to these source files:

Option	Type	Required	Description
paths	object	yes
paths.outPath	string	yes	output path (default: `./output`)
paths.customContentPath	string	yes	path to the folder with custom markdown files (use slug for names and custom markdown will be injected)
renderFormat	enum ["md", "html", "conf"]	yes
pageTreeFileName	string	yes	filename for the JSON file with the list of all generated pages
conf	object		Confluence-specific configuration
conf.createNotesPage	boolean		`true` by default
conf.notesSlugSuffix	string	yes (only if parent object is present)	Slug suffix for "notes" pages - empty pages that may be editted directly at Confluence. Notes pages are injected into catalog pages via macro.
conf.notesTitleSuffix	string	yes (only if parent object is present)	Suffix for "notes" page titles.
conf.modelConfig	object	yes (only if parent object is present)
conf.modelConfig.spaceKey	string	yes (only if parent object is present)	Confluence space key
catalogs	object	yes (only if parent object is present)	Catalogs configuration (there may be many)
catalogs.<catalogName>	object	yes	Catalog configuration object. Only latin letters.
catalogs.<catalogName>.collection	string	yes	path to array of catalog items (from the model root)
catalogs.<catalogName>.title	string	yes	catalog title
catalogs.<catalogName>.slug	string	yes	catalog slug (url-friendly string)
catalogs.<catalogName>.sectionBy	object		sectioning configuration (at least `first` level should be described, only two levels are supported)
catalogs.<catalogName>.sectionBy.first	object	yes (only if parent object is present)	first level of grouping
catalogs.<catalogName>.sectionBy.first.properties[]	array	yes (only if parent object is present)	properties to group by
catalogs.<catalogName>.sectionBy.first.properties[].name	string	yes	property name (for internal use only)
catalogs.<catalogName>.sectionBy.first.properties[].path	string	yes	property path (relative to catalog item)
catalogs.<catalogName>.sectionBy.first.properties[].title	string	yes	property title
catalogs.<catalogName>.sectionBy.first.properties[].show	boolean		show property
catalogs.<catalogName>.sectionBy.first.properties[].showInLinkTable	boolean		show property on the parent page if `"inParentAs": "linkTable"`
catalogs.<catalogName>.sectionBy.first.inParentAs	enum ["linkTable", "linkList", "section"]	yes (only if parent object is present)	sets how to show sections on the parent page (catalog root page for the first level of grouping)
catalogs.<catalogName>.sectionBy.first.tableLinkProperty	string		sets which property should be used for linking to the section details page
catalogs.<catalogName>.sectionBy.first.titleTemplate	string	yes (only if parent object is present)	section title template (variable expansion is supported)
catalogs.<catalogName>.sectionBy.first.slugTemplate	string	yes (only if parent object is present)	section slug template (variable expansion is supported)
catalogs.<catalogName>.sectionBy.second	object		see `catalogs.<catalogName>.sectionBy.first`
catalogs.<catalogName>.details	object	yes	see `catalogs.<catalogName>.sectionBy.first`
catalogs.<catalogName>.detailsCollections	object		describes inner collections for the catalog's item (e.g.: if database table is a catalog item, then table fields is the inner collection)
catalogs.<catalogName>.detailsCollections.<name>	object	yes (only if parent object is present)	Collection configuration object. Only latin letters.
catalogs.<catalogName>.detailsCollections.<name>.collection	string	yes (only if parent object is present)	path to array of collection items (relative to item)
catalogs.<catalogName>.detailsCollections.<name>.properties[]	array	yes (only if parent object is present)	collection properties
catalogs.<catalogName>.detailsCollections.<name>.properties[].name	string	yes (only if parent object is present)	property name (for internal use only)
catalogs.<catalogName>.detailsCollections.<name>.properties[].path	string	yes (only if parent object is present)	property path (relative to collection item)
catalogs.<catalogName>.detailsCollections.<name>.properties[].title	string	yes (only if parent object is present)	property title
catalogs.<catalogName>.detailsCollections.<name>.heading	string	yes (only if parent object is present)	section's title
relationshipsCollection	string	yes	path to array of relationships (relative to `collections` object within the latona model)
relationshipTypesCollection	string	yes	path to array of relationship types (relative to `collections` object within the latona model)
dropDefaultCatalogs	boolean		`false` by default, removes default catalogs configuration if `true`

Upload pages to Confluence Cloud

The latona-datacatalog package also includes dc2confcloud utility which enables automated upload of pages to your Confluence Cloud space.

Install the package globally to use this tool:

npm i -g latona-datacatalog

Create the configuration json:

{
  "confluence": {
    "baseUrl": "<your_account>.atlassian.net",
    "spaceKey": "<your_space_key>"
  },
  "contentFolder": "./path/to/generated/conf/output",
  "notesSuffix": " - Notes", // see addon options
  "pageTree": "___page_tree___.json", // see addon options
  // specify parent pages for each "root" catalog page
  "parents": [
    {
      "slug": ".datacatalog",
      "confPageId": 123456789 // you need to create this page manually
    },
    {
      "slug": ".datacatalog-notes",
      "confPageId": 123456789 // you need to create this page manually
    },
    {
      "slug": ".bpcatalog",
      "confPageId": 123456789 // you need to create this page manually
    },
    {
      "slug": ".bpcatalog-notes",
      "confPageId": 123456789 // you need to create this page manually
    },
    {
      "slug": ".appcatalog",
      "confPageId": 123456789 // you need to create this page manually
    },
    {
      "slug": ".appcatalog-notes",
      "confPageId": 123456789 // you need to create this page manually
    }
  ]
}

Run the tool:

dc2confcloud -u <your_user_name> -t <your_access_token> -c <./path/to/dc2confcloud.json>

Usage details:

Usage: dc2confcloud [options]

Options:
  -u, --user <user>      Confluence user name (email)
  -t, --token <token>    Confluence access token
  -c, --config <config>  path to dc2confcloud's configuration json (default: "./dc2confcloud.json")
  -h, --help             display help for command