generator-di-pyoperator v1.0.27

generator-di-pyoperator

Yeoman generator for developing and testing standalone/offline SAP Data Intelligence python operators.

The main features are

• Downloading from Data Intelligence instances and
• Uploading the operators
• Providing a 'mock-api'-package
• Creating operator python script if it does not exist
• Adding 'mock-api'-imports for offline development
• Creating a separate test script for testing the operator-script
• Providing a 'operator-test' package for mimicking inport-data

Usage

CLI call

yo di-operator [--init]

The option --initcreates the folders needed for the project and saves the 'mock-di-api.py' to 'utils/. You only need this option for initializing the project. Attention: If you have modified 'mock-di-api' it will be overridden.

Prompts

1. Download from DI or Upload to DI (stored)
2. SAP Data Intelligence URL (stored)
3. Tenant (stored, default = 'default')
4. User (stored): Sap Data Intelligence user of Workspace
5. Password (stored): Sap Data Intelligence user password of Workspace
6. Operator (stored): Operator name (=folder name in vflow)

Installation

YEOMAN

Yeoman is a scaffolding tool creating the framework and templates that you need for a quick start.

1. Installing npm, a JavaScript software registry and package installation tool
   • MacOs: brew install node
2. Installing yeoman
   • `npm install -g yo
3. Installing the di-pyoperator generator
   • npm install -g generator-di-pyoperator

Local SAP Data Intelligence Operator deployment

0. Create Git - Repository
1. Clone repository locally or make project directory
2. `cd project
3. Run yo-generator for preparing local DI operator development yo di-pyoperator

Details of the di-pyoperator Generator

Creates Project Folder Structure

• operators - where all the the Data Intelligence operators are stored
• utils - that stores the support packages mock_di_api and operator_test
• testdata - that keeps all test data for the operator. For each operator a separate folder is created

Download operator-Files

di-pyoperator downloads all operator files from the SAP Data Intelligence user workspace to the project folder

`/operators//

Operator Script File

This release is not supporting the operator development from scratch but is based of the upfront operator creation in the Data Intelligence modeler. If nothing is added to the script-editor in the modeler then no script-file is created by Data Intelligence.

If there is no script-file to which the operator.json attribute config:scriptis referencing to a new script.py is created with the necessary scaffolding.

# First 3 lines generated by di-pyoperator - DO NOT CHANGE (Deleted again when uploaded.)
from utils.mock_di_api import mock_api
api = mock_api(__file__)

def on_input(msg) :

	out_msg = None
	api.send('output',out_msg)


api.set_port_callback('input',on_input)

For each inport the callback-call and the callback-function is created (naming convention:'on_'). The first 3-lines of the script with the import of the mock-di-api package is going to be deleted once it is uploaded.

script_test File

For the standalone testing of the operator a separate file is created that imports the callback-functions of the operator. Although it will be uploaded to Data Intelligence it will not used there. The config parameters are read from the configSchema.json (datatype) and the operator.json (default value).

import script
from utils.mock_di_api import mock_api
from utils.operator_test import operator_test
        
api = mock_api(__file__)
optest = operator_test(__file__)

# config parameter 
api.config.person1 = 'Peter Nowhere'    # datatype : string
api.config.person2 = 'Odua Any'    # datatype : string
api.config.score = 3   # datatype : integer

msg = api.Message(attributes={'operator':'person.connectness'},body=None)
on_input(msg)

operator_test

For the standalone testing of an operator you need input data of the format specified in the operator.json 'inports' attribute. Commonly used formats are the file-data (provided by the 'Read File' operator) or message.table :

• message.file
• message.table

To support these there are methods of the operator_test class that reads csv-data from the folder `testdata//<testfile.csv>:

• get_file(<testfile.csv>) -> byte-code
• get_msgtable(<testfile.csv>) -> message with table attributes and 2-dim data array.

This is only a start and can be extended. Be careful when using the cmd yo di-operator --init any changes to operator_test are overwritten. h

Open Tasks

• Generating new operator standalone without previously creating one in SAP Data Intelligence modeler
• Inline scripts
• Adding git repository
• Utility functions for creating test data like reading csv-file from testdata and providing it as a message.table