bdp-page-api v0.8.5

The Page API in Big Data Processor

(Click to view the jsDoc on Github Pages).

Introduction

Workflow or package developers can freely design web interfaces to provide documentations, guide users to execute workflows, or visualize results interactively, etc. In Big Data Processor (BDP), we provide a web sandbox environments to display these freely customized Pages. To reduce efforts communicating with servers, we provide a set of javascript APIs to communicate between our BDP web client and the Page which runs in the sandbox (see the following figure).

In this way, Page developers can easily get the information by just calling javascript functions. Also, developers need not handle many details such as file uploading or downloading.

Usage

To use the API, you need to include the javascript file in your Page.

<script src="https://cdn.jsdelivr.net/gh/big-data-processor/bdp-page-api/bdp-page-api.js"></script>

Next, you need to new an API instance and then call the .initialize() function to make a handshake with the BDP web client. After the handshake, the API is ready to use. Most functions are asynchronous, you can wrap the function calls with the async/await pattern.

const bdpAPI = new BdpPageAPI();                      // initiate an API instance
(async () => {
    await bdpAPI.initialize();                        // make the first handshake.
    const dataFileRecords = await bdpAPI.listFiles(); // get the DataFile records in the current project.

    /**
     * The bdpAPI object is ready to use.
     * You can call these API functions to interact with BDP.
     */
})().catch((err) => console.log);

API Categories

1. Query information on BDP

It is very easy to get metadata of data records on BDP. The following functions allow you to get many kinds of information:

• getCurrentFileInfo: Retrieve the current DataFile information. Only works in File Pages since users have selected a DataFile record.
• getCurrentPackageInfo: Retrieve the current Package that contains this customized Page.
• getCurrentProjectInfo: Retrieve the current Project that a user has selected.
• getCurrentResultInfo: Retrieve the current Result information. Only works in Result Pages since users have selected a Result record.
• getCurrentUserInfo: Retrieve the current user information.
• listFiles: List DataFile records in the current Project.
• listResults: List Result records in the current Project.
• listPackages: List the Packages selected in the current Project.
• listTasks: Get the Task information by task key (or task id) and package id.
• listAllTasks: List all related Tasks from Packages that are selected in the current project.
• listProjects: List projects of the current user, shared projects and public projects.

2. Navigation

The following functions may leave the current Page and navigate to another Page.

• navigateToBdpDataFile: Navigate to view the DataFile record. Similar to the BdpPageAPI.openFileLink function except that it does not open a new window.
• navigateToBdpResult: Navigate to view the DataFile record. Similar to the BdpPageAPI.openFileLink function except that it does not open a new window.
• navigateToProjectList: Navigate to the project list page of BDP-client.
• navigateToFilePage: Navigate to a File Page.
• navigateToResultPage: Navigate to a Result Page.
• navigateToProjectPage: Navigate to a Project Page.
• navigateToProxyPage: Navigate to a Proxy Page corresponding to a Result record.
• navigateToStaticPage: Navigate to a Static Page corresponding to a folder DataFile record.
• openFileLink: Open a new window to display the DataFile record.
• openResultLink: Open a new window to display the Result record.
• openStaticLink: Open a new window to directly display the file or file list in a Folder.

3. Project related API

• createProject: create a new project
• getProjectInfo: get a detailed object of the project

4. DataFile handling

• createFolder: Create an empty folder and return the corresponding DataFile record.
• deleteFile: Delete a DataFile record.
• getFileBlob: Use this function to automatically download the file content in the Blob type. You may call BDPPageUtils.readFileBlob to process the Blob object.
• globFilesInFolder: Get a list of valid file paths located inside a folder that corresponds to the DataFile. Under the hood, we use the npm package, globby, to use the globby expressions.
• importFileFromPath: Import files that already on the server (the same machine that BDP is hosted on). This function can only be executed by at least the level of Power User.
• updateFileInfo: Update the metadat of the DataFile record. Currently, you cannot change the format of a DataFile.
• uploadFiles: Upload files from FileList or from an array of Blob objects.
• uploadFileBlob: Upload a Blob object to create a DataFile record.
• importFilesToProject: export the dataFile records in the current Project to a selected project.

5. Task related API

• executeTask: Execute a Task and generate a Result record.
• getTaskArguments: Get the argument settings by task key (or task id) and package id.
• getTaskInfo: Get the Task information by task key (or task id) and package id.
• getTaskInputGuide: Providing a reference to generate Task arguments for users to specify before executing Tasks.

6. Result related API

• updateResultInfo: Update the metadata of a Result record.
• deleteResult: Delete the Result record. All output DataFile records of this Result are also deleted.
• stopResult: Stop a task which corresponds to a Result record.
• resumeResult: Stop a task which corresponds to a Result record. (coming soon)

7. Watch server changes

Using these functions to register listener functions (or event handlers). These registered functions will be called once the BDP receives state updates. You may use these listeners to update the corresponding information. For example, if you register listeners for Result messages, you can get the Task execution messages (standard output and standard error) in real-time.

• watchFileChange: Register a listener function for DataFile changes. Only the DataFiles that are owned by you or shared with you can be listen. Whenever there is a listenable DataFile changes on the server, the registered functions will be called. You can use this function to get notified and make some updates on your Page.
• stopWatchFileChange: Remove the registered listener function for DataFile changes.
• watchResultChange: Register a listener function for Result changes. Whenever there is a listenable Result changes on the server, the registered functions will be called. You can use this function to get notified and make some updates on your Page.
• stopWatchResultChange: Remove the registered listener function for Result changes.
• watchResultMsgChange: Register a listener function to listen Result messages. Only the related Results in the Projects that are owned by you or shared with you can be listened. Result messages are standard outputs and standard errors generated by running Tasks.
• stopWatchResultMsgChange: Remove the registered listener function for Result messages.

8. UI related API

We also provide API functions to control the UI of the BDP client. For example, you can use toggleFullPage function to enter full page mode. In this mode, many components such as the lefthand-side navigation bar or top navigation header are hidden on the BDP web client. Your customized Page will be displayed in full page as if there are no BDP web client.

• toggleFullPage: Toggle the full page mode.
• togglePageList: Toggle the Page list.
• toggleRightSideMenu: toggleRightSideMenu
• toggleRightSideBar: (coming soon.)

9. indexedDB-related API

IndexedDB is one of the web storage techniques. It allows you to persistently store data inside a browser. However, for security reasons, the customized Pages are displayed in a sandboxed environment. By default, Pages have no origin (no web address) and can not use cookies or the indexedDB directly. Therefore, We provide an interface to handle the indexedDB and the BDP-client controls the indexedDB. Databases of indexedDB are under BDP-client's control and each Page per User can have one indexedDB. Under the hood, We used the npm package, idb, to handle the indexedDB. For more information about the indexedDB, please see this reference.

Database related

• requestDatabase: Request an indexedDB database to use.
• requestDatabaseRemove: Remove the requested indexedDB database.

Adding/updating data in a data store

• dataStoreAdd: Inserting data records to the data store in the indexedDB.
• dataStorePut: Similar to BdpPageAPI.dataStoreAdd, this function performs the update or insert method. Data records of the same key are updated.

Querying data in a data store

• dataStoreQuery: Query data in a dataStore. The iterateFn will be called with each of the resulting data records.
• dataStoreGet: A simplified version of BdpPageAPI.dataStoreQuery. This function will directly return the query results.
• dataStoreCount: Counting the record numbers of a data store in the indexedDB.

Remove data or data store

• dataStoreClear: Clearing data records in a data store in the indexedDB. The data store exists and becomes empty.
• dataStoreRemove: Removing a data store in the indexedDB. The data store is removed and not existed.

10. Low-level API

• requestHttp: A low level API to make requests to BDP server.

License

Licensed under the MIT license (see the license)