Using Integrated Data Lake nodes¶
You can read the files or objects of the turbo engine assets by using "read object" node from Integrated Data Lake. The extracted data can be used for further analysis as per customer's requirement.
Procedure¶
To read the object content from Integrated Data Lake, follow these steps:
-
From the node palette, drag and drop the "inject" node from the input palette, "read object" node from the IDL and Data ctx palette and "debug" node from the output palette to the working area. Connect the nodes in a flow as shown below:
-
Double click on the "inject" node to edit the properties. For more information, refer to Editing nodes.
-
Double click on the "read object" node to edit the properties.
- Enter the "Name".
- To select the file, click .
- In the "Mode" section, you can select any one of the following options from the dropdown menu.
- Click "Done".
-
Save and execute the flow.
Note
- Path should be specified for the "List object" and "Subscribe object" nodes.
- Select the path from the "Select File" dialog box for the "Read object", "Write object" and "Delete object" nodes from the file browser dialog box.
- Mode defining is applicable for "Read object" and "Write object" nodes.
You can view the results in the message payload.
In this example the "read object" node reads the file or object from the selected path and displays the data in the message payload.
Using Data ctx nodes¶
To use the Data ctx nodes with Integrated Data Lake, follow these steps:
- Create a data registry in Data Contextualization. For more information, refer to Creating the Data Registry.
- Once the data registry is created, upload the data to Integrated Data Lake. For more information, refer to Uploading a file in Integrated Data Lake.
Visual Flow Creator can now use data ctx nodes to execute the following tasks:
Using data ctx create query node¶
To create a query using "data ctx create query" node, follow these steps:
-
From the node palette, drag and drop the "inject" node from the input palette, "data ctx create query" node from the IDL and Data ctx palette and "debug" node from the output palette to the working area. Connect the nodes in a flow as shown below:
-
Double click on the "inject" node to edit the properties. For more information, refer to Editing nodes.
-
Double click on the "data ctx create query" node to edit the properties:
- Enter the "Name" as "My new query".
- Enter the "Description" as "Creating a query".
- In the "Dynamic" section, select "Yes" from the dropdown.
- Enter the "SQL Statement" as
SELECT caseyear4 from sdi_test79 where caseyear4=:"caseyear4
. - Click "Done".
-
Save and execute the flow.
The output is displayed in the debug window:
Using data ctx execution job node¶
To execute jobs using "data ctx execution job" node, follow these steps:
-
From the node palette, drag and drop the "inject" node from the input palette, "data ctx execution job" node from the IDL and Data ctx palette and "debug" node from the output palette to the working area. Connect the nodes in a flow as shown below:
-
Double click on the "inject" node to edit the properties. For more information, refer to Editing nodes.
-
Double click on the "data ctx execution job" node to edit the properties.
- Enter the "Name" as "execution job for WELLINGTON".
- Enter the "Description" as "creating execution job".
- Enter the "Query ID" as
5ec3c9dc55e38921e78e1f66
. - Enter the "Parameters" as
Name="caseyear4" and Value="WELLINGTON
. - Click "Done".
-
Save and execute the flow.
The output is displayed in the debug window:
Note
"data ctx execution job" node is only for dynamic queries.
Using data ctx query results node¶
To execute queries using "data ctx query results" node, follow these steps:
-
From the node palette, drag and drop the "inject" node from the input palette, "data ctx query results" node from the IDL and Data ctx palette and "debug" node from the output palette to the working area. Connect the nodes in a flow as shown below:
-
Double click on the "inject" node from the input palette to edit the properties. For more information, refer to Editing nodes.
-
Double click on the "data ctx query results" node to edit the properties.
- Enter the "Name" as "data ctx query results".
- Enter the "Query ID" as
5ec3c9dc55e38921e78e1f66
. - Enter the "Execution job ID" as
5ec3ee5255e38921e78e1f79
. - Click "Done".
-
Save and execute the flow.
The output is displayed in the debug window:
Using data ctx ingest status node¶
To get the status of ingest node using "data ctx ingest status" node, follow these steps:
-
From the node palette, drag and drop the "inject" node from the input palette, "data ctx ingest status" node from the IDL and Data ctx palette and "debug" node from the output palette to the working area. Connect the nodes in a flow as shown below:
-
Double click on the "inject" node from the input palette to edit the properties. For more information, refer to Editing nodes.
-
Double click on the "data ctx ingest status" node to edit the properties.
- Enter the "Name" as "data ctx ingest node".
- Enter the "Job ID" as
5ec3ee5255e38921e78e1f79
. - Click "Done".
-
Save and execute the flow.
The output is displayed in the debug window: