Skip to content

Using Integrated Data Lake nodes

You can read the files or objects of the turbo engine assets by using "read object" node from Integrated Data Lake. The extracted data can be used for further analysis as per customer's requirement.

Procedure

To read the object content from Integrated Data Lake, follow these steps:

  1. From the node palette, drag and drop the "inject" node from the input palette, "read object" node from the IDL and Data ctx palette and "debug" node from the output palette to the working area. Connect the nodes in a flow as shown below:

    Interconnect node

  2. Double click on the "inject" node to edit the properties. For more information, refer to Editing nodes.

  3. Double click on the "read object" node to edit the properties.

    • Enter the "Name".
    • To select the file, click Select.
    • In the "Mode" section, you can select any one of the following options from the dropdown menu.
    • Click "Done".

    Edit read object

  4. Save and execute the flow.

Note

  • Path should be specified for the "List object" and "Subscribe object" nodes.
  • Select the path from the "Select File" dialog box for the "Read object", "Write object" and "Delete object" nodes from the file browser dialog box.
  • Mode defining is applicable for "Read object" and "Write object" nodes.

You can view the results in the message payload.

In this example the "read object" node reads the file or object from the selected path and displays the data in the message payload.

Read object

Using Data ctx nodes

To use the Data ctx nodes with Integrated Data Lake, follow these steps:

Visual Flow Creator can now use data ctx nodes to execute the following tasks:

Using data ctx create query node

To create a query using "data ctx create query" node, follow these steps:

  1. From the node palette, drag and drop the "inject" node from the input palette, "data ctx create query" node from the IDL and Data ctx palette and "debug" node from the output palette to the working area. Connect the nodes in a flow as shown below:

    Create query node

  2. Double click on the "inject" node to edit the properties. For more information, refer to Editing nodes.

  3. Double click on the "data ctx create query" node to edit the properties:

    • Enter the "Name" as "My new query".
    • Enter the "Description" as "Creating a query".
    • In the "Dynamic" section, select "Yes" from the dropdown.
    • Enter the "SQL Statement" as SELECT caseyear4 from sdi_test79 where caseyear4=:"caseyear4.
    • Click "Done".
  4. Save and execute the flow.

The output is displayed in the debug window:

create query output

Using data ctx execution job node

To execute jobs using "data ctx execution job" node, follow these steps:

  1. From the node palette, drag and drop the "inject" node from the input palette, "data ctx execution job" node from the IDL and Data ctx palette and "debug" node from the output palette to the working area. Connect the nodes in a flow as shown below:

    Execution job node

  2. Double click on the "inject" node to edit the properties. For more information, refer to Editing nodes.

  3. Double click on the "data ctx execution job" node to edit the properties.

    • Enter the "Name" as "execution job for WELLINGTON".
    • Enter the "Description" as "creating execution job".
    • Enter the "Query ID" as 5ec3c9dc55e38921e78e1f66.
    • Enter the "Parameters" as Name="caseyear4" and Value="WELLINGTON.
    • Click "Done".
  4. Save and execute the flow.

The output is displayed in the debug window:

Execution job output

Note

"data ctx execution job" node is only for dynamic queries.

Using data ctx query results node

To execute queries using "data ctx query results" node, follow these steps:

  1. From the node palette, drag and drop the "inject" node from the input palette, "data ctx query results" node from the IDL and Data ctx palette and "debug" node from the output palette to the working area. Connect the nodes in a flow as shown below:

    Result of query

  2. Double click on the "inject" node from the input palette to edit the properties. For more information, refer to Editing nodes.

  3. Double click on the "data ctx query results" node to edit the properties.

    • Enter the "Name" as "data ctx query results".
    • Enter the "Query ID" as 5ec3c9dc55e38921e78e1f66.
    • Enter the "Execution job ID" as 5ec3ee5255e38921e78e1f79.
    • Click "Done".
  4. Save and execute the flow.

The output is displayed in the debug window:

Query result node output

Using data ctx ingest status node

To get the status of ingest node using "data ctx ingest status" node, follow these steps:

  1. From the node palette, drag and drop the "inject" node from the input palette, "data ctx ingest status" node from the IDL and Data ctx palette and "debug" node from the output palette to the working area. Connect the nodes in a flow as shown below:

    Ingest node

  2. Double click on the "inject" node from the input palette to edit the properties. For more information, refer to Editing nodes.

  3. Double click on the "data ctx ingest status" node to edit the properties.

    • Enter the "Name" as "data ctx ingest node".
    • Enter the "Job ID" as 5ec3ee5255e38921e78e1f79.
    • Click "Done".
  4. Save and execute the flow.

The output is displayed in the debug window:

ingest status output


Last update: July 18, 2024