Integrated Data Lake nodes¶

The Integrated Data Lake is a repository that allows you to store structured and unstructured data or objects in its native format as needed. It handles large data pools for which the schema and data requirements are not defined until the data is queried.

Integrated Data Lake nodes allows you to list, read, write, delete, subscribe and query the files or objects from Integrated Data Lake.

List objects¶

The "list objects" node lists the files or objects from Integrated Data Lake and stores the data in the message payload. Additionally, you can filter the files or objects by "Sub-tenant" property. It is possible by the main environment.

Read object¶

The "read object" node reads the file or object content from the specified path and places the content in the message payload. The mode parameter defines the type of content to be read from the file or object. The read object mode parameters are given below:

• Object: Reads only the content of the file or object.
• Object+Metadata: Reads both content and metadata of the file or object.
• Metadata: Reads only the metadata of the file or object.

Write object¶

The "Write object" node writes or updates the file or object to the specified path and places the content in the message payload. The mode parameter defines the type of content to be updated to the file or object. The write object mode parameters are given below:

• Object: Writes only the content to the file or object.
• Object+Metadata: Writes both content and metadata to the file or object.
• Metadata: Writes only the metadata to the file or object.

Subscribe object¶

The "subscribe object" node will notify the modifications that took place in the specified path. The path should be specified in the edit properties dialog box to receive the notification in the message payload. For example, the message object is shown below:

Delete object¶

The "delete object" node deletes the files or objects from the specified path. You must select the path from the "Select File" dialog box.

Using Integrated Data Lake nodes¶

You can read the files or objects of the turbo engine assets by using "read object" node from Integrated Data Lake. The extracted data can be used for further analysis as per customer's requirement.

Procedure¶

To read the object content from Integrated Data Lake, follow these steps:

1. From the node palette, drag and drop the "inject" node from the input palette, "read object" node from the IDL and Data ctx palette and "debug" node from the output palette to the working area. Connect the nodes in a flow as shown below:

   

2. Double click on the "inject" node to edit the properties. For more information, refer to Editing nodes.

3. Double click on the "read object" node to edit the properties.

   • Enter the "Name".
   • To select the file, click Select.
   • In the "Mode" section, you can select any one of the options from the dropdown menu.
   • Click "Done".

   

4. Save and execute the flow.

You can view the results in the message payload.

In this example the "read object" node reads the file or object from the selected path and displays the data in the message payload.