Integrating an AWS Cloud - via MindConnect Integration¶

MindConnect Integration can transfer data from any cloud storage service into Insights Hub. In this example, multiple variable definitions for an aspect are read from a CSV file, which is stored in an AWS S3 bucket. A pipeline is set up to automatically create an asset, which has an aspect with these variables.

Prerequisites¶

• A Insights Hub Account
• The MindConnect Integration
• An Amazon S3 account with access to AWS Identity and Access Management
• The following roles must be assigned to your user:
  • mdsp:core:TenantAdmin, assigned in user management.
  • mdsp:core:mci:adminor mdsp:core:mci.user, assigned in user management.

Preparing Data in AWS¶

This section describes how to create an S3 bucket in AWS and upload aspect data from a CSV file to it.

Creating an AWS User with API Access¶

1. Open the AWS IAM console via https://console.aws.amazon.com/iam/ (login required).
2. Choose "Users" and then "Add user" in the navigation pane.
3. Enter a user name for the user.
4. Select "Programmatic access".
5. Click on "Next: Permissions".
6. Select "Attach existing policies to user directly" and pick the "AmazonS3ReadOnlyAccess" policy. (You can update the policies later, if necessary.)
7. Finish the process by clicking "Next: Review" and then "Create user".
8. Download the access key ID and secret access key and save them. You will not have access to these keys again after this step.

The generated access keys provide access to the AWS S3 APIs.

Creating an S3 Bucket¶

1. Open the Amazon S3 console at https://console.aws.amazon.com/s3/.
2. Click on "Create bucket".
3. Enter a bucket name and select an AWS region, e.g. "sensor-bucket-mindsphere" and "EU (Frankfurt)".
4. Do not make any other configurations and click on "Create".

The new S3 Bucket is available in the Amazon S3 console.

Uploading a CSV file to the S3 Bucket¶

1. Create a CSV file for an aspect with multiple variables using the same format as shown below:

   AspectName;SensorType;DataType;Unit;
   breweryAspect;temperature;DOUBLE;°C;
   breweryAspect;motorVoltage;DOUBLE;V;
   breweryAspect;fluidFlow;DOUBLE;m³/h;
   breweryAspect;pressureBefore;DOUBLE;bar;
   breweryAspect;pressureAfter;DOUBLE;bar;
   

2. Select your S3 bucket (here "sensor-bucket-mindsphere") in AWS.

3. Drag and drop the CSV file into the Amazon S3 console window.
4. Click "Upload".

The CSV file has been uploaded to your S3 bucket.

Integrating an Amazon S3 Bucket into MindConnect¶

A MindConnect integration is set up in three steps:

1. Creating an account: MindConnect Integration requires an account for storing access information to connect to other cloud-based applications, e.g. Amazon S3 bucket.
2. Adding an operation: MindConnect Integration uses application specific operations for reading or writing data to the connected cloud-based applications. Each operation performs one application specific task.
3. Creating a pipeline: MindConnect Integration uses pipelines to define processes for transmitting, interpreting and transforming data. Pipelines can combine multiple operations and other pipelines to define multistep workflows.

Creating a MindConnect Integration Account¶

1. Open MindConnect Integration from the Launchpad.
2. Log in with your MindConnect Integration credentials.
3. Go to "Connect".
4. Open the application "Amazon Simple Storage Service (S3)".
5. Click on "Add New Account".
6. Enter a name for the account, e.g. "My_S3_Bucket".
7. Enter the access key ID and secret access key generated before.
8. Do not change any other settings.
9. Click "Save".

The account has been created.

Adding a new Operation¶

1. Switch to the "OPERATIONS" tab and click on "Add New Operation".
2. Enter a name for your operation, e.g. "RetrieveS3Object".
3. Select the account (here "My_S3_Bucket") from the drop-down and click on "Next".
4. Select "GetObject" from the list of operations.
5. Click "Next" without making any further configurations in the following dialogues.
6. Check the information and click "Finish".

The "GetObject" operation is now visible in the list.

Setting up a Pipeline¶

This pipeline transfers the data from the S3 bucket into MindConnect. It is constructed using building blocks, which first retrieve the data from the CSV file, then convert it into bytes and finally store them in a document. The building blocks, as well as the input and the output for the pipeline itself, are configured after the assembly.

1. Switch to the "INTEGRATIONS" tab and click on "Add New Integration".
2. Select "Orchestrate two or more applications" in the pop-up window.
3. Enter a name for your integration, e.g. "My_S3_Bucket_Pipeline".
4. Open "Applications" in the tool bar on the left.
5. Search for "Amazon Simple Storage Service (S3)" and drag it under the integration block so it connects with the anchor point.
6. Click on the settings icon of this block and select the account (here "My_S3_Bucket") and operation (here "RetrieveS3Object") from the drop-down menus.
7. Open "Services" in the tool bar on the left.
8. Drag "IO" from Services under the "Amazon Simple Storage Service S3" block so it connects with the anchor point.
9. Open the drop-down menu of the "IO" block and select "streamToBytes".
10. Drag "Flat File" from Services under the "IO" block so it connects with the anchor point.
11. Open the drop-down menu of the "Flat File" block and select "delimitedDataBytesToDocument".
12. Click "Save".

The finished pipeline looks as shown below:

Configuring the Input/Output Signature¶

Every MindConnect integration requires an input/output signature, which must define at least one input parameter. Output parameters are optional. This integration shall take an S3Object as input parameter and output a document with rows and columns.

1. Click on the menu icon at the very right of the integration block on top of your pipeline.
2. Select "Define Input/Output Signature".
3. Click on the plus button in the Input tab to create an input field.
4. Enter a name, e.g. S3Object, and set the type to "String".
5. Switch to the Output tab and click the plus button to add an output field of type "Document".
6. Click the plus button again to add another field of type "Document" and activate the "Array" checkbox. This field is nested inside the other Document and represents the rows of the CSV file.
7. Add four fields of type "String" representing the columns of the CSV file. The intended structure is shown below:
8. Click "Apply" and then "Save".

The Input/Output signature is defined.

Configuring the Mapping for the Operation¶

The "Amazon Simple Storage Service (S3)" block is configured to retrieve an S3Object from the Amazon S3 bucket and forward it as a stream.

1. Click on the menu icon at the very right of the "Amazon Simple Storage Service (S3)" block.
2. Select "Map Input and Output".
3. Configure the input mapping as shown below:
4. Double-click on the field bucketName in "RetrieveS3ObjectInput" and enter the name of your S3 bucket (here "sensor-bucket-mindsphere").
5. Click "Next".
6. Configure the output mapping as shown below:
7. Click on "Finish" and then "Save".

The mapping enables the operation "RetrieveS3Object" to read the data from the S3 bucket and output it as a stream.

Configuring the Mapping for the IO Service¶

The "IO" block is configured to convert the stream into bytes and forward it.

1. Open the "Map Input and Output" dialogue of the "IO" block.
2. Configure the input mapping as shown below:
3. Configure the output mapping as shown below:
4. Click "Finish" and then "Save".

The mapping enables the streamToBytes service to convert the input stream into bytes.

Configuring the Mapping for the Flat File Service¶

The "Flat File" block is configured to interpret the bytes and store them in a document with rows and columns.

1. Open the "Map Input and Output" dialogue of the "Flat File" block.
2. Configure the input mapping as shown below:

3. Double-click on the other four fields in "delimitedDataBytesToDocument Input" and fill in the following values:

   Parameter	Selection
   fieldQualifier	"Semicolon"
   textQualifier	"none"
   useHeaderRowForFieldNames	"true"
   Encoding	"windows-1252: Windows Latin"

4. Configure the output mapping as shown below:

5. Click on "Finish" and then "Save".

The mapping enables the delimitedDataBytesToDocument to interpret the bytes it receives as text and store it in a document.

Testing the Integration (optional)¶

1. Click "Test" at the top right.
2. Enter a name for the output document, e.g. "MindSphereAsset_Creation.csv".
3. Click "Run".

The integration is executed in real time and the results are displayed on the "Test Results" panel. Make sure the success message is shown and verify that the data from the CSV file is displayed in the document.

Integrating the AWS Cloud into Insights Hub¶

The following steps show how to set up an automatic pipeline to create an asset in Insights Hub, which has an aspect with the variables given in CSV file. This pipeline automatically creates the required aspect type, asset type and asset.

Creating an Account for Connecting to Insights Hub¶

If you already have an account for the "Siemens Insights Hub" application in MindConnect Integration, jump to Adding Insights Hub Operations.

1. Open the application "Siemens Insights Hub" in MindConnect Integration.
2. Click on "Add new account".
3. Enter an account name, e.g. "MindSphere_AWS_Integration".
4. Leave the other settings as default and click "Save".

MindConnect Integration can now import data into your Insights Hub tenant.

Adding Insights Hub Operations¶

The pipeline requires operations for the "Siemens Insights Hub" application to create assets, asset types and aspect types, as well as read aspect types.

Create these operations using the configuration details listed below. The required steps are the same as described above.

Setting up the Pipeline for Creating Aspect Types¶

This pipeline shall create a new aspect type, if an aspect type of this name does not exist yet.

1. Add a new orchestrated integration.

2. Set up the integration "MindSphereAspectType_AWS" as shown below using the following blocks:

   Group	Block name	Configuration
   Control Flow	try catch	-
   Applications	Siemens MindSphere 3.0	Account: "MindSphere_AWS_Integration" Operation: "CreateAspectType_AWS"
   Applications	Siemens MindSphere 3.0	Account: "MindSphere_AWS_Integration" Operation: "GetAspectType_AWS"

   

3. Click "Save".

Configuring the Input/Output Signature¶

Define the input signature as shown below in order to fill the required fields for creating aspect types using input parameters.

1. Open the "Define Input/Output Signature" dialogue of the "MindSphereAspectType_AWS" block.

2. Create 3 input fields of type "String" and 1 input field of type "Document" as shown below:

   

3. Click on "Apply" and then "Save".

Configuring the Mapping¶

The first "Siemens Insights Hub" block is configured to fill all required fields for creating a new aspect type. The second "Siemens Insights Hub" block is configured to check if an aspect type with the given aspectTypeId already exists on the tenant.

1. Configure the input mapping for the "Siemens Insights Hub" block in the "try" section according to the figure and table below:

   

   Field	Value
   category	"dynamic"
   scope	"private"
   searchable	"true"
   length	Do not set any value
   qualitycode	"false"

2. Configure the output mapping as shown below:

3. Click "Finish" and then "Save".

4. Configure the input mapping for the "Siemens Insights Hub" block in the "catch" section as shown below:

   

5. Configure the output mapping as shown below:

6. Click "Finish" and then "Save".

Testing the Pipeline for Creating Aspect Types (optional)¶

Test the integration by manually providing the input values as shown below:

Setting up the Pipeline for Creating Asset Types¶

1. Add a new orchestrated integration.

2. Set up the integration "MindSphereCreateAssetType_AWS" as shown below using the following blocks:

   Group	Block name	Configuration
   Control Flow	for each	-
   Services	String	Service: "concat"
   Services	String	Service: "concat"
   Applications	Siemens MindSphere 3.0	Account: "MindSphere_AWS_Integration" Operation: "CreateAssetType_AWS"

   

3. Click "Save".

Configuring the Input/Output Signature¶

Define the input signature as shown below in order to fill the required fields for creating asset types using input parameters.

1. Open the "Define Input/Output Signature" dialogue of the "MindSphereCreateAssetType_AWS" block.

2. Create 2 input fields of type "String" and 1 input field of type "Document" as shown below:

   

3. Click on "Apply" and then "Save".

4. Select "/document/rows" for the input field of the "for each" block.
5. Click "Save".

Configuring the Mapping¶

The "String" blocks are configured to construct a string analogous to {tenantName}.{assetTypeId}. The "Siemens Insights Hub" block is configured to fill all required fields for creating an asset type.

1. Configure the input mapping for the upper "String" block according to the figure and table below:

   

   Field	Value
   inString1	"."

2. Configure the output mapping as shown below:

3. Click "Finish" and then "Save".
4. Configure the input mapping for the lower "String" block according to the figure below:
5. Configure the same output mapping as in step 2.
6. Click "Finish" and then "Save".

7. Configure the input mapping for the "Siemens Insights Hub" block as shown below:

   

   Field	Value
   scope	"private"
   parentTypeId	"core.basicasset"

8. Configure the output mapping as shown below:

9. Click "Finish" and then "Save".

Testing the Pipeline for Creating Asset Types (optional)¶

Test the integration by manually providing the input values as shown below:

Setting up the Pipeline for Creating Assets¶

This pipeline retrieves data from AWS and transfers it into Insights Hub as aspect data of an asset.

1. Switch to the tab "Develop".
2. Add a new orchestrated integration.

3. Set up the integration "MindSphereCreateAsset_AWS" as shown below using the following blocks:

   Group	Block name	Configuration
   Integrations	My_S3_Bucket	-
   Control Workflow	Transform Pipeline	-
   Integrations	MindSphereAspectType_AWS	-
   Control Workflow	Transform Pipeline	-
   Integrations	MindSphereCreateAssetType_AWS	-
   Services	String	Service: "concat"
   Services	String	Service: "concat"
   Applications	Siemens MindSphere 3.0	Account: "MindSphere_AWS_Integration" Operation: "CreateAsset_AWS"

   

4. Click "Save".

Configuring the Input/Output Signature¶

The pipeline receives variable definitions from a CSV file in AWS. In order to create an asset with these variables, the user must provide required parameters which cannot be read from the input file and specify the input file. The user input is defined in the input signature as shown below.

1. Open the "Define Input/Output Signature" dialogue of the "MindSphereCreateAsset_AWS" block.

2. Create 4 input fields of type "String" as shown below:

   

3. Create 1 required output field of type "String" and name assetId (optional).

4. Click on "Apply" and then "Save".

Configuring the Mapping¶

The following configurations enable the integration to read an input file from AWS and create an aspect type with the variables provided in the file. Afterwards, the integration creates an associated asset type and instantiates it.

My_S3_Bucket_Pipeline¶

This block retrieves the user defined CSV file and forwards the content as document.

1. Configure the input mapping for the "My_S3_Bucket_Pipeline" block according to the figure below:

   

2. Configure the output mapping as shown below:

3. Click "Finish" and then "Save".

Upper Transform Pipeline¶

This block reads the first entry in the document and forwards it as AspectName.

1. Open the mapping dialogue for the upper "Transform Pipeline" block.
2. Add a new field of type "String" and name AspectName in the Pipeline Output.
3. Configure the mapping as shown below:
4. Click "Finish" and then "Save".

MindSphereCreateAspectType_AWS¶

This block creates an aspect type using the AspectName on the user defined tenant and forwards its configuration as document.

1. Configure the input mapping for the "MindSphereCreateAspectType_AWS" block according to the figure below:

   

2. Click "Next", "Finish" and then "Save".

Lower Transform Pipeline¶

This block reads the aspectTypeId and name of the aspect type and forwards them as aspectTypeIdArray.

1. Open the mapping dialogue for the lower "Transform Pipeline" block.
2. Add a new field of type "Document" and name aspectTypeIdArray in the Pipeline Output.
3. Add 2 fields of type "String" and names aspectTypeId and name in this document.
4. Configure the mapping as shown below:
5. Click "Finish" and then "Save".

MindSphereCreateAssetType_AWS¶

This block creates an asset type using the aspectTypeIdArray on the user defined tenant and forwards its details as document. The assettypeId is given by the input signature.

1. Configure the input mapping for the "MindSphereCreateAssetType_AWS" block according to the figure below:

   

2. Click "Next", "Finish" and then "Save".

Upper String¶

This block creates a string starting with "." followed by the assetTypeId and forwards it as assetTypeIdWithPrefix.

1. Configure the input mapping for the "MindSphereCreateAspectType_AWS" block according to the figure and table below:

   Field	Value
   inString1	"."

2. Create a new field of type "String" and name assetTypeIdWithPrefix "Pipeline Output".

3. Configure the output mapping according to the figure below:
4. Click "Finish" and then "Save".

Lower String¶

This block creates a string starting with tenantPrefix followed by assetTypeIdWithPrefix and forwards it as assetTypeIdWithPrefix.

1. Configure the input mapping for the "MindSphereCreateAspectType_AWS" block according to the figure below:
2. Configure the same output mapping as for the upper "String" block.
3. Click "Finish" and then "Save".

CreateAsset_AWS¶

This block creates an asset using the assetTypeIdWithPrefix on the user defined tenant. The assetName is given by the input signature and the parentId is set to a fixed value.

1. Configure the input mapping for the Siemens MindSphere" block according to the figure and table below:

   Field	Value
   parentId	Enter the ID of the desired parent asset

   Parent ID

   Open the desired parent asset in the Asset Manager and get the parentId from the URL as shown below.

2. Click "Next", "Finish" and then "Save".

Testing the Pipeline for Creating Assets (optional)¶

Test the integration by manually providing the input values as shown below:

In addition to verifying the results in "Test Results" window, you can open the Asset Manager from the MindSphere Launchpad in order to inspect the created asset.

Related Links¶

• MindConnect Integration Documentation

Except where otherwise noted, content on this site is licensed under the Development License Agreement.