Skip to content

Uploading Agent Data

If an agent needs to upload data to MindSphere, MindSphere needs additional configuration to know how to interpret the agent's data stream. This configuration requires the following definitions within MindSphere:

  • Data Source Configuration
  • Property Set Provisioning
  • Mapping for Data Source and Property Set

Preparing Data Upload

Creating a Data Source Configuration

MindSphere needs a data source configuration for interpreting the data it receives from an agent. Without this configuration MindSphere cannot understand the data. The data source configuration contains data sources and data points. Data sources are logical groups, e.g. a sensor or a machine, which contain one or more measurable data points, e.g. temperature or pressure.

Use the Agent Management Service to create the data source configuration as shown below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
PUT /api/agentmanagement/v3/agents/{{agent_id}}/dataSourceConfiguration HTTP/1.1
Content-Type: application/json
If-Match: etag
Authorization: Bearer eyx...
{
    "configurationId": "string",
    "dataSources": [
        {
        "name": "string",
        "description": "string",
        "dataPoints": [
            {
            "id": "string",
            "name": "string",
            "description": "string",
            "type": "int",
            "unit": "string",
            "customData": {}
            }
        ],
        "customData": {}
        }
    ]
}
Parameter Description Remarks
dataSource.name Name of the data sources Mandatory. Any string is allowed.
dataSource.description Description of the data sources Optional. Any string is allowed.
dataPoint.id Data point ID Mandatory. Must be unique per agent. No duplicates are allowed
dataPoint.name Name of the data point Mandatory. Any string is allowed, e.g. pressure, voltage, current, etc.
dataPoint.description Description of the data point Optional. Any string is allowed.
dataPoint.type Data type of the data point.

By default, MindSphere provides the following base types: Double, Long, Int, String, Boolean		 Mandatory
dataPoint.unit Unit of the data point Mandatory. Any string is allowed, e.g. percent, SI, etc. Empty strings are allowed.
dataPoint.customData Custom data, if any will be provided Optional
dataSources.customData Custom data, if any will be provided Optional

If successful, MindSphere returns an HTTP response 200 OK with a JSON body that holds the created data sources configuration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
{
    "id": "string",
    "eTag": "2",
    "configurationId": "string",
    "dataSources": [
        {
        "name": "string",
        "description": "string",
        "dataPoints": [
            {
            "id": "string",
            "name": "string",
            "description": "string",
            "type": "int",
            "unit": "string",
            "customData": {}
            }
        ],
        "customData": {}
        }
    ]
}

Warning

When an existing Data Source Configuration is updated, all Data Point Mappings of the agent are deleted.

Note

For consuming exchange services the parameter configurationId needs to be provided to MindSphere.

Creating a Data Point Mapping

Creating a data point mapping requires not only a data source configuration, but also a corresponding property set in MindSphere.

MindSphere needs a data point mapping for storing the data it receives from an agent. This maps the data points from the data source configuration to properties of the digital entity, that represents the agent. When MindSphere receives data from an agent, it looks up which property the data point is mapped to and stores the data there.

Use the MindConnect API to create the data point mapping as shown below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
POST /api/mindconnect/v3/dataPointMappings HTTP/1.1
Content-Type: Application/Json
Authorization: Bearer eyc..
{
    "agentId": "11961bc396cd4a87a9b26b723f5b7ba0",
    "dataPointId": "DP0001",
    "entityId": "83e78008eadf453bae4f5c7bef3db550",
    "propertySetName": "ElectricalProperties",
    "propertyName": "Voltage"
}
Parameter Description Trouble Shooting
agentId String, validated, null-checked. Must exist in Agent Management. HTTP 400: Agent is empty!

HTTP 404: AgentId, {...} in the data point mapping does not exist!
dataPointId String, validated, null-checked. Must exist in Agent Management and belong to the specified agent's data source configuration. HTTP 400: DataPointId is empty!

HTTP 404: DataPointId, {...} in the data point mapping does not exist in the related agent!
entityId String, validated, null-checked. Must exist in IoT Entity Service. HTTP 400: Entity ID is empty!

HTTP 404: Not Found
propertySetName String, validated, null-checked. Must exist in IoT Type Service and belong to entity type of the specified entity. HTTP 400: PropertySetName is empty!

HTTP 404: Property set not found with name {...}

HTTP 404: Entity type does not own a property set with name {...}
propertyName String, validated. Must exist in IoT Type Service and belong to the specified property set. HTTP 400: PropertyName is empty!

HTTP 404: Property set does not own a property with name {...}

The following aspects have to be considered:

  • Unit and type of data point and property should match.
  • If the property does not have a unit (NULL), the data point unit should be empty string.
  • The selected property cannot be static.
  • One data point cannot be mapped to multiple properties with the same property name.

Before creating a data point mapping, MindSphere executes the following checks:

Create Data Point Mapping workflow and checks

If successful, MindSphere returns an HTTP responds 201 Created with a JSON body that holds the created mapping configuration. It only maps a single data point to a single property. If more mapping is needed, this step must be repeated.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
{
    "id": "4fad6258-5def-4d84-a4c2-1481b209c116",
    "agentId": "11961bc396cd4a87a9b26b723f5b7ba0",
    "dataPointId": "DP0001",
    "dataPointUnit": "%",
    "dataPointType": "INT",
    "entityId": "83e78008eadf453bae4f5c7bef3db550",
    "propertySetName": "ElectricalProperties",
    "propertyName": "Voltage",
    "propertyUnit": "%",
    "propertyType": "INT",
    "qualityEnabled": true
}

Warning

When an existing data source configuration is updated, all data point mappings of the agent are deleted.

Uploading Data

The MindSphere exchange endpoint of the MindConnect API provides the agent with the capability of uploading data to MindSphere. This data can be of type:

  • Time Series
  • File
  • Event

Agents require an access token with the mdsp:core:DefaultAgent scope to upload data.

The format conforms to a subset of the HTTP multipart specification, but only permits nesting of 2 levels.

Mandatory New Line

<CR><LF> at the end of a request represents a mandatory new line (Carriage Return and Line Feed). This representation is used for emphasis in the following samples.
Replace these characters with \r\n when uploading data to MindSphere.

Uploading Time Series Data

Below is a sample request with a multipart message to upload time series data:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
POST {{_gateway_url}}/api/mindconnect/v3/exchange
Content-Type: multipart/mixed; boundary=f0Ve5iPP2ySppIcDSR6Bak
Authorization: Bearer access_token ...

--f0Ve5iPP2ySppIcDSR6Bak
Content-Type: multipart/related;boundary=penFL6sBQHJJUN3HA4ftqC

--penFL6sBQHJJUN3HA4ftqC
Content-Type: application/vnd.siemens.mindsphere.meta+json

{
    "type": "item",
    "version": "1.0",
    "payload": {
        "type": "standardTimeSeries",
        "version": "1.0",
        "details": {
            "configurationId": "{{_configuration_id}}"
        }
    }
}
--penFL6sBQHJJUN3HA4ftqC
Content-Type: application/json

[
    {
        "timestamp": "2017-02-01T08:30:03.780Z",
        "values": [
            {
                "dataPointId": "{{_datapoint_id_1}}",
                "value": "9856",
                "qualityCode": "0"
            },
            {
                "dataPointId": "{{_datapoint_id_2}}",
                "value": "3766",
                "qualityCode": "0"
            }
        ]
    }
]
--penFL6sBQHJJUN3HA4ftqC--
--f0Ve5iPP2ySppIcDSR6Bak--
<CR><LF>

The multipart message starts with the --f0Ve5iPP2ySppIcDSR6Bak identifier. It closes with the --f0Ve5iPP2ySppIcDSR6Bak-- identifier, which is equal to the starting identifier appended by a double dash. After accepting the structure of a multipart request, MindSphere returns an HTTP response 200 OK with an empty body. MindSphere validates and stores data asynchronously. The agent can continue uploading data as long as its access token is valid.

The multipart messages occur within this initial boundary. Each multipart message consists of metadata and a payload, therefore each message contains two boundary start identifiers and a single end identifier at the end:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
--initial boundary

--boundary1
Metadata

--boundary1
Payload

--boundary1--

--initial boundary--
<CR><LF>

Note

The property configurationId in the metadata part of the multipart message tells MindSphere how to interpret the data it receives (see section Creating a Data Source Configuration).

Uploading Files

Agents can upload files into MindSphere, where they are stored in the respective asset. Within MindSphere metadata and payload couples are referred as tuples. Each tuple may contain different type of data (for example one tuple may contain a specific timestamp data, whereas the other may contain octet/stream data). The example below contains an exchange payload for an octet/stream mime type:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
--5c6d7e29ef6868d0eb73
Content-Type: application/vnd.siemens.mindsphere.meta+json

{
    "type": "item",
    "version": "1.0",
    "payload": {
        "type": "file",
        "version": "1.0",
        "details":{
            "fileName":"data_collector.log.old",
            "creationDate":"2017-02-10T13:00:00.000Z",
            "fileType":"log"
        }
    }
}

--5c6d7e29ef6868d0eb73
Content-Type: application/octet-stream

  ... File content here ...

--5c6d7e29ef6868d0eb73--
--f0Ve5iPP2ySppIcDSR6Bak--
<CR><LF>

Uploading Events

Agents can upload events into MindSphere, where they are stored in the respective asset. The events must be of an event type derived from type AgentBaseEvent. The following rules apply for uploading events:

  • The fields eventId and version are required in the payload. Allowed values for the version are: MAJOR, MINOR.
  • The event type must exist in Event Management. Otherwise, the event is dropped.
  • The severity is optional, but events with invalid severity values are dropped. Allowed integer values depend on the version of an event.
    version 1.0: 1 (URGENT), 2 (IMPORTANT), 3 (INFORMATION)
    version 2.0: 20 (ERROR), 30 (WARNING), 40 (INFORMATION)
Model of AgentBaseEvent
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
{ "id": "core.connectivity.event.type.AgentBaseEvent",
  "name": "AgentBaseEvent",
  "ttl": 35,
  "parentId": "com.siemens.mindsphere.eventmgmt.event.type.MindSphereStandardEvent",
  "fields": [
    {
      "name": "eventId",
      "filterable": true,
      "required": true,
      "updatable": true,
      "type": "STRING"
    },
    {
      "name": "version",
      "filterable": true,
      "required": true,
      "updatable": true,
      "type": "STRING"
    }
  ]
}

Here is a sample custom event type which is derived from AgentBaseEvent:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
{
  "name": "FileUploadCompletedEvent",
  "parentId": "core.connectivity.event.type.AgentBaseEvent",
  "ttl": 35,
  "scope": "LOCAL",
  "fields": [
    {
      "name": "fileNumber",
      "filterable": true,
      "required": false,
      "updatable": true,
      "type": "INTEGER"
    },
    {
      "name": "fileSize",
      "filterable": true,
      "required": false,
      "updatable": true,
      "type": "DOUBLE"
    },
    {
      "name": "fileName",
      "filterable": true,
      "required": true,
      "updatable": true,
      "type": "STRING"
    },
    {
      "name": "isValid",
      "filterable": true,
      "required": false,
      "updatable": true,
      "type": "BOOLEAN"
    }
  ]
}

Here is a sample request for event upload:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
--FRaqbC9wSA2XvpFVjCRGry
Content-Type: multipart/related;boundary=5c6d7e29ef6868d0eb73

--5c6d7e29ef6868d0eb73
Content-Type: application/vnd.siemens.mindsphere.meta+json

{
    "type": "item",
    "version": "1.0",
    "payload": {
      "type": "businessEvent",
      "version": "1.0"
    }
}
--5c6d7e29ef6868d0eb73
Content-Type: application/json

[
    {
        "id": "7ba7b810-9dad-11d1-80b4-00c04fd430c8",
        "correlationId": "fd7fb194-cd73-4a54-9e53-97aca7bc8568",
        "timestamp": "2018-07-11T11:06:25.317Z",
        "severity": 2,
        "type": "FileUploadCompletedEvent",
        "description": "file uploaded event",
        "version": "1.0",
        "details":{
            "fileName": "test1",
            "fileSize": 11.2,
            "isValid": "true",
            "fileNumber": 15
        }
    }
]
--5c6d7e29ef6868d0eb73--
--FRaqbC9wSA2XvpFVjCRGry--
<CR><LF>

Replaying an Exchange Request

If MindSphere cannot process data after it has been uploaded via MindConnect API, MindConnect's Recovery Service stores the unprocessed data for 15 calendar days. For a list of the stored unprocessed data, use the recoverableRecords endpoint. It responds with a page of recoverable records as shown below. The response can be filtered by the agentId, correlationId, requestTime and dropReason fields using a JSON filter.

1
2
3
4
5
6
7
{
      "id": "4fad6258-5def-4d84-a4c2-1481b209c116",
      "correlationId": "7had568-5def-4d84-a4c2-1481b209c116",
      "agentId": "33238f98784711e8adc0fa7ae01bbebc",
      "requestTime": "2018-08-27T16:40:11.235Z",
      "dropReason": " <[Dropped] TimeSeries Data is dropped. Validation failed for reason <Pre-processing of <153000> data points ended with <60600> valid and <92400> dropped data points. Following issues found: Data point with no mapping count is <92400>, including <[variable101, variable102, variable103, variable104, variable105, variable106, variable107, variable108, variable109, variable110]>.>>"
}

To inspect a record's payload, the MindConnect API provides the /recoverableRecords/{id}/downloadLink endpoint. The response contains a URL, to which the data is uploaded by MindConnect API, e.g.:

1
2
GET /api/mindconnect/v3/recoverableRecords/{id}/downloadLink` HTTP/1.1
"https://bucketname-s3.eu-central-1.amazonaws.com/c9bcd-44ab-4cfa-a87e-d81e727d9af4?X"

If an invalid mapping or registration has caused the data not to be processed, the exchange request can be replayed after correcting the definition. Use the /recoverableRecords/{id}/replay endpoint of the MindConnect API to trigger a replay.

Note

If the data can successfully be processed after a replay, it is removed from the recovery system after 2 days.

Diagnosing Data Upload

Data uploaded via MindConnect API can be diagnosed with the Diagnostic Service feature. In order to activate an agent for the Diagnostic Service, MindSphere provides the diagnosticActivations endpoint of the MindConnect API, which needs to be used with following parameters:

1
2
3
4
5
6
7
POST /api/mindconnect/v3/diagnosticActivations HTTP/1.1
Content-Type: Application/Json
Authorization: Bearer eyc..
{
    "agentId": "11961bc396cd4a87a9b26b723f5b7ba0",
    "status": "ACTIVE"
}
Parameter Description Trouble Shooting
agentId String, validated HTTP 400: Agent is empty!

HTTP 404: Agent id {...} exists!

Agent id {...} could not be enabled. Because agent limitation is {...} for tenant {...}!
status String, validated
Allowed inputs are: ACTIVE, INACTIVE, NULL. If NULL, value is set as ACTIVE by default.		 HTTP 400: Custom statuses are not allowed. Use one of the allowed input values

Note

The Diagnostic Service can be activated for up to 5 agents per tenant. Diagnostic activation statuses are set to inactive after 1 hour. Activations can be deleted by the diagnosticActivations/id endpoint.

A list of diagnostic activations is provided by the diagnosticActivations endpoint. It responds with a page of diagnostic activations as shown below. The response can be filtered by the agentId and status fields using a JSON filter.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
{
  "content": [
    {
      "id": "0b2d1cdecc7611e7abc4cec278b6b50a",
      "agentId": "3b27818ea09a46b48c7eb3fbd878349f",
      "status": "ACTIVE"
    }
  ],
  "last": true,
  "totalPages": 1,
  "totalElements": 1,
  "numberOfElements": 1,
  "first": true,
  "sort": {},
  "size": 20,
  "number": 0
}

An activation can be updated or deleted using the diagnosticActivations/id endpoint:

1
2
3
4
PUT /api/mindconnect/v3/diagnosticActivations HTTP/1.1
{
  "status": "INACTIVE"
}
Parameter Description Trouble Shooting
status String, validated
Allowed inputs are: ACTIVE, INACTIVE		 HTTP 400: Custom statuses are not allowed. Status value must be one of: [ACTIVE, INACTIVE]

Diagnostic data is listed by the diagnosticInformation endpoint. Its response is a page as shown below, which can be filtered by the agentId, correlationId,message,source,timestamp and severity fields using a JSON filter.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
  "content": [
    {
      "agentId": "3b27818ea09a46b48c7eb3fbd878349f",
      "correlationId": "3fcf2a5ecc7611e7abc4cec278b6b50a",
      "severity": "INFO",
      "message": "[Accepted] Exchange request arrived",
      "source": "TIMESERIES",
      "state": "ACCEPTED",
      "timestamp": "2018-08-27T16:40:11.235Z"
    },
    {
      "agentId": "78e73978d6284c75ac2b8b5c33ffff57",
      "correlationId": "39301469413183203248389064099640",
      "severity": "ERROR",
      "message": "[Dropped] Registration is missing. Data type is: notstandardTimeSeries"
      "source": "TIMESERIES",
      "state": "DROPPED",
      "timestamp": "2018-08-27T16:40:11.235Z"
    },
  ],
  "last": true,
  "totalPages": 1,
  "totalElements": 2,
  "numberOfElements": 2,
  "first": true,
  "sort": {},
  "size": 20,
  "number": 0
}

Diagnostic data of a given activation is provided by the /diagnosticActivations/{id}/messages endpoint. Its response is a page as shown below, which can be filtered by the correlationId,message,source,timestamp and severity fields using a JSON filter.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
{
  "content": [
    {
      "correlationId": "3fcf2a5ecc7611e7abc4cec278b6b50a",
      "severity": "INFO",
      "message": "[Accepted] Exchange request arrived",
      "source": "TIMESERIES",
      "state": "ACCEPTED",
      "timestamp": "2018-08-27T16:40:11.235Z"
    },
    {
      "correlationId": "39301469413183203248389064099640",
      "severity": "ERROR",
      "message": "[Dropped] Registration is missing. Data type is: notstandardTimeSeries"
      "source": "TIMESERIES",
      "state": "DROPPED",
      "timestamp": "2018-08-27T16:40:11.235Z"
    },
  ],
  "last": true,
  "totalPages": 1,
  "totalElements": 2,
  "numberOfElements": 2,
  "first": true,
  "sort": {},
  "size": 20,
  "number": 0
}

Any questions left?

Ask the community

Except where otherwise noted, content on this site is licensed under the MindSphere Development License Agreement.