

subscription-service/v1

Overview¶

The subscription-service enables event-driven communication between applications and datasources. It provides an asynchronous interface for communication in a "fire and forget" manner. You can use this service to:

Write data and events to the databus
Define the provided data's message format
Enable the available datasources to send data in a specified quality

To use this service, provider and consumer applications need to define the provided/required data in their configuration and implement the databus client library for sending/receiving data.

creating a consumer application¶

Create the configuration¶

Hint: This section describes the developer guideline for creating an application which uses the subscription-service. Information how to configure an application instance can be found in the operator manual.

Example configurations¶

Sample applications inside AppSDK delivery contain sample meta_config files. Below link describes where to find sample applications mentioned:

AppSDK package content.

Schema¶

requiredDatasource

element	rule	type
subscriptions	required, shall not be empty	array of object
- messageId	required	string
- messageName	required, shall not be empty	string
- quality	required	string
- description	optional	string
- meta	optional	object
- datapoints	required, shall not be empty	array of object
-- address	required, shall not be empty	string
-- datapointName	optional	string
-- description	optional	string
-- meta	optional	object

messageId: The identifier of a message object in the metaconfig. The messageId must match a provided messageId in the providedDatasource config. It is used to connect to available messages in a providedDatasource. May be changed depending on the documentation provided by the consumer application.
- provided by: metaconfig
- used by: configuration process
messageName: The identification of a databus message at application runtime. Must be unique in a requiredDatasource Object. Used to subscribe to the databus.
- provided by: metaconfig
- used by: consumer application at runtime
quality: The definition of an available data quality. Is defined by the providedDatasource message Object.
- provided by: metaconfig
- used by: configuration process, consumer application and datasource at runtime
description: Descriptive text to support the operator during the configuration process based on the application's documentation. Can be removed after configuration.
- provided by: metaconfig
- used by: operator
meta: Object for application-specific configuration. Can be changed by the operator depending on the application's documentation.
- provided by: metaconfig
- configured by: operator
- used by: consumer application at runtime
datapoints: The contents of a subscription. Available datapoints shall be defined by the provided datasource.
- provided by: metaconfig
- configured by: operator
- used by: consumer application at runtime
address: Data identifier, specific to the provided datasource. May be changed by the operator.
- provided by: datasource
- configured by: operator
- used by: provided datasource and consumer application at runtime
datapointName: Data identifier used to address a configured datapoint at application runtime. Must be unique.
- provided by: metaconfig
- used by: consumer application at runtime

Step 1 – find the available data¶

To get started, you must refer to the provider application's documentation. As a rule of thumb, it is a good idea to have a look in the metaconfiguration file of this application and refer to the subscription-service in the providedDatasource section. There you can find all messages that are subscribable.

example metaconfig file of a datasource providing application

{
    "datasourceConfig": 
    { 
        "providedDatasource":[
        { 
            "datasourceId": "DATASOURCE_01", 
            "type" : "SAMPLE_DATA",
            "services": 
            { 
                "subscription-service/v1":
                { 
                    "subscriptions": [
                    { 
                        "messageId": "data_trigger",
                        "quality": "sample_event",
                        "merge":false, 
                        "datapoints": [ 
                            {"address": "CTRL_DATAPOINT($id)"},
                            {"address": "CURR_DATAPOINT($id)"} 
                        ] 
                    }] 
                }
            } 
        }] 
    }
}

In this example you see one available message with the ID “data_trigger” and the quality “sample_event”. This message includes two datapoints that you can subscribe to.

To subscribe to an available datapoint, fill the requiredDatasource section of your config file with the relevant information. Giving them unique names via “messageName” will help you identify the data at runtime.

example how to subscribe to messages with the Id data_trigger.

{
  "datasourceConfig": {
    "requiredDatasource": [
      {
        "datasourceId": "DATASOURCE_01",
        "type": "SAMPLE_DATA",
        "services": {
          "subscription-service/v1": {
            "subscriptions": [
              {
                "messageId": "data_trigger",
                "messageName": "myMessageName",
                "quality": "sample_event"
              }
            ]
          }
        }
      }
    ]
  }
}

Your application is now connected to the data_trigger message from DATASOURCE_01. Your application will receive this message with the message_id: "myMessageName".

Attention: The messageName must be unique in the application's config. If a combination of machineConfig and datasourceConfig is used, messageNames in the subscription-service must be named differently.

Using the example provider from Step 1, input the required data in the "datapoints" array.

Attention: If merge=true in the provider app's config, all required datapoints of this messageID will be put into the same message packet. As such, merge=true means that there can be more data than expected in the received message, depending on the subscription configuration of other consumer applications. If merge=false, a single message for each datapoint will be created.

Attention: A subscription to two identical datapoints will be merged into a single subscription. You will not receive duplicate datapoints in your application.

example: showing subscription to the CTRL_DATAPOINT parameter of index 1 and 2:

{
  "datasourceConfig": {
    "requiredDatasource": [
      {
        "datasourceId": "DATASOURCE_01",
        "type": "SAMPLE_DATA",
        "services": {
          "subscription-service/v1": {
            "subscriptions": [
              {
                "messageId": "data_trigger",
                "messageName": "myMessageName",
                "quality": "sample_event",
                "datapoints": [
                  {
                    "address": "CTRL_DATAPOINT(1)"
                  },
                  {
                    "address": "CTRL_DATAPOINT(2)"
                  }
                ]
              }
            ]
          }
        }
      }
    ]
  }
}

Attention: It is not allowed to define two messages with identical quality and datapoints. For technical reasons, such configurations will be handled as a configuration error.

example: configuration error:

            "subscriptions": [
              {
                "messageId": "data_trigger",
                "messageName": "myMessageName1",
                "quality": "sample_event",
                "datapoints": [
                  {
                    "address": "CTRL_DATAPOINT(1)"
                  }
                ]
              },
              {
                "messageId": "data_trigger",
                "messageName": "myMessageName2",
                "quality": "sample_event",
                "description":"This message is a duplicate of myMessageName1 and will be handled as configuration error!"
                "datapoints": [
                  {
                    "address": "CTRL_DATAPOINT(1)"
                  }
                ]
              }
            ]

subscription configuration using wildcards

If your subscription requires specific information about the environment that is only known at configuration time, you can use wildcards in your metaconfig. Usually you do not know the settings of the machine with which your application will communicate. As such, the operator must specify the correct parameter address during deployment (in the Insights Hub, for example). To support the operator, you may specify wildcards in your datapoint declaration. To define wildcards, you can either use the wildcard section in the datasourceConfig, use a description text in the datapoint definition, or a combination of both. See the two examples below.

example 1:

{
  "datasourceConfig": {
    "wildcards": [
      {
        "key": "($index)",
        "description": "array index of the parameter"
      }
    ]
  }
}

example 2:

{
  "datapoints": [
    {
      "address": "CTRL_DATAPOINT($id)",
      "description": "configure one datapoint for each required ID"
    }
  ]
}

Attention: You can use the metaconfig as documentation to help the user configure an instance of your application. To do so, handle the "description" and wildcard elements with care. In general, it is considered best practice to always add additional information to document your application.

optional: customize¶

You can use the datasourceConfig "meta" element to include additional datasource-specific information. This may be used in the sections "requiredDatasource", "subscriptions", and "datapoints" and will be copied to the appconfig file at configuration time.

example:

{
  "datasourceConfig": {
    "requiredDatasource": [
      {
        "datasourceId": "DATASOURCE_01",
        "type": "SAMPLE_DATA",
        "meta": {"key":"value"},
        "services": {
          "subscription-service/v1": {
            "subscriptions": [
              {
                "messageId": "data_trigger",
                "messageName": "myMessageName",
                "quality": "sample_event",
                "meta": {"key":"value"},
                "datapoints": [
                  {
                    "address": "CTRL_DATAPOINT(1)",
                    "meta": {"key":"value"},
                  },
                  {
                    "address": "CTRL_DATAPOINT(2)",
                    "meta": {"key":"value"},
                  }
                ]
              }
            ]
          }
        }
      }
    ]
  }
}

In this example, we use the keyword “meta” for custom information. The consumer application can read the appconfig file to retrieve this specific configuration data.

identify configured parameters¶

The operator is able to change the address of a parameter. The application might still have the requirement to identify configured parameters beside the parameter address. For that reason, you can use the optional element datapointName to define an unique name to a datapoint .

example:

"datapoints": [{
    "address":"some generic address", 
    "datapointName":"MyParameter1" 
}]

usage of the APP-SDK interface¶

Attention: There is currently no APP-SDK support for reading data from the appconfig file. The appconfig.json file in the application space has to be used directly.

During the configuration process the appconfig file in your application's container will include the valid message configuration. Messages will now have the defined messageName as Id.

example appconfig after configuration

{
  "datasourceConfig": {
    "requiredDatasource": [
      {
        "datasourceId": "DATASOURCE_01",
        "type": "SAMPLE_DATA",
        "meta": {"key":"value"},
        "services": {
          "subscription-service/v1": {
            "subscriptions": [
              {
                "messageId": "myMessageName",
                "quality": "sample_event",
                "meta": {"key":"value"},
                "datapoints": [
                  {
                    "address": "CTRL_DATAPOINT(1)",
                    "meta": {"key":"value"},
                  },
                  {
                    "address": "CTRL_DATAPOINT(2)",
                    "meta": {"key":"value"},
                  }
                ]
              }
            ]
          }
        }
      }
    ]
  }
}

This config defines one message with the ID "myMessageName". The application must register a callback for this messageId using the databus client library to receive messages from a provider application.

Creating a provider application¶

Create a configuration¶

Every application that writes messages to the databus should define a subscription-service in the providedDatasource section of the application's configuration. This enables client applications to subscribe to the provided messages.

providedDatasource

element	rule	type
subscriptions	required, shall not be empty	array of object
- messageId	required	string
- description	optional	string
- merge	optional – default: false	boolean
- quality	required	string - shall not include space characters
- datapoints	required, shall not be empty	array of object
-- address	required, shall not be empty	string
-- description	optional	string

messageId: The identifier of a message object in the metaconfig. It is used to connect consuming applications to available messages
- provided by: metaconfig
- used by: configuration process
quality: The definition of an available data quality.
- provided by: metaconfig
- used by: configuration process, consumer application and datasource at runtime
- attention: space characters are not supported. Use underscore '_' instead.
description: Descriptive text to support the operator during the configuration process based on the application's documentation. Can be removed after configuration.
- provided by: metaconfig
- used by: operator
merge: Option to change the datapoint subscription for that specific message. Shall not be changed by the operator!
provided by: metaconfig
used by: configuration process
datapoints: Can be used as documentation of available datapoints and at least one is required. Shall not be changed by the operator!
provided by: metaconfig
used by: operator
address: Data identifier, specific to the provided datasource.
- provided by: datasource
- used by: operator

Configure the message generation¶

As a data provider, input the subscribable data in your application's configuration file. This will not only let consumer applications know what data is available, but the operator who controls application communication during configuration time as well. Note that the defined metaconfiguration will not be identical to the application's configuration at runtime. Your application's configuration file at runtime will only contain the actual amount of messages and datapoints required by consumer applications. The configuration process automatically creates topics, permissions and routing information for the databus. As such, the provider application can define how those messages should be created:

Option 1: merge

All consumer applications that require a subscription including merge=true will receive the same message. This means that the message will contain every consumer application's subscribed data. To do so, use the element “merge”:true in the subscription definition of the subscription-service. As a result, the appconfig of the provider application will contain only one message with the original messageId.

example:

"datasourceConfig": 
{ 
    "providedDatasource":
    [{ 
        "datasourceId": "DATASOURCE_01", 
        "type" : "SAMPLE_DATA", 
        "services": 
        {
            "subscription-service/v1":{ 
                "subscriptions": [{ 
                    "messageId": "data_trigger", 
                    "quality": "sample_event", 
                    "merge":true, 
                    "datapoints": [ {
                        "address": "CTRL_DATAPOINT($id)"
                    }] 
                }] 
            } 
        } 
    }] 
}

Option 2: default

The default behavior merge=false will optimize the subscriptions to create a single message for each application. With this option, the data provider application may need to copy datapoints into multiple messages. As a result, the appconfig will contain multiple messages with automatically generated messageIds.

Attention: If defining a providedDatasource, always document how to use the provided data. If you enhance the predefined message format, please provide a JSON schema file for those messages.

Produce messages using the databus¶

Step 1: Read the list of configured messageIds from the subscription-service/v1 section in the appconfig.json file.

Step 2: Create a databus producer instance from databus client library for the configured messageIds.

Step 3: Read the list of required datapoints out of your application's configuration.

Step 4: Implement your application to retrieve required datapoints read from step 3.

Step 5: Push the required data to the databus. With the “merge”:false option, your application might have to copy datapoints to multiple messages.

Databus message format definition¶

To enable portability and decouple applications from app-specific data, messages pushed to the databus by the subscription-service have a predefined format.

The message payload contains head and body sections as follows:

{
  "header": {

  },
  "body": {

  }
}

Type Object, mandatory

Usage Information regarding the whole block that is necessary to pack and unpack the body.

The header includes message_type, key and source_id:

example

"header" : {
  "message_type": "dataobject",
  "key": "dataobject", 
  "source_id": "the source of the data"
}

source id: This is interface-specific per component and unique in overall catalogue

key: Can be used by the provider application to identify a unique key of the message

message_type: Currently, only dataobject is supported.

body¶

The Body section includes an array of datapoint objects and can be extended by application-specific metadata.

example 1:

{
  "body": {
    "datapoint": [
      {
        "address": "Parameter1[#2]",
        "quality_mode": "sample",
        "period_ms": -1.0000000,
        "timestamp": "2015-04-09T11:40:59.011739",
        "content": [
          {
            "value": [ 0 ]
          },
          {
            "value": [ 2 ]
          }
        ]
      }
    ],
    "someMetaObject":{}
  }
}

example 2:

{
  "body": {
    "datapoint": [
      {
        "address": "CYCLE",
        "quality_mode": "frequential",
        "period_ms": 2.0000000,
        "timestamp": "2015-04-09T11:40:59.011739",
        "content": [
          {
            "value": [
              0,
              1,
              2
            ]
          }
        ]
      }
    ]
  }
}

datapoint/address: A unique address to identify the parameter.

datapoint/timestamp: Shows timestamp of acquired data in ISO standard 8601.

datapoint/quality_mode: Defines the quality of the data. Available qualities are defined in the metaconfig file of the provider application. Attention: The subscription-service/v1 only supports datapoints of identical quality in one message.

datapoint/period_ms: If the message contains time series data, then the period_ms is used to define the time interval in milliseconds. If not, use period_ms=-1 to define datapoints as not frequential.

datapoint/content: The "content" object contains an array of value objects. Every value object contains an array of values. The array of values is filled when receiving time series data.

Example 2 shows a message that contains 3 time series values with an iteration time of 2 ms. The timestamp for the 3^rd index of the CYCLE datapoint (2) is calculated by adding 2 * period_ms to the timestamp which is 2015-04-09T11:40:59.011739.

The array of data values is also used for data received in the same time period (or not frequential).

Example 1 shows a message which contains 2 values of a parameter array. Each array of data values contains only one value.

Attention: Please always refer to the documentation of the provider application to find details how the required datasource type defines the databus messages.

Hint: In the developer SDK you will find a schema definition file for the databus messages.

Message size limitations: Sending data through the databus consumes RAM and CPU. The message size and the frequency of the messages are the main factors for the memory and processor load. To avoid general overload and occasional peaks in resource consumption, the message size/frequency tradeoff shall be in an optimal range.

Increasing the frequency of messages increases the average CPU consumption.
Increasing the message size increases the CPU peak interval and RAM demand.

Attention: The memory requirement is three times the size of the message itself. We do not recommend sending messages above 50 MB, the theoretical maximum value is around 250 MB.
Assuming an average CPU usage of 70%, the field marked in green in the figure below is the ideal area in message size/frequency tradeoff. For example, instead of sending 10 MB of data every fifth second (which is close to the border of the green area), send 1 MB packets every half second (which is much more inside the green area).

Configuring application instances¶

Details how to configure applications on an EdgeBox instance can be found in the operator manual.

Any questions left?

Ask the community

Except where otherwise noted, content on this site is licensed under the The Siemens Inner Source License - 1.1.

subscription-service/v1

Overview¶

creating a consumer application¶

Create the configuration¶

Example configurations¶

Schema¶

Step 1 – find the available data¶

Step 2 – subscribe to available messages¶

Step 3 – subscribe to available data¶

optional: customize¶

identify configured parameters¶

usage of the APP-SDK interface¶

Creating a provider application¶

Create a configuration¶

Configure the message generation¶

Produce messages using the databus¶

Databus message format definition¶

header¶

body¶

Configuring application instances¶

Any questions left?