Skip to content

IoT Time Series Aggregates Service

Announcement

Aggregate V4, new API is now available since June, 2020.. For more information about new features and limitations, please refer Version 4.0 - What's New?. Older version of Aggregate V3 will be available without any support & reduced quota till 31st March'2024. Please refer Migration guide. From 1st March 2022 onwards, New Tenants or Tenants Migrated from older offerings to new Packaging will be provisioned only with Aggregate V4 API.

Idea

The IoT Time Series (TS) Aggregates Service creates aggregated summaries of numeric time series data and provides interfaces to read them. This allows applications to retrieve smaller data sets that cover a long time range with much better performance than processing all the raw time series data.

For example, an aspect could create new data every second, which adds up to ~2.5 million records per month. An application could use the IoT TS Aggregates Service to a summary for each day of the month, obtaining only 30 records.

Access

For accessing this service you need to have the respective roles listed in IoT Time Series Aggregates Service roles and scopes.

A user can only read data within their environment and subtenants.

For accessing the Secure Data Sharing (SDS) protected APIs you need to have appropriate Policy Definitions in place. Please refer here for the list of supported APIs and Required Actions.

Basics

Hint

In the IoT context, assets are referred to as entity and aspects as propertyset.

Aggregated Data

The data summarized in an aggregate is the data with timestamps greater than the start time and less than or equal to the end time of the interval. This ensures that count type data is correctly reflected in the aggregate. For example, if a device provides data at the end of every five minutes, the hourly aggregation includes the data with timestamps from five minutes after the hour until the end of the hour (e.g., 3:05 to 4:00).

Each aggregate summarizes the following information:

  • value and time of the first data point of the interval
  • value and time of the last data point of the interval
  • value and time of the minimum data point during the interval
  • value and time of the maximum data point during the interval
  • the count of data points during the interval
  • the sum of the values during the interval
  • the average of the values during the interval
  • the count of the good, uncertain, and bad values during the interval
  • standard deviation of the values within the interval

Pre-Calculated Aggregates

The IoT TS Aggregates Service automatically creates pre-calculated aggregates and stores them for later retrieval to enhance performance of aggregate queries. Aggregates of the following interval lengths are pre-calculated:

Interval Length Description
1 millisecond Aggregates end at 1 millisecond after the hour, 2 milliseconds after the hour, and so on.
This interval length is only available on simulation entities.
10 milliseconds Aggregates end at 10 milliseconds after the hour, 20 milliseconds after the hour, and so on.
This interval length is only available on simulation entities.
1 second Aggregates end at 1 second after the hour, 2 seconds after the hour, and so on.
This interval length is only available on simulation entities.
1 minute Aggregates end at 1 minutes after the hour, 2 minutes after the hour, and so on. They are created within two minutes after the one-minute interval has ended.
This interval length is only available on performance entities.
1 hour Aggregates end at the top of the hour. They are created within eight minutes after the end of the hour.
1 day Aggregates end at midnight in the asset's time zone. They are created within fifteen minutes after midnight in UTC time.

The aggregated data is wall clock aligned based on the asset's time zone.

Note

Aggregated data is not immediately available. E.g., if aggregated data of a performance entity is requested at 10:00 with interval length of 1 hour, the response will not contain the aggregate from 9:00 to 10:00, since it has not been calculated yet.

On Demand Aggregates

On demand aggregates are only available on performance entities. They are created when a client requests aggregated data, which cannot be calculated using pre-calculated aggregates (e.g. interval length of 1 minute or starting at 10:01). The IoT TS Aggregates Service uses suitable pre-calculated aggregates and calculates the missing items on demand.
The following interval lengths are supported:

  • 1-120 seconds
  • 1-60 minutes
  • 1-24 hours
  • 1+ days
  • 1+ weeks
  • 1+ months

Aggregate Creation

Data Availability

On performance entities, aggregated data is only created if time series data exists for that time range. Aggregates are created based on the data that is available at the time. Data that arrives late is not included in the initial aggregated results, but triggers a recalculation after a delay.

On simulation entities, aggregated data is created when the ingestion job for time series data is completed. If data for the same hour is ingested by another job, the respective aggregates are re-calculated.

Quality Code Data

If a variable is configured to contain quality code data, the service uses it to rate the aggregates. The quality code is compared to the good and uncertain thresholds defined in the environment (by default the values are set according to the OPC UA standard).

Good data is defined as anything less than or equal to the goodThreshold value.
Uncertain data is defined as anything greater than the goodThreshold value and less than or equal to the uncertainThreshold value.
Bad data is defined as anything greater than the uncertainThreshold.

Only good and uncertain data is used to calculate the summary values listed above. Bad data is ignored for these calculations. Each aggregate contains counts of the three types of data: countgood, countuncertain, and countbad. If no quality code data is available for a time series record, it is considered good.

Features

The IoT TS Aggregates Service exposes its API for realizing the following tasks:

  • Read aggregated interval data using pre-calculated aggregates
  • Generate aggregated interval data for custom intervals

Limitations

  • On demand aggregates can only be created from up to 5,000 raw time series entries.
  • On demand aggregates are only available for performance entities.
  • On the fly aggregate query response is slower than pre-calculated aggregate request.
  • Aggregates are not supported for variables with data type "STRING", "BIG_STRING", "TIMESTAMP".
  • A query can only return up to 200 aggregate intervals in one response.
  • Time zones that are 15 or 45 minutes off a UTC hour (e.g. Nepal standard time (UTC+05:45)) are not supported. Time zones that are an integer number of hours off GMT or 30 minutes off a UTC hour (e.g. India (UTC+05:30)), are supported.
  • The '+' character used to include timezone in timestamp must be encoded to '%2B' in time range. For example, 2017-09-21T14:08:00%2B02:00 instead of 2017-09-21T14:08:00+02:00.

To get the current list of restrictions go to release notes and choose the latest date. From there, go to "MindAccess Developer Plan Subscribers and MindAccess Operator Plan Subscribers" and pick the IoT TS Aggregates Service.

Example Scenario

A wind turbine produces time series data for the speed of the blades. The data is sent continuously to Industrial IoT.

The IoT TS Aggregates Service API allows you to collect aggregated data for the speed of the windmill.


Last update: December 4, 2023

Except where otherwise noted, content on this site is licensed under the Development License Agreement.