Create and edit data set¶

The data sets used as input for process data or quality data can be created using the Data Set Manager. This tool can be accessed by clicking the corresponding button in the left menu bar of the application or by selecting the data set in the relevant view within the Model Setup Wizard, as discussed in the previous chapter. In the initial view of the Data Set Manager, a list of available data sets is displayed allowing to edit existing data sets or to create the new one.

To create a new data set, follow these steps:

1. Click "Add dataset".

   

2. Select the data source from the options “Assets” or “Data Lake file”.

   

3. For an asset data source, proceed by selecting the aspects and asset variables. Multiple assets and aspects can be chosen as the data source. Refer to the Requirements and Limitations section for details on application constraints.

   

   After selecting the variables, the chosen variables and their current values are displayed in the preview window.

   

4. For an asset data source, begin with selecting the CSV file from the integrated data lake and defining the types of variables. Numerical types, such as integer and double, can later be used as input parameters for machine learning models. Categorical data types serve as supporting data for the models, for example as identifiers or filter variables. The variables to be included in the dataset should be selected using the corresponding checkboxes. The time variable representing the X-axis of the charts can be selected from a dropdown menu containing all variables of the Timestamp type. The data can be displayed by clicking on “Preview” button in the preview window.

   

5. Apply one of the following options to handle missing values in the data.

   

6. In the filtering menu, click on “Add parameter” to add a filtering variable. A time-based filter can be used to specify a time range, narrowing down the data for model training. Value-based filters can help to select data from productive process phases while excluding unproductive data. Alternatively, specific data ranges for selected variables can be defined for model training.

   You can apply the following comparison operations to filter values: =, !=, >, <, >=, <=. Additionally, the IN operation can be used to check if the values of a selected variable are within a defined list. To handle outliers, the z-score method can be applied to remove extreme values in the selected variables.

   

7. In the aggregation menu, the data can be pre-processed by grouping and generating aggregated features. To perform aggregation, select the aggregation attribute and the methods to be applied to the data columns. Data aggregation can be used to extract features from the data by grouping based on selected attributes. Functions such as Mean, Standard Deviation (Std), Minimum (Min), Maximum (Max), First, Last, Count, Sum, and Range (Max-Min) can be applied to the dataset variables. These functions are useful for aggregating data over the process cycles of a cyclic process. The Range function, when applied to a column with the Timestamp type, extracts the duration of the grouping cycle. It is recommended to extract the Max or Last value from the column representing the timestamp to determine the end time of the production cycle.

   

8. Finalize the dataset in the last step of the setup dialog by entering the name of the dataset and adding tags to help with searching for it. It is possible to set a maximum number of data points for your dataset in the advanced settings menu to trim the dataset and ensure it adheres to the specified data point limit.
   Store the data set by clicking on the corresponding button.

   

9. The dataset can be located in the list view using the sorting and search functions at the top of the view. By selecting the dataset, general information, a data preview with a chart, and an overview of the data materializations in the logging tab will be displayed.