Evaluate the Model training results¶

To view and evaluate the training results, follow these steps,

1. Model accuracy

   Evaluate the model accuracy with selected metrics with Regression score(R²), Root Mean Square Error(RMSE) and Mean Absolute Error(MAE) for the training and testing data sets.

   

   Upon completion of the training, the training results is available after a certain period of time. The status of the model changes to "Training Completed".

   The model accuracy metrics is represented by the set of characteristics to quantify the quality of the model results aligned to the quality data for the training and testing data sets. The application uses 75% of the overall training data to train the model and remaining 25% to test the model accuracy.

   The R² accuracy score is a metric that quantifies the correlation between predicted and actual results. The values for the test dataset should be as high as possible, approaching 100% in the ideal case. The root mean square error and mean absolute error describing deviations of the predictions and true results should be as low as possible.

2. Prediction

   Navigate to “Prediction” to review the training results.

   

   Review the comparison of prediction and true results for the training data set as Runtime chart, Scatter chart and Deviations histogram. The runtime chart displays predicted and actual quality results on a timeline. The correlation between prediction and actual results is visualized at the correlation chart, where each data point should be as close as possible to the green line representing a perfect fit.

   The histogram on the chart displays statistical distribution of the deviations of predictions from the actual results. A well-trained model leads to acceptable level of prediction deviation and actual results without critical outliers.

   If the accuracy metrics is not sufficient or the model generates too many outliers, Click “Edit" to adjust the model setting. Click "Save as new version" to save model, if the model accuracy and the visual results are acceptable.

   Options "Save as new version" and "Edit" are available at the Training Board until the model is in the “Draft” mode.

3. Feature importance

   Navigate to “Feature Importance” to review the impact of the selected process features on the quality result.

   Review the analysis of the feature importance to define the process parameters having the major impact on the predicted quality result.

   

   In the above shown image,

   Importance chart displays the feature importance obtained by permutation of the input features.

   Feature importance (SHAP-values) chart displays SHAP values for each input feature. The SHAP value reflects the how a single feature affects the prediction result. The process parameters with the highest importance and highest SHAP value have the most impact on the quality result. For more information refer evaluation of feature importance.

   Feature effects (SHAP-values) chart displays the distribution of the SHAP values for each data point in the training data set. The color of the data point represents the feature value and the position on the axis refers to the positive or negative impact on the predicted result.

4. Input Data

   Navigate to “Input Data” to review the graphs of the process features and the quality results.

   

   

   Proceed with analysis of the process parameters and quality results used for the model training as timeseries charts. The charts visualize the process and quality data used to train the model. The charts are interactive, magnifiable and navigate to functionalities to analyze the data points.

5. Data Analysis

   Navigate to “Data Analysis” to analyze the distributions and correlation between the process features and quality result.

   

   Review the distribution Histograms of the process features, Training inputs (represented by aggregated and additional features) and correlation heatmap showing the Correlation factor for the pair of parameters.

   In case of any substantial outliers or unexpected parameter distribution are detected in the distribution charts, it is suggested to edit the model draft or to create new version to remove outliers by corresponding setting in the model setup.

6. Configuration

   Navigate to "Configuration” to view the configuration parameters of the ML model.

   

   The whole set of the model parameters can be downloaded as a file with the export function.