Society of Actuaries (SOA) PA Practice Exam

How can one assess the RMSE on the test data compared to the training data?

1. By calculating the absolute RMSE difference

2. By observing the RMSE trend over epochs

3. By assessing the percent increase in RMSE

4. By comparing model accuracy percentage

The correct answer is: By assessing the percent increase in RMSE

To assess the Root Mean Square Error (RMSE) on the test data compared to the training data, evaluating the percent increase in RMSE provides a meaningful insight into how well the model generalizes to unseen data. When RMSE is calculated for both the training set and the test set, a percent increase can show the degree to which the model's performance degrades when faced with new, unseen data compared to the data it was trained on. This metric helps in understanding overfitting. If the RMSE on the training data is significantly lower than on the test data, this indicates that the model has learned the noise in the training dataset rather than the underlying patterns, which is what overfitting means. Therefore, assessing the percent increase gives quantifiable evidence of the model's ability to generalize and highlights any issues in the training process that need to be addressed, such as adjustments in model complexity, data preprocessing, or selection of features. While other methods such as calculating the absolute RMSE difference might provide a simple numerical disparity, they do not offer insight into the proportional change relative to the training set. Observing the RMSE trend over epochs is useful for monitoring training but does not directly compare performance between training and testing datasets. Additionally, comparing