Evaluating Process (Evaluate Model) in ML - Python for Integrated Circuits - - An Online Book - |
||||||||
Python for Integrated Circuits http://www.globalsino.com/ICs/ | ||||||||
Chapter/Index: Introduction | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z | Appendix | ||||||||
================================================================================= Evaluating machine learning algorithms solely based on their performance on the training set and picking the one with the lowest training set error is not a good practice in model selection. This is because it often leads to a phenomenon called overfitting, and it doesn't provide a reliable assessment of a model's ability to generalize to unseen data:
To address these issues and make more informed decisions about model selection, it's essential to use a separate dataset called the validation set or, preferably, cross-validation. The validation set is used to evaluate and compare different models, and the test set is used for a final, unbiased evaluation of the selected model. This helps ensure that the chosen model is likely to perform well on new, unseen data and doesn't suffer from overfitting. Figure 4114. Vertex AI providing a unified set of APIs for the ML lifecycle. [1] Overfitting is primarily associated with the training phase, where a model learns the patterns and details of the training data too well, including noise and specific examples. However, the term "overfitting" can also be extended to the evaluation phase in certain contexts. Therefore, writing or evaluating the model multiple times against the test dataset can lead to overfitting to the test data and result in an optimistic view of the model's generalization performance. It is crucial to assess the model's ability to generalize to new, unseen data accurately. Here's how repeated evaluations on the test dataset can contribute to a form of overfitting during evaluation:
============================================
[1] Diagram courtesy Henry Tappen and Brian Kobashikawa.
|
||||||||
================================================================================= | ||||||||
|
||||||||