=================================================================================

Excess risk in the context of machine learning refers to the difference between a model's expected or true error and its empirical error on a given dataset. To understand this concept better, let's break it down:

1. True Error (Risk): The true error of a machine learning model represents how well it generalizes to unseen data. It's the expected error the model will make when making predictions on new, previously unseen examples. True error is typically measured using a loss function that quantifies the difference between the model's predictions and the actual target values.

2. Empirical Error: The empirical error is the error a model makes on the dataset it was trained on. It measures how well the model fits the training data.

3. Excess Risk: The excess risk, often denoted as ΔR, is the difference between the true error and the empirical error. It quantifies how much worse the model is expected to perform on unseen data compared to its performance on the training data. In other words, it represents the generalization error of the model.

Mathematically, excess risk can be expressed as:

ΔR = R_true - R_empirical

Where:

• ΔR is the excess risk.
• R_true is the true error (expected error on unseen data).
• R_empirical is the empirical error (error on the training data).

Minimizing excess risk is a fundamental goal in machine learning because the ultimate objective is to build models that perform well on new, unseen data. Models that have low excess risk are better at generalizing, while models with high excess risk may overfit the training data and perform poorly on new examples. Techniques like cross-validation, regularization, and proper model selection are used to help reduce excess risk and improve a model's generalization performance.

============================================

The concept of excess risk in machine learning is related to the power or complexity of the hypothesis class (also known as the model complexity or capacity). The power of the hypothesis class refers to the family of functions or models that a machine learning algorithm can potentially learn. This includes simple models with low capacity (e.g., linear regression) and complex models with high capacity (e.g., deep neural networks).

Here's how the relationship between excess risk and the power of the hypothesis class works:

1. High-Capacity Models: Models with high capacity, such as deep neural networks, have the ability to fit complex patterns and functions in the training data very closely. They can represent a wide range of functions and have low empirical error on the training data. However, if not properly regularized or constrained, these models are more prone to overfitting. Overfitting occurs when a model captures noise in the training data rather than the underlying true patterns, leading to high excess risk. In other words, they may perform poorly on unseen data despite fitting the training data well.

2. Low-Capacity Models: Models with lower capacity, like linear models, have a simpler representation and are less prone to overfitting. They tend to have higher bias and may not fit the training data as closely, resulting in a higher empirical error. However, if the underlying data can be accurately represented by a simpler model, these models may have lower excess risk because they generalize better to unseen data.

The relationship between excess risk and model complexity can be summarized as follows:

• Increasing the complexity or power of the hypothesis class can reduce empirical error (training error) but may increase excess risk (generalization error).

• Decreasing the complexity of the hypothesis class can lead to higher empirical error but may decrease excess risk, resulting in better generalization to new data.

Therefore, finding the right balance between model complexity and generalization is a crucial part of model selection and training in machine learning. Regularization techniques, cross-validation, and other strategies are employed to control and optimize model complexity to achieve the best trade-off between empirical error and excess risk.

============================================

Excess risk itself does not directly determine the best model, but it is a critical concept in the process of model selection and evaluation. Excess risk helps you understand the trade-off between a model's performance on the training data (empirical error) and its ability to generalize to new, unseen data (true error). The goal is to choose a model that minimizes the excess risk, indicating good generalization performance.

Here's how excess risk factors into the process of finding the best model:

1. Model Evaluation: Excess risk is used as a measure of a model's generalization error. When comparing different models or model configurations (e.g., different hyperparameters, architectures, regularization techniques), you can compute and compare their estimated excess risks on a validation dataset or through techniques like cross-validation. The model with the lower estimated excess risk is often preferred because it is expected to perform better on unseen data.

2. Model Selection: The choice of the best model often involves selecting the one with the lowest estimated excess risk. This process may involve comparing models with varying complexities, regularization settings, or other hyperparameters. By analyzing the trade-off between training error and validation error (which estimates excess risk), you can make an informed decision about which model is likely to generalize best to new data.

3. Regularization and Hyperparameter Tuning: Excess risk analysis can guide the selection of appropriate regularization techniques and hyperparameters. For example, you might choose regularization parameters that balance the model's fit to the training data and its ability to generalize. Regularization techniques like L1 and L2 regularization are designed to mitigate overfitting and reduce excess risk.

4. Ensemble Learning: Ensemble methods, such as bagging and boosting, are often used to reduce excess risk. These techniques combine multiple models to improve generalization performance by reducing the risk of overfitting inherent in individual models.

In summary, excess risk is a critical concept in the process of model selection and evaluation, as it helps you identify the model that is expected to perform best on unseen data. However, it's important to note that other factors, such as the quality and size of the data, the choice of features, and the problem domain, also play significant roles in determining the best model. Therefore, while minimizing excess risk is a key goal, it's just one aspect of the broader model selection and evaluation process in machine learning.

============================================

Table 3981. Application examples of excess risk.