Unit 6: MODEL PERFORMANCE - Subjective Questions

Bias is the error caused by overly simplistic assumptions in a model. A high-bias model usually underfits the data and performs poorly on both training and unseen data. Variance is the error caused by excessive sensitivity to the training data. A high-variance model usually overfits the training data and performs poorly on unseen data. The expected prediction error can be represented as: The trade-off occurs because reducing bias often increases variance, while reducing variance may increase bias. A good predictive model balances both errors to achieve strong generalization performance.

Underfitting occurs when a model is too simple to capture important patterns in the data. It has high bias, low variance, high training error, and high test error. Examples include using a linear model for a strongly nonlinear relationship. Overfitting occurs when a model learns noise and random fluctuations in the training data. It has low training error, high variance, and high test error. A highly complex decision tree is a common example. The ideal model lies between these extremes, where both training and validation performance are satisfactory. Cross-validation, regularization, pruning, and selecting an appropriate model complexity can help achieve this balance.

Let the observed target be , where is the true function and is random noise with and . Suppose a learning algorithm produces the estimate . The expected squared error is: Substituting and expanding gives: Add and subtract inside the first term. This produces: Therefore: The bias term measures systematic error, the variance term measures model instability, and the noise term cannot be eliminated by changing the model.

As model complexity increases, the model gains greater flexibility to represent relationships in the training data. Training error generally decreases because the model can fit the observations more closely. Bias also tends to decrease because the model makes fewer simplifying assumptions. However, variance generally increases because the model becomes more sensitive to the particular training sample. Test error usually follows a U-shaped curve: it first decreases as underfitting is reduced, reaches a minimum at an appropriate complexity, and then increases because of overfitting. The best complexity is selected using validation data or cross-validation rather than training error alone.

Ensemble learning is a technique that combines predictions from multiple individual models, called base learners, to produce a stronger final model. The base learners may be trained on different samples, features, or sequentially modified versions of the data. Combining them can improve accuracy, stability, and generalization. For regression, predictions are often averaged: For classification, the final class may be selected by majority voting. Ensembles work well when the component models make errors that are not perfectly correlated. Their main advantages are reduced variance, improved predictive accuracy, and greater robustness.

Bagging, or bootstrap aggregating, trains multiple versions of the same learning algorithm on different bootstrap samples of the training data. Each bootstrap sample is created by sampling observations with replacement. The predictions of the individual models are then aggregated using averaging for regression or majority voting for classification. If the individual model predictions have variance and are independent, averaging models reduces the variance approximately to: In practice, the models are correlated, so the reduction is smaller. Bagging is especially useful for high-variance models such as decision trees because it stabilizes their predictions without substantially increasing bias.

For a training set containing observations, bagging creates a bootstrap sample by drawing observations with replacement. Some observations appear multiple times, while others are omitted. The probability that a particular observation is not selected is approximately: Thus, about 36.8% of the observations are typically excluded from each bootstrap sample. These excluded observations are called out-of-bag observations. They can be used to evaluate the model trained without those observations. By aggregating out-of-bag predictions across all trees, an out-of-bag error estimate can be obtained without requiring a separate validation set.

Bagging trains base learners independently and in parallel on bootstrap samples. It primarily reduces variance by averaging diverse models and is relatively resistant to overfitting. Boosting trains learners sequentially, with each new learner focusing on observations that previous learners predicted poorly. It primarily reduces bias, although it can also reduce variance. Bagging commonly uses deep decision trees, while boosting often uses weak learners such as shallow trees. Bagging assigns equal importance to models during aggregation in many implementations, whereas boosting assigns weights according to learner performance. Bagging is generally more robust to noise, while boosting may be sensitive to outliers and noisy labels.

Boosting is an ensemble method that combines many weak learners into a strong learner. The learners are trained sequentially rather than independently. At each stage, the algorithm identifies errors made by the current ensemble and gives greater attention to difficult observations or residuals. The next learner attempts to correct these errors. For an additive model, the prediction can be written as: where is the new weak learner and is its contribution or learning rate. Repeating this process produces a model capable of representing complex relationships. Boosting can substantially improve accuracy, but excessive iterations, large learner complexity, or noisy data may lead to overfitting.

AdaBoost adjusts the weights of training observations. Misclassified observations receive higher weights so that subsequent weak learners focus on them. The final prediction is a weighted combination of the learners. Gradient boosting fits each new learner to the negative gradient of the loss function, which is often equivalent to fitting residuals in regression. It constructs an additive model through functional gradient descent. AdaBoost is closely associated with classification and exponential loss, whereas gradient boosting can support many loss functions for both regression and classification. Gradient boosting provides greater flexibility but requires careful tuning of the learning rate, number of learners, and learner complexity.

The learning rate controls the contribution of each new learner. A small learning rate usually requires more estimators but can improve generalization. The number of estimators determines how many learners are added to the ensemble. Too few learners may cause underfitting, while too many may cause overfitting, particularly with complex base learners. Weak learner complexity, such as the maximum depth of a tree, controls the interaction complexity captured at each stage. Shallow learners have higher bias and are less likely to overfit, whereas deep learners can fit complex patterns but increase variance. These parameters must be tuned together using validation or cross-validation.

A random forest classifier is built by creating many decision trees. For each tree: (1) draw a bootstrap sample from the training data, (2) grow a decision tree using that sample, and (3) at every split, randomly select a subset of the available features and choose the best split only from that subset. The final classification is obtained through majority voting: Bootstrap sampling introduces variation among the trees, while random feature selection reduces correlation between them. The combination of many diverse trees generally produces a more accurate and stable classifier than a single decision tree.

If every tree considered all features at each split, a few dominant predictors could cause many trees to develop similar structures. Highly correlated trees provide less benefit when their predictions are aggregated. Random forests therefore select only a random subset of features at each split. This increases diversity among the trees and lowers the correlation of their errors. The ensemble variance is reduced when predictions are averaged across less-correlated trees. The method also gives less dominant features opportunities to contribute to useful splits. The number of features considered at each split is a hyperparameter that must be selected according to the data and task.

Suppose an ensemble contains base learners, each with variance , and the pairwise correlation between any two learners is . For the average prediction, the variance is: If the learners are independent, then and the variance becomes . As becomes very large, the variance approaches , so simply adding more correlated learners cannot eliminate all variance. Random forests reduce variance in two ways: bootstrap samples create different training sets, and random feature selection decreases correlation between trees. This explains why both tree diversity and ensemble size matter.

A single decision tree is easy to interpret but may have high variance because small changes in the training data can produce a different tree. Bagged trees train trees on bootstrap samples and aggregate their predictions, thereby reducing variance. However, if all features are considered at every split, the trees may remain highly correlated. A random forest adds random feature selection at each split, producing more diverse and less-correlated trees. Consequently, random forests usually generalize better than a single tree and often better than ordinary bagged trees. The trade-off is that random forests are less interpretable than a single tree and may require more computational resources.

Important random forest hyperparameters include: number of trees, which generally improves stability but increases computation; maximum feature count, which controls tree correlation and split diversity; maximum tree depth, which controls individual tree complexity; minimum samples for splitting or leaf nodes, which can prevent overly specific leaves; bootstrap sampling, which determines whether samples are drawn with replacement; and class weights, which can address class imbalance. Increasing tree depth usually lowers bias but can increase the complexity of individual trees. Reducing the number of features per split can lower correlation but may increase bias. These parameters should be tuned using validation or cross-validation.

Random forests commonly estimate feature importance using impurity-based importance and permutation importance. Impurity-based importance sums the reduction in measures such as Gini impurity or squared error caused by splits involving a feature, averaged over all trees. Permutation importance measures the increase in validation or out-of-bag error after randomly shuffling one feature. A large increase indicates that the model depends on that feature. Impurity-based importance can be biased toward high-cardinality or continuous variables. Permutation importance is often more reliable for model interpretation but can be misleading when predictors are strongly correlated, because one correlated feature may substitute for another.

Each tree in a random forest is trained on a bootstrap sample, leaving some observations out. For a particular observation, predictions can be generated only from trees for which that observation was out-of-bag. The aggregated out-of-bag prediction is compared with the true target to calculate an out-of-bag error estimate. For classification, this may be the proportion of incorrect predictions; for regression, it may be mean squared error. Out-of-bag evaluation provides an internal estimate of generalization performance without requiring a separate validation set. It is useful for monitoring whether adding trees improves performance, although a separate test set is still recommended for final unbiased evaluation.

Bagging and random forests primarily reduce variance. They are especially useful when individual models are unstable, such as deep decision trees, and when training error is low but test error is high. Boosting primarily reduces bias by adding learners that correct the errors of previous learners. It is useful when the base learner is too simple and systematically misses important patterns. An ensemble cannot remove irreducible noise, and reducing variance may not solve a high-bias problem. The appropriate method depends on validation results, the complexity of the base learner, noise level, and the relationship between component-model errors.

Advantages: Random forests can model nonlinear relationships and interactions, require relatively little preprocessing, work with large feature sets, provide strong predictive accuracy, and offer out-of-bag evaluation and feature-importance estimates. They are also less prone to overfitting than individual decision trees. Limitations: They are less interpretable than a single tree, can require substantial memory and computation, may produce less accurate extrapolations in regression, and may be affected by severe class imbalance unless class weights or resampling are used. Feature importance can also be misleading when predictors are correlated. Therefore, random forests should be evaluated with suitable metrics and interpreted carefully.