Unit 6: MODEL PERFORMANCE

I. Foundations of Model Performance

Model performance describes how effectively a predictive model generalizes from observed training data to previously unseen data. Its governing principle is generalization: a useful model must learn stable patterns in the data without memorizing random noise. Performance is therefore estimated on validation or test data using measures appropriate to the prediction task.

• Generalization error: The expected loss on new observations drawn from the same population as the training data.
• Training error: The average loss calculated on observations used to fit the model; it usually decreases as model complexity increases.
• Test error: The average loss on an independent test set; it estimates generalization error when the test set has not influenced model selection.
• Loss function: A numerical penalty for prediction errors.
  • For regression, squared-error loss is (L(y,\hat{y})=(y-\hat{y})^2).
  • For classification, zero-one loss is (L(y,\hat{y})=\mathbf{1}(y\ne\hat{y})), where (\mathbf{1}) equals 1 when the condition is true.
• Model complexity: The flexibility available to fit patterns, such as tree depth, polynomial degree, or the number of model parameters.
• Underfitting: Failure to capture the underlying relationship, producing poor performance on both training and test data.
• Overfitting: Excessive adaptation to training-specific noise, producing low training error but relatively high test error.
• Data-splitting convention: Training data fit model parameters, validation data select models or hyperparameters, and test data provide a final unbiased evaluation.
• Resampling methods: Techniques such as (k)-fold cross-validation repeatedly change the validation observations, producing a more stable estimate of expected performance.
• Evaluation context: Accuracy alone may be misleading for imbalanced classification; precision, recall, F1-score, ROC-AUC, mean absolute error, or root mean squared error may be more suitable.

II. Bias and Variance — Sources of Prediction Error

Bias and variance explain why a model may perform poorly on unseen data. Bias measures systematic error caused by restrictive assumptions, whereas variance measures sensitivity to the particular sample used for training.

A. Bias-variance trade-off

The bias-variance trade-off states that increasing model flexibility generally reduces bias but increases variance, so model complexity must be selected to minimize expected prediction error.

• Regression setting: Assume that the response follows
  

  TEXT

    Y = f(X) + ε
    E(ε) = 0,    Var(ε) = σ²

  
  Here, (X) is the predictor, (Y) is the response, (f(X)) is the true relationship, (\varepsilon) is random noise, and (\sigma^2) is the irreducible noise variance.

• Error decomposition: For a fitted predictor (\hat{f}(x)), expected squared prediction error at an input (x) is
  

  TEXT

    E[(Y - f̂(x))²]
      = Bias[f̂(x)]² + Var[f̂(x)] + σ²

  
  The expectation is taken over possible training samples and future response noise.

• Bias: The difference between the model’s average prediction and the true function:
  

  TEXT

    Bias[f̂(x)] = E[f̂(x)] - f(x)

  
  High bias commonly results from an overly simple model, such as fitting a straight line to a strongly curved relationship.

• Variance: The amount by which fitted predictions vary across different training samples:
  

  TEXT

    Var[f̂(x)] = E[(f̂(x) - E[f̂(x)])²]

  
  High variance is common in deep decision trees because small changes in training observations can produce different splits.

• Irreducible error: The term (\sigma^2) represents randomness that cannot be removed by improving the model, such as measurement error or unobserved influences.

1. High-bias models: Simple models tend to make stable but systematically inaccurate predictions.

   • A shallow tree may omit important interactions.
   • Training and validation errors are both likely to be high.
   • Increasing appropriate complexity can improve performance.

2. High-variance models: Highly flexible models can represent complex relationships but may follow accidental sample patterns.

   • An unrestricted tree may create leaves containing very few observations.
   • Training error may approach zero while validation error rises.
   • Regularization, pruning, more data, or ensembling can reduce variance.

• Trade-off curve: As complexity increases, training error usually falls continuously, while test error often follows a U-shaped pattern. Its minimum represents a useful balance between approximation error and estimation instability.

• Worked example: Suppose three models have estimated error components:

  Model	Bias²	Variance	Noise	Expected error
  Simple	16	2	4	22
  Moderate	6	5	4	15
  Complex	1	18	4	23

  The moderate model performs best because its total expected error, (6+5+4=15), is lowest, even though it has neither the smallest bias nor the smallest variance.

• Practical selection: Cross-validation estimates validation error across candidate complexity levels. Hyperparameters such as tree depth, minimum leaf size, or regularization strength are selected from training and validation results, not from the final test set.

• Classification qualification: The exact additive decomposition above applies directly to squared-error regression. Bias and variance remain useful concepts in classification, but their mathematical decomposition depends on the loss function used.

III. Ensemble Learning — Combining Predictive Models

Ensemble learning combines predictions from multiple base learners to obtain a model that is more accurate or stable than an individual learner. Diversity is essential: combining nearly identical models provides little benefit because they tend to make the same errors.

A. Ensemble Learning: Basic Concept of Bagging and Boosting

Bagging and boosting both construct multiple models, but bagging fits learners independently to reduce variance, whereas boosting fits learners sequentially to correct previous errors.

1. Bagging: Bootstrap aggregating trains base learners on different bootstrap samples and then averages or votes across their predictions.

   • Bootstrap sampling: For a training set of (n) observations, each bootstrap set contains (n) draws made with replacement. Some observations appear multiple times, while about (36.8\%) are omitted from a particular sample.
   • Parallel construction: Each learner is fitted independently, so training can be distributed across processors.
   • Regression aggregation:
     
     TEXT

          f̂bag(x) = (1/B) Σ[b=1 to B] f̂b(x)

     
     Here, (B) is the number of learners and (\hat{f}_b(x)) is learner (b)'s prediction.
   • Classification aggregation: The final class is usually selected by majority vote, or class probabilities are averaged before applying a decision threshold.
   • Variance reduction: Averaging stabilizes unstable learners such as deep trees. If learners have variance (v) and pairwise correlation (\rho), the average has approximate variance
     
     TEXT

          ρv + (1 - ρ)v/B

     
     Increasing (B) reduces the second term, but strongly correlated learners leave the first term largely unchanged.
   • Out-of-bag estimation: An observation can be predicted using only learners whose bootstrap samples excluded it. Aggregated out-of-bag predictions provide an internal performance estimate without a separate validation set.
   • Main limitation: Bagging primarily addresses variance; it does not automatically correct strong bias shared by all base learners.

2. Boosting: Boosting builds learners sequentially, with each new learner concentrating on errors made by the current ensemble.

   • Sequential construction: Learner (m) depends on the fitted ensemble from iterations (1) through (m-1), so ordinary boosting is less naturally parallel than bagging.
   • AdaBoost principle: Initially, observations receive equal weights. After each weak classifier is fitted, misclassified observations receive greater relative weight, directing the next learner toward difficult cases.
   • Weighted classification:
     
     TEXT

          H(x) = sign(Σ[m=1 to M] αm hm(x))

     
     Here, (M) is the number of learners, (h_m(x)) is weak learner (m)'s class prediction, and (\alpha_m) gives more influence to better-performing learners.
   • Gradient boosting principle: Each new learner approximates the negative gradient of the chosen loss function. Under squared-error loss, this is equivalent to fitting residuals.
   • Additive update:
     
     TEXT

          Fm(x) = Fm-1(x) + η hm(x)

     
     Here, (F_m) is the updated ensemble, (h_m) is the new learner, and (\eta) is the learning rate.
   • Bias reduction: Repeated correction can transform weak learners, such as shallow trees, into a flexible predictor capable of representing nonlinear effects and interactions.
   • Regularization: Small learning rates, shallow trees, subsampling, and early stopping control overfitting. A smaller (\eta) generally requires more boosting iterations.
   • Main limitation: Boosting is sensitive to hyperparameters and may concentrate excessively on noisy observations or mislabeled cases.

B. Applications and Limitations

The choice between bagging and boosting depends on whether instability, systematic underfitting, computational constraints, or noisy observations dominate the problem.

• Bagging applications: Appropriate when the base learner has low bias but high variance, as with deep decision trees used for tabular classification or regression.
• Boosting applications: Effective when many weak rules must be combined to model complex patterns, particularly in structured business, financial, and operational data.
• Interpretability cost: An ensemble of hundreds of learners is harder to explain than one tree; feature importance and local explanation methods provide approximations rather than a single transparent rule set.
• Validation requirement: The number of learners and other hyperparameters should be selected using cross-validation or out-of-bag evidence where available.
• Shared constraint: Ensembles cannot repair poor target definitions, leakage, unrepresentative samples, or features unavailable at prediction time.

IV. Random Forests — Decorrelated Tree Ensembles

A random forest is a bagging-based ensemble of decision trees that adds random feature selection at each split. This extra randomization reduces correlation among trees, making averaging more effective.

A. Random forests

Random forests generate bootstrap samples, grow many usually unpruned trees, and aggregate their predictions while restricting each split to a random subset of predictors.

• Training mechanism:
  

  TEXT

    For b = 1, ..., B:
        Draw a bootstrap sample from the training set
        Grow a decision tree:
            At each node, randomly select mtry predictors
            Choose the best split only among those predictors
    Aggregate predictions from all B trees

  
  Here, (B) is the number of trees and mtry is the number of candidate predictors considered at each split.

• Feature randomization: A powerful predictor cannot dominate every split because it is sometimes excluded from the candidate set. Trees consequently explore different structures and become less correlated.

• Prediction rule: Regression forests average tree outputs, while classification forests use majority voting or averaged class probabilities.

• Bias-variance effect: Deep trees individually have low bias and high variance. Bootstrap averaging reduces variance, and random feature selection strengthens that reduction by lowering inter-tree correlation.

• Hyperparameters:

  • Number of trees: More trees stabilize predictions but increase computation and memory; adding trees generally does not cause classical overfitting through ensemble size alone.
  • mtry: Smaller values increase diversity but may weaken individual trees; larger values strengthen trees but increase correlation.
  • Tree size: Maximum depth, minimum leaf size, and minimum split size control individual-tree complexity.
  • Sampling controls: Bootstrap size, class weights, or balanced sampling can address computational needs and class imbalance.

• Out-of-bag performance: Each observation is omitted from roughly (36.8\%) of bootstrap samples. Trees that omitted the observation predict it, and those predictions are combined to calculate out-of-bag error.

• Feature importance: Permutation importance measures the increase in prediction error after a feature’s values are shuffled. Impurity-based importance sums split improvements but can favor continuous or high-cardinality predictors.

B. Applications and Limitations

Random forests provide a strong general-purpose baseline for tabular data because they capture nonlinear relationships and interactions with limited preprocessing.

• Applications: They support credit-risk prediction, customer churn classification, demand estimation, fraud detection, medical prediction, and other tasks involving mixed tabular variables.
• Preprocessing advantage: Trees do not require feature scaling, and monotonic transformations usually do not change split ordering.
• Robustness: Averaging makes forests less sensitive than a single tree to sampling fluctuations and isolated observations.
• Limitations: Forests are computationally larger, less interpretable, and poor at extrapolating regression outcomes beyond values represented in training leaves.
• Data considerations: Missing values and categorical variables require handling according to the software implementation; high-cardinality categories can still create misleading patterns.
• Evaluation discipline: Out-of-bag error is useful for development, but a separate test set remains necessary for final evaluation after extensive tuning or repeated model comparison.