Unit 5: Model Evaluation and Improvement

I. Orientation: Measuring and Improving Predictive Models

Model evaluation is the process of measuring how well a machine-learning model performs on data it has not seen during training. The central principle is generalisation: a useful model should learn meaningful patterns rather than memorise the training set. Evaluation methods therefore use suitable metrics, separate data, and controlled comparisons to estimate real-world performance.

• Task alignment: Regression predicts numerical values; classification predicts categories or class probabilities.
• Unseen-data evaluation: Test data must remain separate from training and model-selection data.
• Metric dependence: No single metric describes every type of error or every application.
• Baseline comparison: A model should be compared with a simple baseline, such as predicting the mean in regression or the majority class in classification.
• Overfitting control: A model that performs well on training data but poorly on validation data has learned noise or excessive detail.
• Reproducibility: Data splits, random seeds, preprocessing, and evaluation metrics should be recorded consistently.

II. Regression Metrics: Measuring Numerical Prediction Error

Regression metrics compare predicted numerical values with their actual target values. Let (y_i) be the actual value, (\hat{y}_i) the predicted value, and (n) the number of observations.

A. Regression Metrics (MAE, MSE, RMSE, R² Score)

These metrics quantify prediction accuracy from different mathematical perspectives, so the appropriate choice depends on the consequences of errors.

• Error terms: For observation (i), the residual is (e_i = y_i-\hat{y}_i); positive and negative residuals show underprediction and overprediction.
• Interpretation: Lower MAE, MSE, and RMSE indicate smaller errors, while a higher R² Score generally indicates better explanatory performance.
• Scale: MAE, RMSE, and the target variable use the same units; MSE uses squared units; R² is dimensionless.
• Sensitivity: Squaring errors makes MSE and RMSE more affected by unusually large mistakes than MAE.

B. MAE

Mean Absolute Error measures the average absolute distance between predictions and actual values.

MAE = (1/n) * Σ |y_i - ŷ_i|

• Absolute error: (|y_i-\hat{y}_i|) removes the sign, so underprediction and overprediction contribute equally.
• Practical meaning: An MAE of 4.2 minutes means predictions are off by 4.2 minutes on average.
• Robustness: MAE is less influenced by outliers than MSE because errors are not squared.
• Limitation: MAE does not distinguish between a moderate error and a very large error as strongly as MSE or RMSE.

C. MSE

Mean Squared Error averages the squared residuals and strongly penalises large errors.

MSE = (1/n) * Σ (y_i - ŷ_i)²

• Large-error penalty: A residual of 10 contributes (10^2=100), whereas a residual of 2 contributes only (2^2=4).
• Optimisation value: MSE is differentiable and commonly used as a training loss for linear regression and neural networks.
• Units: If the target is measured in dollars, MSE is measured in dollars squared, which can be difficult to interpret directly.
• Outlier effect: A few extreme observations can dominate the score and make the model appear worse than its typical performance.

D. RMSE

Root Mean Squared Error is the square root of MSE, returning the error to the original target scale.

RMSE = √[(1/n) * Σ (y_i - ŷ_i)²]

• Interpretability: An RMSE of 5 kilograms means the typical error magnitude is approximately 5 kilograms, with extra emphasis on large errors.
• Comparison: RMSE is always at least as large as MAE for the same dataset because squaring gives greater weight to larger residuals.
• Use case: RMSE is suitable when large errors, such as severe demand underestimates, are especially costly.
• Limitation: Like MSE, RMSE is sensitive to outliers.

E. R² Score

The coefficient of determination measures how much of the target variation is explained by the model compared with predicting the mean.

R² = 1 - [Σ(y_i - ŷ_i)² / Σ(y_i - ȳ)²]

• Symbols: (\bar{y}) is the mean of the actual target values; the numerator is residual error; the denominator is total variation around the mean.
• Reference value: (R^2=0) means the model performs like the mean-prediction baseline.
• Interpretation: (R^2=0.80) indicates that the model reduces squared error by 80% relative to that baseline.
• Possible values: R² can be negative on test data when predictions are worse than simply using (\bar{y}).
• Limitation: A high R² does not prove accurate predictions, causal relationships, or acceptable performance on every subgroup.

III. Confusion Matrix and Classification Metrics: Evaluating Classes

Classification evaluation begins with a confusion matrix, which counts correct and incorrect predictions by actual and predicted class. For binary classification, the positive class is the class of interest.

A. Confusion Matrix and Classification Metrics (Accuracy, Precision, Recall, F1-score, ROC-AUC)

The confusion matrix provides the concrete counts from which most classification metrics are calculated.

• True Positive (TP): The model predicts positive and the actual class is positive, such as correctly identifying a fraudulent transaction.
• True Negative (TN): The model predicts negative and the actual class is negative.
• False Positive (FP): The model predicts positive when the actual class is negative; this is a Type I error.
• False Negative (FN): The model predicts negative when the actual class is positive; this is a Type II error.
• Threshold dependence: Accuracy, precision, recall, and F1-score usually depend on the probability threshold used to convert scores into class labels.

B. Accuracy

Accuracy is the proportion of all predictions that are correct.

Accuracy = (TP + TN) / (TP + TN + FP + FN)

• Strength: Accuracy is simple and useful when classes are reasonably balanced and error costs are similar.
• Example: With 90 correct predictions out of 100, accuracy is (90/100=0.90), or 90%.
• Imbalance problem: If 98% of transactions are legitimate, predicting “legitimate” every time gives 98% accuracy but detects no fraud.
• Evaluation requirement: Accuracy should be reported with class distribution and complementary metrics.

C. Precision

Precision measures how many predicted positives are actually positive.

Precision = TP / (TP + FP)

• Positive prediction quality: High precision means few false alarms among positive predictions.
• Application: Precision matters in email filtering when incorrectly marking a legitimate email as spam is costly.
• Concrete interpretation: If (TP=45) and (FP=5), precision is (45/(45+5)=0.90).
• Trade-off: Increasing the decision threshold often raises precision but may reduce recall by creating more false negatives.

D. Recall

Recall, also called sensitivity or true-positive rate, measures how many actual positives are detected.

Recall = TP / (TP + FN)

• Detection focus: High recall means few actual positive cases are missed.
• Application: Medical screening often prioritises recall because failing to identify a disease case can be dangerous.
• Concrete interpretation: If (TP=45) and (FN=15), recall is (45/(45+15)=0.75).
• Trade-off: Lowering the classification threshold can increase recall but usually produces more false positives.

E. F1-score

F1-score is the harmonic mean of precision and recall, rewarding models that perform well on both.

F1 = 2 * (Precision * Recall) / (Precision + Recall)

• Balance: F1 becomes low when either precision or recall is low; it is not an ordinary arithmetic average.
• Example: With precision (0.90) and recall (0.75), (F1 \approx 0.818).
• Use case: F1 is useful for imbalanced classification when false positives and false negatives both matter.
• Limitation: F1 ignores true negatives and does not express the different financial or safety costs of each error type.

F. ROC-AUC

ROC-AUC measures how well a classifier ranks positive examples above negative examples across classification thresholds.

TPR = TP / (TP + FN)
FPR = FP / (FP + TN)

• ROC curve: The curve plots true-positive rate (TPR) against false-positive rate (FPR) as the threshold changes.
• AUC interpretation: An AUC of 0.5 represents random ranking; an AUC of 1.0 represents perfect separation.
• Threshold independence: ROC-AUC evaluates ranking across thresholds rather than one chosen operating threshold.
• Imbalance caution: With very rare positive classes, precision-recall curves may provide a more informative view than ROC-AUC alone.

IV. Model Improvement: Validation and Search Strategies

Model improvement involves changing the training procedure or model configuration and verifying that the change improves generalisation. The comparison must be made on validation data, with a final test set kept untouched.

A. Model Improvement

The purpose of model improvement is to reduce generalisation error while maintaining an appropriate balance between underfitting and overfitting.

• Underfitting: A model with excessive bias is too simple and performs poorly on both training and validation data.
• Overfitting: A model with excessive variance performs very well on training data but poorly on unseen data.
• Improvement actions: Useful actions include better features, regularisation, more representative data, suitable algorithms, and tuned hyperparameters.
• Data leakage: Information from validation or test data must not influence training, feature construction, or parameter selection.

B. Cross Validation

Cross Validation estimates performance by repeatedly training and validating a model on different partitions of the available training data.

• K-fold procedure: In (k)-fold cross-validation, data are divided into (k) folds; each fold is used once for validation while the remaining (k-1) folds train the model.
• Performance estimate: The reported score is commonly the mean of the (k) validation scores, often accompanied by their standard deviation.
• Stratification: Stratified k-fold preserves class proportions in classification folds, reducing the risk of a fold containing too few positive examples.
• Time ordering: Time-series data generally require time-aware splits because randomly mixing future observations into training creates leakage.

C. Hyperparameter Tuning

Hyperparameter tuning selects configuration values fixed before training, rather than learned directly from the training examples.

• Examples: A decision tree may tune maximum depth; k-nearest neighbours may tune (k); a neural network may tune learning rate and batch size.
• Parameter distinction: Model parameters, such as regression coefficients, are learned during fitting; hyperparameters control that fitting process.
• Validation basis: Each candidate configuration should be evaluated through validation or cross-validation.
• Overfitting risk: Repeatedly selecting configurations using the same validation data can overfit the validation process, so a final untouched test set is necessary.

D. Grid Search

Grid Search evaluates every specified combination in a predefined hyperparameter grid.

For each combination in the grid:
    perform cross-validation
Select the combination with the best mean validation score
Refit the selected model on all training data

• Exhaustiveness: A grid containing 3 depth values and 4 minimum-sample values requires (3\times4=12) combinations.
• Strength: Grid Search is systematic and easy to reproduce when the search space is small.
• Limitation: Computation grows multiplicatively with each added hyperparameter and may waste trials in unimportant regions.
• Scoring: The scoring function must match the objective, such as negative MAE for regression or F1 for an imbalanced classifier.

E. Random Search

Random Search samples a fixed number of hyperparameter combinations from specified distributions or lists.

• Efficiency: With a budget of 30 trials, Random Search explores 30 combinations even when the grid contains hundreds or thousands.
• Advantage: It can discover useful values for influential hyperparameters without exhaustively testing every value of less important ones.
• Reproducibility: A fixed random seed makes sampled configurations repeatable.
• Limitation: Random Search can miss narrow high-performing regions, especially when the trial budget is too small.

V. Introduction to Explainable AI: Understanding Model Decisions

Explainable AI (XAI) comprises methods that make model behaviour understandable to people. Explanations can describe the overall model or clarify one particular prediction, supporting debugging, trust, accountability, and responsible deployment.

A. Introduction to Explainable AI

XAI connects a model’s inputs and outputs with human-interpretable reasons, while recognising that an explanation is not automatically proof of causation.

• Global explanation: Describes overall behaviour, such as which features generally influence predictions across a dataset.
• Local explanation: Explains one prediction, such as why a loan application received a particular risk score.
• Feature importance: Permutation importance measures performance change after a feature’s values are shuffled; a large drop suggests predictive usefulness.
• SHAP values: SHAP assigns each feature a contribution relative to a baseline prediction; positive and negative values indicate movement toward or away from the predicted outcome.
• LIME: LIME fits an interpretable local model around one prediction, approximating the behaviour of a more complex model nearby.
• Limitations: Explanations can be unstable, affected by correlated features, or misleading when the underlying data contain bias; interpretability does not establish a causal relationship.
• Practical standard: A useful explanation should be faithful to the model, understandable to its audience, and checked against domain knowledge and model performance.