Unit 5: Model Evaluation and Improvement - Subjective Questions

Mean Absolute Error (MAE) measures the average absolute difference between the actual and predicted values.

The formula is:

where is the actual value, is the predicted value, and is the number of observations.

• A lower MAE indicates better model performance.
• MAE is expressed in the same units as the target variable.
• Advantage: It is easy to interpret and is less sensitive to outliers than MSE.
• Limitation: It treats all errors linearly and does not penalize large errors as strongly as MSE.

Mean Squared Error (MSE) is the average of the squared differences between actual and predicted values:

Root Mean Squared Error (RMSE) is the square root of MSE:

• MSE is measured in squared units, while RMSE is measured in the same units as the target variable.
• RMSE is generally easier to interpret than MSE.
• Both metrics strongly penalize large errors because the errors are squared.
• RMSE is useful when large prediction errors are especially undesirable.
• A lower MSE or RMSE indicates better regression performance.

The relationship is defined by taking the square root of the Mean Squared Error:

For the given values:

• Actual values:
• Predicted values:
• Errors:
• Squared errors:

Therefore:

Thus, the RMSE is approximately units. It represents the typical size of the prediction error in the same units as the target variable.

The score, or coefficient of determination, measures the proportion of variance in the dependent variable explained by the regression model.

Its formula is:

where is the mean of the actual target values.

• indicates perfect predictions.
• means the model performs no better than predicting the mean of the target values.
• means the model performs worse than the mean-based baseline.
• A higher generally indicates a better fit, but it does not prove that the model generalizes well.
• should be considered together with metrics such as MAE or RMSE.

The regression metrics differ in their error treatment and interpretation:

• MAE: Averages absolute errors. It is robust to outliers and easy to interpret in the original target units.
• MSE: Averages squared errors. It heavily penalizes large errors and is useful when large mistakes are costly.
• RMSE: The square root of MSE. It also penalizes large errors but is expressed in the original target units.
• score: Measures the proportion of target variance explained by the model. It is scale-independent but does not directly show the average prediction error.

When outliers are important and should receive strong penalties, MSE or RMSE is appropriate. When interpretability is the priority, MAE is usually preferred because its value directly represents the average absolute prediction error.

A confusion matrix summarizes the results of a classification model by comparing predicted classes with actual classes.

• True Positive (TP): The model correctly predicts the positive class.
• True Negative (TN): The model correctly predicts the negative class.
• False Positive (FP): The model predicts positive when the actual class is negative. This is also called a Type I error.
• False Negative (FN): The model predicts negative when the actual class is positive. This is also called a Type II error.

The four values form the basis for calculating accuracy, precision, recall, F1-score, and other classification metrics.

The main classification metrics are defined as follows:

Accuracy measures the proportion of all predictions that are correct:

Precision measures the proportion of predicted positive cases that are actually positive:

Precision focuses on reducing false positives.

Recall, also called sensitivity or true positive rate, measures the proportion of actual positive cases correctly identified:

Recall focuses on reducing false negatives.

Accuracy is suitable when classes are reasonably balanced and errors have similar costs. Precision is important when false alarms are costly, while recall is important when missing a positive case is dangerous.

The F1-score is the harmonic mean of precision and recall:

Using the confusion matrix values, it can also be written as:

The F1-score is useful because:

• It combines precision and recall into a single metric.
• The harmonic mean becomes low if either precision or recall is low.
• It is more informative than accuracy when the classes are imbalanced.
• It is useful when both false positives and false negatives matter.

For example, a classifier that predicts the majority class for every observation may have high accuracy but poor recall for the minority class. Its F1-score will reflect this weakness.

An ROC curve plots the True Positive Rate against the False Positive Rate at different classification thresholds.

The two quantities are:

The ROC-AUC is the area under the ROC curve. It represents the model's ability to distinguish between positive and negative classes across all thresholds.

• An AUC of indicates perfect separation.
• An AUC of indicates performance similar to random guessing.
• An AUC below indicates performance worse than random, although reversing predictions may improve it.

Lowering the classification threshold usually increases both true positives and false positives. Raising it usually decreases both. ROC-AUC summarizes this trade-off across thresholds rather than evaluating only one threshold.

The total number of observations is:

Accuracy:

Therefore, accuracy is .

Precision:

Therefore, precision is approximately .

Recall:

Therefore, recall is .

F1-score:

Therefore, the F1-score is approximately . The model has good overall performance and precision, but it misses of the actual positive cases.

Accuracy can be misleading when one class is much more common than the other. For example, if of transactions are legitimate, a model that labels every transaction as legitimate achieves accuracy but detects no fraud.

In such cases, the following metrics should be considered:

• Precision: Useful when false positive predictions are costly.
• Recall: Useful when failing to detect positive cases is costly.
• F1-score: Balances precision and recall.
• ROC-AUC: Measures ranking and class separation across thresholds.
• Precision-Recall AUC: Often more informative than ROC-AUC when the positive class is rare.
• Specificity: Measures the proportion of actual negatives correctly identified.

The best metric depends on the practical costs of false positives and false negatives.

Cross-validation is a resampling technique used to evaluate how well a model is likely to perform on unseen data.

In -fold cross-validation:

1. The dataset is divided into approximately equal-sized folds.
2. The model is trained on folds.
3. The remaining fold is used for validation.
4. This process is repeated times so that every fold serves as the validation set once.
5. The validation scores are averaged to obtain the final performance estimate.

For example, in 5-fold cross-validation, the model is trained and validated five times.

Cross-validation provides a more reliable estimate than a single train-validation split because it uses multiple validation subsets. It also helps identify models that perform inconsistently across different portions of the data.

In ordinary -fold cross-validation, the data is divided into folds without necessarily preserving the class proportions in each fold.

Stratified -fold cross-validation creates folds that maintain approximately the same class distribution as the complete dataset.

Stratification is important because:

• It ensures that every fold contains representative examples of each class.
• It reduces the chance that a minority class is absent or severely underrepresented in a validation fold.
• It produces more stable and reliable estimates of classification metrics.
• It is especially valuable for imbalanced datasets.

For regression problems, variants such as shuffled -fold cross-validation are commonly used because the target is continuous rather than categorical.

Hyperparameter tuning is the process of selecting values for settings that control a machine learning algorithm before or during training.

The distinction is:

• Model parameters are learned from the training data. Examples include the weights in linear regression or the split thresholds in a decision tree.
• Hyperparameters are selected by the practitioner or an optimization procedure. Examples include the learning rate, number of trees, maximum tree depth, regularization strength, and number of nearest neighbors.

Hyperparameters influence the model's complexity and learning behavior. Poor choices can cause:

• Underfitting, when the model is too simple.
• Overfitting, when the model memorizes the training data.

Cross-validation is commonly used to compare hyperparameter settings and select a configuration that generalizes well.

Grid search evaluates every combination of hyperparameter values specified by the user.

Procedure:

1. Select the hyperparameters to tune.
2. Define a finite set of candidate values for each hyperparameter.
3. Form all possible combinations of those values.
4. Train and evaluate a model for each combination, usually using cross-validation.
5. Select the combination with the best validation score.
6. Retrain the model using the selected settings and the available training data.

Advantages:

• It is systematic and easy to understand.
• It can find the best combination within the specified grid.
• It is reproducible when the data split and settings are fixed.

Limitations:

• Computational cost increases rapidly as more hyperparameters and values are added.
• It may waste resources evaluating unimportant combinations.
• The best result is limited to the values included in the grid.

Grid search evaluates all combinations from predefined candidate lists, whereas random search samples a specified number of combinations from distributions or ranges of possible values.

Random search can be more efficient when:

• Only a few hyperparameters strongly affect performance.
• The search space contains many hyperparameters.
• Continuous ranges are more appropriate than a small fixed grid.
• Computational resources are limited.

The number of trials and the random seed should be recorded to make the process reproducible.

A suitable model improvement strategy would include the following steps:

1. Prepare the data: Clean the data, encode categorical features, scale features when required, and divide the data into training and final test sets.
2. Establish a baseline: Train a simple model and record suitable metrics.
3. Create a validation procedure: Use -fold or stratified -fold cross-validation on the training set.
4. Define a search space: Select relevant hyperparameters and reasonable candidate ranges.
5. Run the search: Apply grid search or random search using only the training data and cross-validation.
6. Select the model: Choose the configuration with the best average validation score.
7. Inspect variability: Examine the standard deviation of scores across folds.
8. Evaluate once: Test the final model on the untouched test set.

To avoid overfitting:

• Do not use the test set during tuning.
• Place preprocessing inside a pipeline so it is fitted separately within each fold.
• Avoid excessively complex models.
• Use regularization or early stopping where appropriate.
• Prefer a simpler model when performance differences are negligible.

Explainable Artificial Intelligence (XAI) refers to methods and techniques that make the predictions and behavior of machine learning models understandable to humans.

Explainability is important because it can:

• Build trust among users, developers, and decision-makers.
• Help identify errors, bias, and data leakage.
• Support debugging and model improvement.
• Provide evidence for decisions in regulated domains.
• Help users understand when a prediction should not be trusted.
• Improve accountability and compliance.

For example, in a loan approval system, an explanation might show that income, credit history, and debt level contributed most to a prediction. Explainability does not necessarily mean that the model becomes simple; it means that suitable explanations are provided for its behavior.

Global explanations describe the overall behavior of a model across the dataset.

• They identify which features generally influence predictions.
• They help reveal the model's learned patterns and overall structure.
• Example: A feature importance report showing that income and credit history are the most influential variables in loan predictions.

Local explanations describe why the model produced a particular prediction for one instance.

• They explain the contribution of features for a specific observation.
• They help users understand an individual decision.
• Example: Explaining that a particular loan application was rejected mainly because of high debt and a short credit history.

Global explanations provide an overall view, while local explanations focus on individual predictions. Both are useful because a model may behave fairly on average but still produce problematic decisions for particular cases.

Several techniques can be used to interpret machine learning models:

• Feature importance: Many tree-based models calculate an importance score for each feature based on how much it contributes to reducing impurity or error. Higher values indicate greater influence within that model.
• Permutation importance: A feature is randomly shuffled, and the change in validation performance is measured. A large performance decrease suggests that the feature is important for prediction.
• SHAP explanations: SHAP values assign each feature a contribution to a prediction relative to a baseline prediction. Positive values push the prediction higher, while negative values push it lower.

These methods help identify influential variables, detect irrelevant features, discover possible bias, and communicate model behavior. However, feature importance indicates association with predictions and does not by itself establish causation.