Unit 3: Supervised Machine Learning - Subjective Questions

Supervised machine learning is a learning approach in which a model is trained using labeled examples. Each training example contains an input feature vector and a known target value . The model learns a mapping that can be used to predict targets for unseen inputs.

Main components:

• Features: Input variables used for prediction.
• Target or label: The expected output associated with an input.
• Training data: Labeled examples used to learn model parameters.
• Learning algorithm: The procedure that minimizes prediction error.
• Loss function: A measure of the difference between actual and predicted outputs.
• Test data: Unseen examples used to evaluate generalization.

Regression versus classification:

• Regression predicts a continuous numerical value, such as price, temperature, or salary.
• Classification predicts a discrete class label, such as spam or not spam.
• Mean Squared Error is commonly used for regression, while accuracy, precision, recall, and log loss are commonly used for classification.

Simple linear regression models the relationship between one independent variable and a continuous dependent variable as

where is the intercept and is the slope. The residual for the th observation is .

The least-squares method chooses and to minimize the Sum of Squared Errors:

Setting the partial derivatives of with respect to and equal to zero gives the normal equations. Solving them produces

and

Interpretation:

• represents the expected change in for a one-unit increase in .
• is the predicted value of when .
• The fitted line minimizes the total squared vertical distance between observed and predicted values.

The major assumptions of linear regression are:

• Linearity: The expected target has a linear relationship with the predictors.
• Independence: Observations and their errors are independent.
• Homoscedasticity: The error variance remains approximately constant for all predictor values.
• Normality of errors: Residuals are approximately normally distributed, especially when confidence intervals or significance tests are required.
• No perfect multicollinearity: In multiple regression, predictors should not be exact linear combinations of one another.
• Zero conditional mean: The expected error given the predictors is zero, written as .

Consequences of violations:

• Nonlinearity can cause systematic prediction errors and underfitting.
• Correlated observations can make standard errors and statistical conclusions unreliable.
• Heteroscedasticity can produce incorrect confidence intervals and hypothesis tests.
• Severe multicollinearity makes coefficient estimates unstable and difficult to interpret.
• Important omitted variables can bias coefficients.
• Outliers may strongly influence the fitted line because errors are squared.

Residual plots, correlation analysis, and domain knowledge can help detect these problems.

Let be the actual value, the predicted value, and the number of observations.

• Mean Absolute Error:

It is easy to interpret and is less sensitive to extreme errors than MSE.

• Mean Squared Error:

It penalizes large errors more heavily because each error is squared.

• Root Mean Squared Error:

It is expressed in the same units as the target variable.

• Coefficient of determination:

measures the proportion of target variation explained by the model. A value near generally indicates a better fit, while a negative test-set value indicates performance worse than predicting the mean. These metrics should be computed on validation or test data when evaluating generalization.

Polynomial regression extends linear regression by including powers of an input variable. A degree- model is written as

Although the model is nonlinear with respect to , it is linear with respect to the coefficients . Therefore, its coefficients can be estimated using linear least squares after creating polynomial features.

Effect of degree:

• Degree produces a straight line.
• Degree produces a quadratic curve.
• Higher degrees allow more bends and can represent more complex relationships.
• A degree that is too low can cause underfitting and high bias.
• A degree that is too high can fit noise, causing overfitting and high variance.

The degree should be selected using validation data or cross-validation. Polynomial features may also require scaling because powers of a feature can have very different numerical ranges.

A decision tree regressor divides the feature space into a collection of non-overlapping regions using a sequence of binary decisions.

Working procedure:

• Begin with all training observations at the root node.
• Consider candidate splits of the form , where is a feature and is a threshold.
• Divide the observations into left and right child nodes.
• Select the split that produces the greatest reduction in prediction error.
• Repeat the process recursively until a stopping condition is reached.

For squared-error regression, the impurity of a node can be measured by

A split is selected to minimize

The prediction at a leaf is usually the mean target value of its training observations:

Decision trees can model nonlinear relationships and feature interactions, but unrestricted trees often overfit. Maximum depth, minimum samples per leaf, and pruning can control their complexity.

Linear regression:

• Models a linear relationship between predictors and target.
• Has highly interpretable coefficients.
• May require feature scaling for some optimization methods, though ordinary least squares itself does not require it.
• Usually has relatively high bias and low variance.
• Cannot naturally capture strong nonlinear patterns without transformed features.

Polynomial regression:

• Represents curved relationships by adding powers and interactions.
• Is interpretable at low degrees but becomes difficult to interpret at high degrees.
• Often benefits from feature scaling.
• Becomes increasingly sensitive to noise as the degree rises.
• May behave poorly when extrapolating beyond the observed range.

Decision tree regression:

• Uses hierarchical feature-based splits and piecewise-constant predictions.
• Can represent nonlinear patterns and interactions automatically.
• Does not require feature scaling.
• Can handle mixed feature effects but produces discontinuous predictions.
• Deep trees have high variance and can overfit.

Thus, the preferred model depends on the relationship in the data, interpretability requirements, available sample size, and validation performance.

Logistic regression estimates the probability that an input belongs to the positive class. It assumes that the log-odds are a linear function of the features:

Exponentiating both sides gives

Solving for produces the sigmoid function:

Because the sigmoid output lies between and , it can be interpreted as a probability.

Decision rule:

• Predict class if .
• Predict class if .
• The common threshold is , but it may be changed according to error costs or class imbalance.

The coefficients are generally estimated by maximum likelihood, equivalently by minimizing binary cross-entropy:

A positive coefficient increases the log-odds of class , while a negative coefficient decreases them.

Ordinary linear regression is unsuitable for binary classification for several reasons:

• Its output is unbounded, so predictions may be less than or greater than and cannot always be interpreted as probabilities.
• A fixed threshold applied to a regression line may create an inappropriate classification boundary.
• Binary targets do not satisfy the constant-variance and normal-error assumptions of ordinary linear regression.
• Squared error is not the most suitable loss for modeling Bernoulli outcomes.
• Outliers can strongly move the fitted regression line and therefore the classification boundary.

Logistic regression addresses these limitations by applying the sigmoid function:

This confines predictions to the interval . It models a Bernoulli-distributed target and uses maximum likelihood or cross-entropy loss. A threshold converts the estimated probability into a class label. Logistic regression therefore provides both probabilistic output and a principled framework for binary classification.

K-Nearest Neighbour, or KNN, is a non-parametric, instance-based classification algorithm. It stores the training examples and classifies a new point using nearby observations.

Algorithm:

• Choose a value of .
• Calculate the distance from the query point to every training point.
• Select the points having the smallest distances.
• Assign the class occurring most frequently among those neighbours.

A common distance measure is Euclidean distance:

Other choices include Manhattan distance and Minkowski distance.

Role of :

• A small creates flexible boundaries but is sensitive to noise and has high variance.
• A large creates smoother boundaries but may ignore local structure and have high bias.
• An odd is often used in binary classification to reduce ties.

Feature scaling is important because a feature with a large numerical range can dominate the distance. Standardization or normalization should therefore be fitted on the training data before computing distances.

Advantages of KNN:

• It is simple to understand and implement.
• It makes few assumptions about the data distribution.
• It can learn nonlinear and irregular decision boundaries.
• It naturally supports multiclass classification.
• Training is fast because the algorithm mainly stores the examples.

Limitations:

• Prediction can be slow because distances to many training examples must be calculated.
• It may require substantial memory for a large training set.
• It is sensitive to the choice of , distance metric, and feature scale.
• Noisy or irrelevant features can distort neighbourhoods.
• It suffers from the curse of dimensionality: in high-dimensional spaces, distances become less informative and data points become sparse.
• Class imbalance can cause the majority class to dominate voting.

The basic prediction cost for one query is approximately for training examples and features. Search structures can accelerate some low-dimensional problems, but their benefit decreases in high dimensions. Weighted voting, where closer neighbours receive larger weights, can improve some datasets.

A decision tree classifier chooses feature thresholds that make child nodes purer than their parent. If is the proportion of class in a node, two common impurity measures are:

Gini impurity:

Entropy:

Both measures are zero when every observation in a node belongs to the same class. A candidate split creates left and right children. Its weighted impurity is

where and are the child sizes. The impurity reduction is

The tree selects the split with the largest impurity reduction. At a leaf, the predicted class is usually the majority class, while class probabilities can be estimated from class proportions in that leaf. Gini and entropy often produce similar trees, though Gini is slightly simpler to compute.

A decision tree classifier has a hierarchical structure:

• The root node contains the complete training sample.
• Internal nodes test a feature condition.
• Branches represent outcomes of that condition.
• Leaf nodes produce class predictions or class probabilities.

For prediction, an example starts at the root and follows the branch associated with each condition until it reaches a leaf. The leaf's majority class is returned as the prediction.

Advantages:

• Easy to visualize and explain.
• Captures nonlinear relationships and feature interactions.
• Requires little preprocessing and no feature scaling.
• Can perform implicit feature selection.
• Supports binary and multiclass problems.

Limitations:

• A deep tree can overfit and have high variance.
• Small changes in training data may produce a substantially different tree.
• Greedy split selection does not guarantee a globally optimal tree.
• Axis-aligned splits may require a complex tree for some boundaries.
• Unconstrained trees may favor highly specific rules.

Complexity can be controlled using maximum depth, minimum samples for splitting, minimum samples per leaf, maximum leaf nodes, and post-pruning.

A Random Forest is an ensemble of decision trees designed to improve predictive performance and reduce the variance of a single tree.

Training process:

• For each tree, create a bootstrap sample by drawing training observations with replacement.
• Grow a decision tree using that sample.
• At each split, evaluate only a random subset of the available features.
• Repeat the process to create many diverse trees.

Aggregation:

• In classification, the forest normally predicts by majority vote or by averaging class probabilities.
• In regression, it averages the numerical predictions of the trees.

Bootstrap sampling and random feature selection reduce correlation among trees. Averaging less-correlated models reduces variance while retaining the ability to represent complex patterns.

An observation excluded from a tree's bootstrap sample is called an out-of-bag observation for that tree. Its out-of-bag prediction is obtained from trees that did not train on it. Combining these predictions provides an out-of-bag estimate of generalization error without a separate validation set.

Random Forests are robust and effective but are less interpretable and require more memory and computation than one decision tree.

Single decision tree:

• Uses one hierarchical collection of decision rules.
• Is easy to inspect, visualize, and explain.
• Trains and predicts relatively quickly.
• Can have high variance and may overfit the training set.
• Is unstable because small data changes can alter its structure.

Random Forest:

• Combines many decision trees trained on bootstrap samples.
• Uses random feature subsets to make trees less correlated.
• Usually provides better generalization and greater robustness to noise.
• Reduces variance by aggregating predictions.
• Can estimate out-of-bag error and feature importance.
• Requires more computation and memory.
• Is harder to interpret as a complete model.

A decision tree may be preferred when interpretability and simple rules are essential. A Random Forest is generally preferred when predictive accuracy and stability are more important. Although a Random Forest reduces overfitting relative to a deep individual tree, its hyperparameters should still be validated.

Bayes' theorem relates a posterior probability to a likelihood and a prior probability:

Here:

• is the posterior probability of class after observing features .
• is the likelihood of observing in class .
• is the prior probability of class .
• is the evidence and is the same for every candidate class.

Naïve Bayes assumes that features are conditionally independent given the class:

Therefore, the prediction rule is

In implementation, logarithms are commonly used to avoid numerical underflow:

Despite its strong independence assumption, Naïve Bayes often performs well in text classification and other high-dimensional problems.

Gaussian Naïve Bayes:

• Used for continuous numerical features.
• Assumes each feature follows a Gaussian distribution within each class.
• Its likelihood is

• Suitable for measurements such as height, temperature, or sensor values.

Multinomial Naïve Bayes:

• Used for non-negative counts or frequencies.
• Commonly applied to word-count or term-frequency vectors.
• Suitable for document topic classification and spam filtering.

Bernoulli Naïve Bayes:

• Used for binary features indicating presence or absence.
• Considers both the occurrence and non-occurrence of a feature.
• Suitable when documents are represented by whether each vocabulary term appears.

The correct variant depends on the statistical form of the input features. Smoothing, such as Laplace smoothing, is often added to Multinomial and Bernoulli Naïve Bayes to prevent zero probabilities for unseen feature-class combinations.

Bias is the systematic error caused by restrictive assumptions in a learning algorithm. A high-bias model may fail to capture the true pattern.

Variance is the model's sensitivity to changes in the training sample. A high-variance model may learn random fluctuations and produce very different predictions from different samples.

For squared-error prediction, expected error can be conceptually decomposed as

where is irreducible noise.

Trade-off:

• Simple models tend to have high bias and low variance.
• Highly complex models tend to have low training bias but high variance.
• Increasing complexity can initially reduce test error by capturing genuine patterns.
• Beyond an appropriate complexity, test error may rise because the model starts fitting noise.

The goal is not to minimize bias or variance independently. It is to choose model complexity and regularization that minimize generalization error. Validation sets and cross-validation are commonly used to find this balance.

Underfitting occurs when a model is too simple to learn the underlying relationship.

Indicators of underfitting:

• High training error.
• High validation or test error.
• Small gap between training and validation performance.
• Poor predictions caused by excessive bias.

Possible corrections:

• Use a more expressive model.
• Add useful features or nonlinear transformations.
• Reduce excessive regularization.
• Train longer when optimization is incomplete.

Overfitting occurs when a model learns noise and sample-specific details rather than general patterns.

Indicators of overfitting:

• Very low training error.
• Significantly higher validation or test error.
• A large training-validation performance gap.
• Unstable predictions across different training samples.

Possible corrections:

• Collect more representative training data.
• Reduce model complexity.
• Apply regularization or tree pruning.
• Perform feature selection.
• Use cross-validation for hyperparameter tuning.
• Use suitable ensemble methods or early stopping.

All preprocessing and model selection should use only training or validation information so that the test set remains an unbiased final evaluation.

Model complexity:

• Increasing polynomial degree, tree depth, or the number of flexible decision rules generally reduces training error.
• Too little complexity causes underfitting.
• Too much complexity can cause overfitting.

Regularization:

• Regularization adds a penalty for complexity to the objective function.
• Ridge regression uses an penalty:

• Lasso regression uses an penalty:

• Larger usually increases bias and decreases variance.

Data splitting:

• The training set estimates model parameters.
• The validation set selects models and hyperparameters.
• The test set is used once for final, unbiased evaluation.

Cross-validation:

• In -fold cross-validation, data is divided into subsets.
• The model trains on folds and validates on the remaining fold.
• This is repeated so every fold is used for validation, and the scores are averaged.

Together, these methods estimate generalization performance and help select a model that balances bias and variance. Data preprocessing must be fitted separately within each training fold to prevent data leakage.