Unit 2: SUPERVISED LEARNING: REGRESSION - Subjective Questions

Supervised learning is a machine learning approach in which a model learns a relationship between input variables and a known target variable using labeled training data.

• In regression, the target variable is continuous, such as house price, temperature, or sales revenue.
• In classification, the target variable belongs to discrete categories, such as spam or not spam.
• Regression models predict numerical values, while classification models predict class labels or class probabilities.
• Examples of regression algorithms include linear regression, polynomial regression, and logistic regression for binary probability modeling.

The objective of regression is generally to minimize the difference between actual and predicted numerical values.

Simple linear regression models the relationship between one independent variable and one dependent variable using a straight line.

The general model is:

where:

• is the intercept, representing the expected value of when .
• is the slope, representing the average change in for a one-unit increase in .
• is the random error term.

After estimating the parameters from data, the prediction equation becomes:

The fitted line is selected so that the predicted values are as close as possible to the observed values according to a chosen loss function, usually the sum of squared errors.

The main assumptions of simple linear regression are:

• Linearity: The relationship between and the expected value of should be approximately linear.
• Independence: Observations and their error terms should be independent of one another.
• Homoscedasticity: The variance of the error terms should remain approximately constant for all values of .
• Normality of errors: Errors should be approximately normally distributed, especially when confidence intervals and hypothesis tests are required.
• Absence of influential outliers: Extreme observations should not disproportionately determine the fitted line.

Violating these assumptions can lead to biased estimates, unreliable standard errors, poor predictions, or misleading conclusions about the relationship between variables.

Multiple linear regression predicts a continuous dependent variable using two or more independent variables. Its general form is:

where is the intercept and are the estimated coefficients.

Comparison:

• Simple linear regression uses one predictor, while multiple linear regression uses several predictors.
• In multiple regression, each coefficient measures the effect of its predictor while holding the other predictors constant.
• Multiple regression can model more realistic situations involving several influencing factors.
• It may suffer from multicollinearity when predictors are strongly correlated with one another.
• Both methods commonly use ordinary least squares to estimate coefficients and minimize squared residuals.

Multicollinearity occurs when two or more independent variables in a multiple regression model are strongly correlated.

Effects include:

• Unstable coefficient estimates.
• Large standard errors and reduced statistical significance.
• Coefficients that change substantially when variables are added or removed.
• Difficulty interpreting the individual effect of each predictor.
• Occasionally, unexpected coefficient signs.

Detection methods include:

• Examining a correlation matrix for highly correlated predictors.
• Calculating the variance inflation factor, or VIF.
• Inspecting the condition number of the design matrix.

Possible remedies include removing redundant variables, combining related variables, collecting more data, or using regularization methods such as ridge regression.

Polynomial regression extends linear regression by including powers of an independent variable. A polynomial model of degree can be written as:

Although the curve is nonlinear in , the model is linear in its coefficients . Therefore, ordinary least squares can still be used for estimation.

Advantages:

• Captures curved relationships between variables.
• Can provide a better fit than a straight line when the relationship is nonlinear.

Limitations:

• High-degree polynomials may overfit the training data.
• Predictions can become unstable at the boundaries of the observed range.
• Interpretation becomes more difficult as the degree increases.

The polynomial degree should be selected using validation techniques and evaluation metrics.

Underfitting occurs when the model is too simple to represent the underlying relationship.

• It usually has high training and testing error.
• A low-degree polynomial may fail to capture important curvature.
• It can be addressed by increasing model complexity or adding useful predictors.

Overfitting occurs when the model is excessively complex and learns noise in the training data.

• Training error is very low, but testing error is high.
• A high-degree polynomial often produces unnecessary oscillations.
• It can be addressed by reducing the polynomial degree, using cross-validation, collecting more data, or applying regularization.

The goal is to choose a model that generalizes well to unseen observations rather than merely fitting the training sample.

For observations and predictions , the residual for observation is:

The ordinary least squares method estimates the coefficients by minimizing the sum of squared residuals:

For a multiple linear regression model:

Therefore, the objective is:

A residual measures the unexplained difference between an observed value and the value predicted by the model. Squaring residuals prevents positive and negative errors from canceling and gives greater weight to large errors.

A linear regression model can be written in matrix form as:

where is the response vector, is the design matrix, is the coefficient vector, and is the error vector.

The ordinary least squares estimator is:

This formula is valid when is invertible. In practice, this requires that:

• The predictors are not perfectly linearly dependent.
• The design matrix has full column rank.
• The observations provide enough information to estimate all coefficients.

If perfect multicollinearity exists, the inverse does not exist and variables must be removed, transformed, or handled with another estimation technique.

Correlation measures the strength and direction of association between two variables. The Pearson correlation coefficient is defined as:

Its value lies between and :

• indicates a perfect positive linear relationship.
• indicates a perfect negative linear relationship.
• indicates no linear relationship.
• Values close to either or indicate a strong linear association.

Correlation does not prove causation. It may also fail to detect nonlinear relationships and can be strongly affected by outliers.

Correlation and regression differ in the following ways:

• Purpose: Correlation measures the strength and direction of association, whereas regression models a relationship for explanation or prediction.
• Role of variables: Correlation treats both variables symmetrically. Regression identifies a dependent variable and one or more independent variables.
• Output: Correlation produces a coefficient between and . Regression produces an equation containing estimated coefficients.
• Units: Correlation is unit-free. Regression coefficients have units determined by the variables.
• Prediction: Correlation alone does not provide a prediction equation, while regression is explicitly designed for prediction.
• Causation: Neither correlation nor regression automatically establishes causation.

A strong correlation may support the use of regression, but regression analysis requires additional assumptions and diagnostics.

Logistic regression is a supervised learning method used to model the probability of a binary outcome. It uses the logistic function:

where:

The output probability is between and . A classification threshold, commonly , can be used to convert the probability into a class label.

Ordinary linear regression is unsuitable for binary outcomes because:

• Its predictions can be less than or greater than .
• Its errors do not satisfy the usual constant-variance assumption for binary data.
• A linear change in predictors does not produce a linear change in probability.

Logistic regression instead models the log-odds linearly:

In logistic regression, a coefficient describes the change in log-odds associated with a one-unit increase in a predictor, while all other predictors remain constant.

The model is:

For predictor :

• is the change in log-odds for a one-unit increase in .
• is the odds ratio.
• If , increasing the predictor increases the odds of the positive class.
• If , increasing the predictor decreases the odds.
• If , the predictor has no change in the odds.

The intercept represents the log-odds of the positive class when all predictors equal zero. Its practical meaning depends on whether zero is meaningful for the predictors.

Mean Absolute Error, or MAE, is the average absolute difference between actual and predicted values:

Advantages:

• It is easy to interpret because it represents the average magnitude of prediction error in the original units.
• It treats all errors linearly.
• It is less sensitive to outliers than MSE and RMSE.
• It is useful when all deviations should have approximately equal importance.

Limitations:

• It does not strongly penalize large errors.
• The absolute value function is not differentiable at zero, which can make some optimization procedures less convenient.
• MAE alone does not indicate whether errors are systematically positive or negative.

Lower MAE values indicate better predictive accuracy for the same target variable and dataset.

Mean Squared Error, or MSE, is the average of the squared prediction errors:

Comparison with MAE:

• MAE uses absolute errors, while MSE uses squared errors.
• MSE gives disproportionately greater weight to large errors.
• MAE is generally more robust to outliers.
• MSE is differentiable everywhere and is convenient for mathematical optimization.
• MSE has squared units, whereas MAE has the same units as the target variable.

MSE is appropriate when large errors are particularly undesirable. MAE may be preferable when a robust and directly interpretable average error is required.

The R-squared score measures the proportion of variation in the dependent variable explained by the regression model. It is defined as:

where:

and

Here, is the unexplained variation and is the total variation around the mean.

• indicates a perfect fit.
• means the model performs no better than predicting the mean, under the usual comparison.
• A larger generally indicates that more variation is explained.
• Adding predictors can increase or leave unchanged, even if the predictors are not useful.

Therefore, should be considered together with error metrics, validation results, and model assumptions.

Root Mean Squared Error, or RMSE, is the square root of the mean squared error:

Its relationship with MSE is:

RMSE has the same units as the dependent variable, making it easier to interpret than MSE. It can be understood as a typical magnitude of prediction error, although it is more strongly affected by large errors than MAE.

• A lower RMSE indicates better predictive performance.
• RMSE is useful when large errors should receive greater importance.
• RMSE should be compared only across models evaluated on the same target scale and dataset.
• It should not be interpreted as exactly equal to the error made for every observation.

MAE:

• Formula: .
• Expressed in the original target units.
• Relatively robust to outliers.

MSE:

• Formula: .
• Expressed in squared units.
• Strongly penalizes large errors.

RMSE:

• Formula: .
• Expressed in the original target units.
• Sensitive to large errors while remaining interpretable.

R-squared:

• Formula: .
• Measures explained variation rather than direct error magnitude.
• Does not necessarily improve when predictive performance on new data improves.

MAE, MSE, and RMSE are error measures where lower values are better. R-squared is generally a goodness-of-fit measure where higher values are preferred, but all metrics should be evaluated on validation or test data.

The errors are:

The absolute errors are , so:

The squared errors are also , so:

Therefore:

Interpretation:

• The average absolute prediction error is unit.
• The average squared error is squared unit.
• The RMSE is unit, indicating that the typical error magnitude is approximately one unit for this small set of observations.
• Because all errors have the same magnitude, MAE and RMSE are equal in this example.

Residual analysis examines the differences between observed and predicted values:

Useful diagnostic procedures include:

• Residuals versus fitted values: A random horizontal pattern suggests an appropriate functional form. Curvature may indicate nonlinearity.
• Residual spread: Increasing or decreasing spread suggests heteroscedasticity.
• Histogram or Q-Q plot: These help assess whether residuals are approximately normally distributed.
• Residuals versus observation order: Patterns may indicate dependence or autocorrelation.
• Influential-point analysis: Extreme residuals or high-leverage observations may have excessive influence on coefficient estimates.

A good model generally has residuals centered near zero, no systematic pattern, approximately constant variance, and no unjustified influential observations.