Unit 9: Regression Models - Subjective Questions

Regression analysis is a statistical and machine-learning technique used to model the relationship between a dependent or target variable and one or more independent or predictor variables.

Main objectives:

• Prediction: Estimate the value of a target variable for unseen observations.
• Explanation: Determine how predictors influence the target variable.
• Trend estimation: Identify patterns and relationships in data.
• Decision-making: Support evidence-based decisions using estimated outcomes.

A general regression model can be written as:

where is the target, represents the predictors, is the relationship learned by the model, and is random error.

Applications include predicting house prices, estimating sales, forecasting demand, assessing risk, and analyzing the effects of variables such as education or advertising expenditure.

Regression and correlation both examine relationships between variables, but they serve different purposes.

• Purpose: Correlation measures the strength and direction of association, whereas regression models the relationship and predicts a target.
• Variable roles: Correlation treats the variables symmetrically. Regression identifies dependent and independent variables.
• Output: Correlation produces a coefficient, usually , between and . Regression produces an equation such as .
• Units: Correlation is unit-free, while regression coefficients depend on the units of the variables.
• Prediction: Correlation alone is not a predictive model; regression can generate predictions.

Neither method automatically establishes causation. A strong association may result from confounding variables or coincidence.

Major regression types include:

• Simple linear regression: Uses one predictor to estimate a continuous target, such as predicting salary from experience.
• Multiple linear regression: Uses several predictors for a continuous target, such as predicting house price from area, age, and location.
• Polynomial regression: Includes powers of predictors to model nonlinear trends.
• Logistic regression: Estimates the probability of a categorical outcome, such as whether a customer will churn.
• Ridge regression: Applies an penalty to reduce overfitting and handle correlated predictors.
• Lasso regression: Applies an penalty and can set some coefficients to zero, thereby performing feature selection.
• Elastic Net regression: Combines and penalties.

The appropriate type depends on the target variable, the shape of the relationship, multicollinearity, and the need for regularization or feature selection.

Simple linear regression represents a continuous target as a linear function of one predictor :

Its fitted form is:

• is the intercept, representing the expected value of when .
• is the slope, representing the expected change in for a one-unit increase in .
• is the random error not explained by the model.
• is the predicted target value.
• The residual is .

For example, in , the predicted value is at , and each one-unit increase in raises the prediction by units.

For the model , ordinary least squares minimizes the residual sum of squares:

Set the partial derivatives equal to zero:

Solving these normal equations gives:

and

Thus, the fitted regression line passes through . The slope is the sample covariance numerator divided by the variation in .

The principal assumptions are:

• Linearity: The expected target is a linear function of the predictors and coefficients.
• Independence: Observations and their errors are independent.
• Zero conditional mean: Errors satisfy , meaning relevant systematic effects are not left in the errors.
• Homoscedasticity: Error variance is constant: .
• No perfect multicollinearity: No predictor is an exact linear combination of other predictors.
• Normality of errors: Errors are approximately normal when classical small-sample confidence intervals and hypothesis tests are required.

Violations can cause biased estimates, inefficient estimates, unreliable standard errors, or poor predictions. Residual plots, variance inflation factors, and quantile-quantile plots help diagnose these issues.

Multiple linear regression models a continuous target using several predictors:

Here, is the target vector, is the design matrix, is the coefficient vector, and is the error vector.

The least-squares objective is:

Differentiating and setting the gradient to zero gives:

Therefore, the normal equations are:

If is invertible, then:

In practice, numerically stable methods such as QR decomposition or singular value decomposition are preferred over explicitly computing the inverse.

• Categorical variables: A category with levels is commonly represented using dummy variables, leaving one level as the reference. Each coefficient measures a difference from that reference level.
• Polynomial terms: Terms such as or allow a linear regression algorithm to represent curved relationships. The model remains linear in its coefficients.
• Interaction terms: A product such as allows the effect of one predictor to depend on another.

For example:

Here, models curvature, may be a dummy variable, and is an interaction. When interaction or polynomial terms are included, their lower-order components should normally remain in the model to preserve interpretability and hierarchy.

A residual is the difference between an observed and predicted value:

Important diagnostic checks include:

• Residuals versus fitted values: Random scatter around zero supports linearity and constant variance. Curvature suggests nonlinearity, while a funnel shape suggests heteroscedasticity.
• Quantile-quantile plot: Large departures from a straight line suggest non-normal errors or outliers.
• Residuals versus time or observation order: Patterns may indicate autocorrelation.
• Leverage: Identifies observations with unusual predictor values.
• Cook's distance: Identifies observations that substantially affect fitted coefficients.
• Standardized residuals: Help locate observations whose errors are unusually large.

Possible remedies include transforming variables, adding nonlinear terms, using robust standard errors, investigating influential observations, or choosing a more appropriate model.

Logistic regression is a supervised classification algorithm that estimates the probability of a binary event. For predictors , it defines:

where

The sigmoid function maps every real-valued into the interval , so the output can be interpreted as a probability. A threshold, such as , converts this probability into a class prediction.

Unlike ordinary linear regression, logistic regression cannot produce probabilities below zero or above one. It also models a linear relationship between predictors and the log-odds, rather than between predictors and the class label directly.

If an event has probability , its odds are:

The log-odds, or logit, are:

Holding other predictors constant, a one-unit increase in changes the log-odds by . Exponentiating the coefficient gives the odds ratio:

• If , the odds increase.
• If , the odds decrease.
• If , there is no change in odds.

For example, if , a one-unit increase in multiplies the odds by , corresponding to a increase in odds, not necessarily a increase in probability.

For observation , logistic regression predicts:

For , the Bernoulli probability is:

Assuming independent observations, the likelihood is:

Taking logarithms gives the log-likelihood:

Training maximizes this log-likelihood, or equivalently minimizes the negative average log-likelihood, called binary cross-entropy:

Because there is generally no closed-form coefficient solution, optimization methods such as gradient descent, Newton's method, or limited-memory quasi-Newton methods are used.

Logistic regression first produces a probability . It then predicts the positive class when , where is a chosen threshold.

At the conventional threshold :

Thus, ordinary logistic regression has a linear decision boundary in the supplied feature space. Polynomial feature transformations can create nonlinear boundaries in the original space.

Changing the threshold affects errors:

• Lowering usually increases recall but may reduce precision.
• Raising usually increases precision but may reduce recall.
• The best threshold depends on class imbalance and the relative costs of false positives and false negatives.

The threshold should be selected using validation data and an appropriate metric or explicit decision cost, rather than automatically assuming that is optimal.

Both are supervised parametric models, learn coefficients from labeled data, can include transformed features, and can be regularized. Applying linear regression directly to a binary target is generally unsuitable because its outputs are unbounded and its error assumptions do not match Bernoulli outcomes.

Regularization adds a penalty for coefficient complexity to a model's training objective. A generic regularized objective is:

where is a penalty and controls its strength.

Regularization helps by:

• Shrinking coefficients and limiting model complexity.
• Reducing variance and overfitting.
• Improving generalization to unseen data.
• Stabilizing estimates when predictors are highly correlated.
• Potentially performing feature selection when an penalty is used.

A very small provides little control and may overfit. A very large can cause underfitting. The intercept is usually not penalized, and is commonly selected through cross-validation.

For residual sum of squares :

Ridge regression uses an penalty:

It smoothly shrinks coefficients toward zero but rarely makes them exactly zero. It is useful with correlated predictors.

Lasso regression uses an penalty:

It can set coefficients exactly to zero and therefore performs feature selection. With strongly correlated predictors, it may select one somewhat arbitrarily.

Elastic Net combines both penalties:

Here, . Elastic Net combines Lasso's sparsity with Ridge's stability and can retain groups of correlated features. Predictors should generally be standardized before applying these penalties.

Regularization controls the bias-variance trade-off:

• A weak penalty produces a flexible model with lower training bias but potentially high variance and overfitting.
• A strong penalty produces a simpler model with lower variance but potentially high bias and underfitting.

Feature scaling is essential because penalties act directly on coefficient magnitudes. If predictors use different scales, a coefficient may be penalized more or less merely because of its measurement unit. Standardization commonly uses:

Cross-validation estimates out-of-sample performance for candidate values of and, for Elastic Net, . Preprocessing must be fitted separately within each training fold to prevent data leakage. After choosing hyperparameters, the complete pipeline is fitted on the training data and evaluated once on an untouched test set.

For observations:

Mean absolute error:

It is in the target's units and is relatively resistant to outliers.

Mean squared error:

It penalizes large errors strongly but is measured in squared units.

Root mean squared error:

It returns to the target's units while retaining sensitivity to large errors.

Coefficient of determination:

It measures performance relative to predicting the mean. Higher is generally better, but test-set may be negative. No single metric is universally best; selection should reflect the practical cost of errors.

A binary confusion matrix contains true positives , true negatives , false positives , and false negatives .

The main metrics are:

• Precision is important when false positives are costly.
• Recall is important when false negatives are costly.
• Specificity measures correct rejection of negative cases.
• F1-score balances precision and recall.
• Accuracy can be misleading for imbalanced data because a model may predict only the majority class and still appear accurate.

• ROC curve: Plots the true positive rate against the false positive rate across thresholds. ROC-AUC summarizes ranking ability; indicates ideal ranking and approximately indicates random ranking.
• Precision-recall curve: Plots precision against recall across thresholds. It is often more informative when the positive class is rare because it emphasizes positive-class performance.
• Log loss: Evaluates predicted probabilities rather than only hard labels:

Lower log loss is better, and confident incorrect predictions receive large penalties.

ROC-AUC is useful for general ranking comparisons, precision-recall analysis is preferable for severe class imbalance, and log loss is appropriate when accurate probability estimates matter. Threshold-based operational metrics should also be reported when deployment uses a fixed decision threshold.