Unit 9: Regression Models

I. Foundations of Regression — Learning Relationships from Data

Regression is a supervised machine-learning and statistical framework for estimating how a dependent variable changes with one or more independent variables. A model learns this relationship from labelled observations and then supports prediction, explanation, or probability estimation on unseen data.

A. Introduction to regression

Regression converts observed input–output relationships into a mathematical function that can generalize beyond the training data.

• Core representation: A regression model expresses an outcome as a systematic component plus unexplained error.

y = f(X) + ε

• (y): dependent variable, response, or target.
• (X): one or more independent variables, features, or predictors.
• (f): relationship learned from data.
• (\varepsilon): random error not explained by the predictors.

• Supervised learning: Training data contain both features and known targets; for example, house area and location are features, while sale price is the target.
• Learning objective: The algorithm selects model parameters that minimize a loss function, such as mean squared error for continuous outcomes or log loss for class probabilities.
• Prediction and inference:
  1. Prediction: Estimates an unknown outcome, such as next month’s sales.
  2. Inference: Examines how a predictor is associated with the outcome, such as the expected sales change for each additional advertising unit.
• General assumptions: Exact assumptions depend on the model, but common concerns include representative observations, correctly measured variables, suitable functional form, and independence between training and test data.
• Generalization: Good performance on training data is insufficient; the model must also predict unseen observations accurately.
• Data convention: With (n) observations and (p) features, the design matrix (X) has (n) rows and (p) columns, while the target vector (y) contains (n) outcomes.
• Workflow: Regression normally involves data cleaning, feature selection, train–test splitting, model fitting, prediction, metric-based evaluation, and diagnostic analysis.

II. Regression Families — Choosing a Model for the Target and Relationship

Regression methods differ in target type, functional form, assumptions, and resistance to model complexity.

A. Types of regression

The appropriate regression type is determined primarily by the outcome being predicted and the shape of its relationship with the features.

• Simple linear regression: Uses one predictor to model a continuous target; for example, predicting electricity consumption from temperature.
• Multiple linear regression: Uses two or more predictors, such as floor area, age, and location score, to estimate a house price.
• Polynomial regression: Adds powers of a feature to represent curvature.

ŷ = β₀ + β₁x + β₂x²

• (\hat y): predicted continuous outcome.
• (x): predictor.
• (\beta_0): intercept.
• (\beta_1,\beta_2): learned coefficients.

• Logistic regression: Estimates the probability of a categorical outcome, commonly a binary class such as “default” or “no default.”
• Ridge regression: Applies an (L_2) penalty to shrink coefficients and stabilize models with correlated predictors.
• Lasso regression: Applies an (L_1) penalty that can reduce some coefficients exactly to zero, thereby performing feature selection.
• Elastic Net regression: Combines (L_1) and (L_2) penalties when both sparsity and coefficient stability are desirable.
• Count regression: Poisson or negative-binomial regression models non-negative counts, such as daily customer arrivals.
• Robust regression: Reduces the influence of extreme observations; Huber regression, for example, behaves quadratically for small residuals and approximately linearly for large ones.
• Selection principle: Continuous targets often suggest linear-family models, class labels suggest logistic regression, and count targets suggest count-specific models; diagnostics then test whether the choice fits the data.

III. Linear Regression — Predicting Continuous Outcomes

Linear regression models the conditional mean of a continuous target as a linear combination of predictors.

A. Machine linear regression

Machine linear regression learns coefficients that produce the smallest prediction errors under a specified loss function.

• Model equation: Multiple linear regression uses:

ŷᵢ = β₀ + β₁xᵢ₁ + β₂xᵢ₂ + ... + βₚxᵢₚ

• (\hat y_i): prediction for observation (i).
• (x_{ij}): value of feature (j) for observation (i).
• (\beta_0): intercept.
• (\beta_j): change in predicted (y) for a one-unit increase in feature (j), holding other features constant.
• (p): number of features.

• Ordinary least squares: OLS chooses coefficients that minimize the residual sum of squares.

RSS = Σᵢ(yᵢ − ŷᵢ)²

• (RSS): residual sum of squares.
• (y_i-\hat y_i): residual for observation (i).
• (\Sigma_i): summation over all observations.

• Fitting methods: Coefficients may be obtained through a closed-form matrix solution or iterative optimization such as gradient descent; the latter is useful for large datasets.
• Interpretation: If a standardized advertising coefficient is (2.4), increasing that feature by one standardized unit raises the predicted target by (2.4) units when other variables remain fixed.
• Important conditions:
  • Linearity: The conditional mean of the target is linear in the model parameters.
  • Independent errors: Residuals should not show systematic dependence.
  • Homoscedasticity: Residual variance should remain approximately constant across fitted values.
  • Low multicollinearity: Predictors should not be near-linear combinations of one another.
• Categorical features: Categories are encoded using indicator variables; one reference category is normally omitted to avoid perfect multicollinearity.
• Diagnostics: Residual-versus-fitted plots reveal curvature or changing variance, while large standardized residuals and high-leverage points identify potentially influential observations.

B. Applications and limitations

Linear regression is valuable when interpretability and continuous prediction matter, but its assumptions constrain its reliability.

• Applications: Typical uses include forecasting demand, estimating costs, modelling physical measurements, and quantifying feature–target associations.
• Advantages: Training is efficient, coefficients are interpretable, and the model provides a strong baseline.
• Limitations: Unmodelled curvature, influential outliers, omitted variables, and extrapolation beyond the observed feature range can produce misleading predictions.
• Causal caution: A coefficient represents an adjusted association, not automatically a causal effect; causal interpretation requires an appropriate design and assumptions about confounding.

IV. Logistic Regression — Estimating Class Probabilities

Logistic regression is a classification method that models a linear relationship between predictors and the log-odds of an outcome.

A. Machine logistic regression

Machine logistic regression transforms a linear score into a probability between zero and one.

• Sigmoid function:

p = 1 / (1 + e^(−z))
z = β₀ + β₁x₁ + ... + βₚxₚ

• (p): estimated probability that the class label equals (1).
• (e): base of the natural logarithm.
• (z): linear decision score.
• (x_j): feature (j).
• (\beta_0,\beta_j): learned intercept and coefficients.

• Log-odds form:

log(p / (1 − p)) = β₀ + β₁x₁ + ... + βₚxₚ

• (p/(1-p)): odds of the positive class.
• (\log): natural logarithm.

• Coefficient meaning: Increasing (x_j) by one unit multiplies the odds by (e^{\beta_j}), holding other predictors constant.
• Training loss: Binary cross-entropy penalizes confident incorrect probabilities.

Log loss = −(1/n)Σᵢ[yᵢlog(pᵢ) + (1−yᵢ)log(1−pᵢ)]

• (n): number of observations.
• (y_i): actual binary label.
• (p_i): predicted positive-class probability.

• Classification threshold: A threshold such as (0.5) converts probabilities into labels, but it should be adjusted when false positives and false negatives have unequal costs.
• Extensions: Multinomial logistic regression handles unordered classes, while one-versus-rest fits one binary classifier per class.
• Limitations: Logistic regression assumes linearity in the log-odds and may underfit complex boundaries unless interactions, transformations, or nonlinear features are added.

V. Controlling Complexity — Penalized Model Fitting

Regularization modifies the training objective to discourage excessively large coefficients and reduce overfitting.

A. Regularization

Regularization trades a small increase in training error for potentially better performance on unseen data.

1. Ridge regularization:
   • Objective: Adds the squared magnitude of coefficients to the loss.

Objective = RSS + λΣⱼβⱼ²

• Effect: Shrinks correlated-feature coefficients toward zero but usually does not remove them.
  2. Lasso regularization:
• Objective: Adds the absolute magnitude of coefficients.

Objective = RSS + λΣⱼ|βⱼ|

• Effect: Can set coefficients exactly to zero, creating a sparse model.

• Penalty strength: (\lambda) is a non-negative hyperparameter; (\lambda=0) gives the unregularized model, while larger values impose stronger shrinkage.
• Elastic Net: A weighted combination of squared and absolute penalties is useful when predictors are numerous and correlated.
• Feature scaling: Standardization is essential because otherwise a feature’s measurement scale changes how strongly its coefficient is penalized.
• Model selection: Cross-validation chooses (\lambda) using validation performance rather than test data.
• Intercept convention: The intercept is commonly excluded from the penalty because it controls the baseline rather than feature complexity.

VI. Model Evaluation — Measuring Predictive Quality

Evaluation metrics must match the target type, decision costs, and purpose of the model.

A. Performance metrics

Performance metrics quantify error or discrimination on data not used to fit the model.

• Mean absolute error: For continuous targets, (MAE=(1/n)\Sigma_i|y_i-\hat y_i|); it is expressed in the target’s units and is less sensitive to large errors than squared metrics.
• Mean squared error: (MSE=(1/n)\Sigma_i(y_i-\hat y_i)^2); squaring makes large errors especially influential.
• Root mean squared error: (RMSE=\sqrt{MSE}); it restores the target’s units while retaining a strong penalty for large residuals.
• Coefficient of determination: (R^2=1-RSS/TSS), where (TSS=\Sigma_i(y_i-\bar y)^2) and (\bar y) is the observed target mean. A test-set value below zero means the model is worse than predicting that mean.
• Confusion-matrix counts: True positives and true negatives are correct predictions; false positives and false negatives are the two error types.
• Classification measures:
  • Accuracy: Correct predictions divided by all predictions.
  • Precision: (TP/(TP+FP)), measuring the reliability of positive predictions.
  • Recall: (TP/(TP+FN)), measuring how many actual positives are detected.
  • F1 score: Harmonic mean of precision and recall.
• Threshold-independent measures: ROC-AUC ranks positives against negatives across thresholds; precision–recall AUC is often more informative for rare positive classes.
• Probability quality: Log loss evaluates probabilistic confidence, while calibration checks whether outcomes predicted at probability (0.7) occur about 70% of the time.
• Evaluation protocol: Fit on training data, tune with validation or cross-validation, and use the test set once for an unbiased final estimate.