Unit 2: SUPERVISED LEARNING: REGRESSION

I. Foundations of Supervised Regression

Supervised learning uses labelled observations to learn a mapping from input variables to a known target variable. In regression, the target is generally numerical, while logistic regression extends regression-based modelling to categorical outcomes by estimating class probabilities.

• Training data: Each observation contains predictors (x_1,x_2,\ldots,x_p) and an observed target (y).
• Model function: A model approximates the relationship as (y=f(X)+\varepsilon), where (f(X)) is the systematic component and (\varepsilon) represents unexplained variation.
• Prediction: The fitted model produces (\hat y), an estimate of (y), for existing or unseen predictor values.
• Parameters: Coefficients such as (\beta_0,\beta_1,\ldots,\beta_p) determine how predictors affect the model output.
• Fitting principle: Training selects parameter values that minimize a loss function, such as the sum of squared residuals.
• Residual: For observation (i), the residual is (e_i=y_i-\hat y_i); it measures observed prediction error.
• Generalization: Performance must be evaluated on validation or test data because low training error can conceal overfitting.
• Core assumptions: Classical linear regression commonly assumes linearity, independent errors, constant error variance, and errors with mean zero; approximate normality is additionally useful for statistical inference.

II. Simple Linear Regression

A. Simple Linear Regression

Simple linear regression models a continuous target as a straight-line function of one predictor.

y_i = β₀ + β₁x_i + ε_i
ŷ_i = b₀ + b₁x_i

• Symbols: (x_i) is the predictor, (y_i) is the observed response, (\varepsilon_i) is random error, and (\hat y_i) is the prediction.
• Parameters: (\beta_0) and (\beta_1) are population parameters; (b_0) and (b_1) are their sample estimates.
• Intercept: (b_0) is the predicted response when (x=0), although this interpretation may be meaningless when zero lies outside the observed range.
• Slope: (b_1) is the predicted change in (y) for a one-unit increase in (x). A slope of (3.2) means that (\hat y) rises by (3.2) units per unit of (x).
• Conditions:
  • Linearity: The conditional mean (E(Y\mid X=x)) follows a straight line.
  • Homoscedasticity: (\operatorname{Var}(\varepsilon_i\mid x_i)=\sigma^2) remains constant.
  • Independence: One observation’s error does not determine another’s error.
• Limitation: A fitted association does not by itself establish causation, and extrapolation beyond observed (x)-values can be unreliable.

III. Multiple Linear Regression

A. Multiple Linear Regression

Multiple linear regression estimates a continuous response from two or more predictors while holding the other included predictors constant.

y_i = β₀ + β₁x_i1 + β₂x_i2 + ... + βₚx_ip + ε_i

• Symbols: (p) is the number of predictors, (x_{ij}) is predictor (j) for observation (i), and (\beta_j) is its population coefficient.
• Coefficient interpretation: (\beta_j) is the expected change in (y) for a one-unit increase in (x_j), conditional on all other included predictors remaining fixed.
• Categorical predictors: A variable with (k) unordered categories is typically represented by (k-1) indicator variables relative to a reference category.
• Interactions: A product term (x_1x_2) allows the effect of (x_1) to depend on (x_2):

ŷ = b₀ + b₁x₁ + b₂x₂ + b₃x₁x₂

• Multicollinearity: Strong correlation among predictors inflates coefficient standard errors and makes individual effects unstable, even when overall prediction remains useful.
• Diagnostics: Residual plots reveal nonlinearity and unequal variance, while variance inflation factors help identify multicollinearity.
• Limitation: Adding predictors always leaves training (R^2) unchanged or higher, so validation performance and adjusted (R^2) are more informative for model comparison.

IV. Polynomial Regression

A. Polynomial Regression

Polynomial regression represents curved relationships by including powers of a predictor while remaining linear in its coefficients.

y_i = β₀ + β₁x_i + β₂x_i² + ... + β_dx_iᵈ + ε_i

• Degree: (d) is the highest included power; (d=2) gives a quadratic curve and (d=3) gives a cubic curve.
• Linearity in parameters: Although the curve is nonlinear in (x), coefficients (\beta_0,\ldots,\beta_d) enter linearly and can therefore be estimated by ordinary least squares.
• Hierarchy: If (x^3) is included, lower-order terms (x) and (x^2) are normally retained so the polynomial has a coherent interpretation.
• Flexibility: Increasing (d) can reduce underfitting, but an unnecessarily high degree may follow random noise and produce extreme boundary predictions.
• Scaling: Centering or standardizing (x) can reduce numerical instability because powers such as (x^4) may otherwise become very large.
• Model selection: Degree should be chosen using validation data or cross-validation, not training error alone.
• Limitation: Polynomial predictions outside the observed predictor range may change rapidly, making extrapolation especially hazardous.

V. Logistic Regression

A. Logistic Regression

Logistic regression models the probability of a categorical outcome, most commonly a binary target (Y\in{0,1}).

p(x) = P(Y=1 | X=x)
log(p(x)/(1-p(x))) = β₀ + β₁x₁ + ... + βₚxₚ
p(x) = 1/(1 + e^(-z))

• Symbols: (p(x)) is the probability of class (1), (z=\beta0+\sum{j=1}^{p}\beta_jx_j), and (e) is Euler’s number.
• Log-odds: The logit (\log[p/(1-p)]) can take any real value, while the transformed probability remains between (0) and (1).
• Coefficient interpretation: Increasing (x_j) by one unit multiplies the odds by (e^{\beta_j}), holding other predictors constant.
• Classification rule: A threshold such as (0.5) converts probabilities into labels, but the threshold should reflect error costs and class prevalence.
• Estimation: Parameters are generally fitted by maximum likelihood, equivalently by minimizing binary cross-entropy, rather than ordinary least squares.
• Assumptions: Observations should be independent, predictors should not exhibit severe multicollinearity, and continuous predictors should relate approximately linearly to the log-odds.
• Extension: Multinomial logistic regression handles more than two unordered classes.

VI. Ordinary Least Squares Estimation

A. Ordinary Least Squares Estimation

Ordinary least squares estimation selects regression coefficients that minimize the sum of squared residuals.

SSE = Σ(y_i - ŷ_i)²
b̂ = (XᵀX)⁻¹Xᵀy

• Symbols: (SSE) is the error sum of squares, (X) is the design matrix, (y) is the response vector, (X^T) is its transpose, and (\hat b) is the coefficient vector.
• Existence condition: The closed-form inverse requires (X^TX) to be nonsingular, meaning no predictor column is an exact linear combination of others.
• Simple-regression estimates:

b₁ = Σ[(x_i-x̄)(y_i-ȳ)] / Σ(x_i-x̄)²
b₀ = ȳ - b₁x̄

• Geometric result: At the OLS solution, residuals are orthogonal to every design-matrix column; with an intercept, residuals sum to zero.
• Statistical property: Under the Gauss-Markov assumptions, OLS is the best linear unbiased estimator, meaning it has minimum variance among linear unbiased estimators.
• Sensitivity: Squaring residuals gives large errors disproportionate influence, so outliers can strongly alter the fitted line.
• Inference condition: Normally distributed errors permit exact small-sample (t)-tests and confidence intervals, although normality is not required merely to compute OLS estimates.

VII. Correlations

A. Correlations

Correlation measures the direction and strength of association between two variables but does not prove a causal relationship.

r = Σ[(x_i-x̄)(y_i-ȳ)] /
    √(Σ(x_i-x̄)² Σ(y_i-ȳ)²)

• Pearson correlation: (r) measures linear association and ranges from (-1) to (1).
  • (r=1) indicates perfect positive linear association.
  • (r=-1) indicates perfect negative linear association.
  • (r=0) indicates no linear association, although a nonlinear relationship may exist.
• Scale invariance: Changing measurement units does not change (r), provided the transformation has a positive scale factor.
• Regression connection: In simple linear regression with an intercept, (R^2=r^2), where (r) is the Pearson correlation between (x) and (y).
• Spearman correlation: Spearman’s rank coefficient measures monotonic association using ranks and is less dependent on linearity.
• Sensitivity: Pearson correlation can be distorted by outliers and restricted ranges.
• Predictor screening: A correlation matrix can expose redundant predictors, but low pairwise correlations do not rule out multivariable collinearity.

VIII. Mean Absolute Error

A. Mean Absolute Error (MAE)

Mean absolute error reports the average absolute distance between observed and predicted values.

MAE = (1/n) Σ|y_i - ŷ_i|

• Symbols: (n) is the number of observations, (y_i) is the actual value, and (\hat y_i) is the predicted value.
• Units: MAE has the same unit as the response; an MAE of (4.5) °C means predictions miss actual temperatures by (4.5) °C on average in absolute terms.
• Symmetry: Overpredictions and underpredictions of equal magnitude receive equal penalties.
• Robustness: MAE is less sensitive to extreme residuals than MSE because errors are not squared.
• Optimization: Minimizing expected absolute error targets the conditional median of (Y), whereas squared error targets the conditional mean.
• Limitation: MAE does not indicate the direction of errors and is not differentiable at a residual of zero, which can complicate some optimization methods.
• Preference: MAE is useful when each unit of error has approximately constant practical cost.

IX. Mean Squared Error

A. Mean Squared Error (MSE)

Mean squared error averages squared prediction residuals and therefore emphasizes large mistakes.

MSE = (1/n) Σ(y_i - ŷ_i)²

• Penalty structure: An error of (10) contributes (100), while an error of (2) contributes (4); the larger error receives twenty-five times the penalty.
• Units: MSE is measured in squared response units, such as dollars squared, which reduces direct interpretability.
• Optimization: Its smooth, differentiable form supports gradient-based learning and leads to the OLS objective in linear regression.
• Statistical meaning: Minimizing expected squared error estimates the conditional mean (E(Y\mid X)).
• Sensitivity: Outliers can dominate MSE because their residuals are squared.
• Dataset comparison: MSE values are comparable only when targets use the same scale; rescaling (y) by a factor (c) rescales MSE by (c^2).
• Training relation: For a fixed dataset, minimizing SSE and minimizing MSE produce identical coefficients because (MSE=SSE/n).

X. R-Squared Score

A. R-squared (R²) Score

The (R^2) score measures the proportion of response variation explained by a regression model relative to predicting the response mean.

R² = 1 - SSE/SST
SST = Σ(y_i - ȳ)²

• Symbols: (SSE=\sum(y_i-\hat y_i)^2), (SST) is total sum of squares, and (\bar y) is the observed response mean.
• Interpretation: (R^2=0.80) means the model explains (80\%) of the sample variation around (\bar y); it does not mean predictions are (80\%) accurate.
• Range: On training data for OLS with an intercept, (0\leq R^2\leq1). On test data, or for models without an intercept, (R^2) can be negative.
• Negative value: (R^2<0) means the model predicts worse than the baseline (\hat y=\bar y) under squared error.
• Model complexity: Ordinary (R^2) cannot decrease when another predictor is added to a training model, even if that predictor is irrelevant.
• Adjusted form:

Adjusted R² = 1 - (1-R²)(n-1)/(n-p-1)

• Limitation: A high (R^2) does not confirm correct assumptions, causal validity, low bias, or good out-of-sample prediction.

XI. Root Mean Squared Error

A. Root Mean Squared Error (RMSE)

Root mean squared error is the square root of MSE and expresses squared-error performance in the target’s original unit.

RMSE = √[(1/n) Σ(y_i - ŷ_i)²] = √MSE

• Interpretation: An RMSE of (6) kilograms indicates a typical error magnitude on a scale of approximately (6) kilograms, with large errors weighted heavily.
• Relationship to MAE: For the same predictions, (RMSE\geq MAE); equality occurs when all absolute residuals are equal.
• Outlier response: A large gap between RMSE and MAE indicates that some errors are substantially larger than the rest.
• Model comparison: Lower RMSE indicates better squared-error performance when models are evaluated on the same observations and target scale.
• Scale dependence: RMSE cannot directly compare targets measured in different units without normalization.
• Use case: RMSE is appropriate when large errors are especially costly or when model assumptions are based on approximately normally distributed residuals.
• Limitation: Because RMSE summarizes errors into one number, it should be accompanied by residual analysis to detect bias, nonlinearity, unequal variance, and unusual observations.