Unit 4: Machine learning-1

Machine learning (ML) is the discipline of building algorithms that improve their performance on a task by learning patterns from data rather than following explicitly programmed rules. It sits inside the wider field of data analysis and simulation because a trained model is itself a compact simulation of the process that generated the data. This unit fixes the two learning paradigms defined by the presence or absence of labels, then studies logistic regression as the canonical supervised classifier.

Defining vocabulary the later sections reuse:

• Feature (x): a measured input variable; a vector x = (x₁, x₂, …, xₙ) describes one sample. Units are whatever the variable measures (kg, °C, count).
• Label / target (y): the quantity to be predicted; discrete for classification, continuous for regression.
• Training set: the collection {(x⁽ⁱ⁾, y⁽ⁱ⁾)} of m examples used to fit the model.
• Model / hypothesis (h): a parameterised function h_θ(x) mapping features to a prediction; θ are the learnable parameters.
• Loss function: a scalar measuring prediction error on one example; the cost function is its average over the training set.
• Generalisation: performance on unseen data, estimated with a held-out test set; the gap between training and test error signals overfitting (high variance) or underfitting (high bias).

II. Supervised Learning — learning from labelled examples

Supervised learning infers a mapping h_θ: x → y from a dataset where every input carries a correct output, so the algorithm has a target to imitate.

A. Definition and workflow

The learner is "supervised" by ground-truth labels that define the error it minimises.

• Objective: find θ minimising the cost J(θ) = (1/m) Σ L(h_θ(x⁽ⁱ⁾), y⁽ⁱ⁾) over the training set.
• Pipeline: collect labelled data → split into train/validation/test → choose a model → fit θ by minimising J → evaluate on the test set.
• Optimisation anchor: gradient descent updates θⱼ := θⱼ − α ∂J/∂θⱼ, where α is the learning rate; iteration continues until J converges.

B. Supervised

Supervised methods divide by the type of the target variable.

• Regression: predicts a continuous y. Example — predicting house price from area; linear regression uses h_θ(x) = θ₀ + θ₁x₁ + … + θₙxₙ and the mean-squared-error cost J(θ) = (1/2m) Σ (h_θ(x⁽ⁱ⁾) − y⁽ⁱ⁾)².
• Classification: predicts a discrete class label. Example — labelling an email spam/not-spam; the output is one of finitely many categories.
• Common algorithms: linear/logistic regression, k-nearest neighbours, decision trees, support vector machines, and naïve Bayes; all share the labelled-data requirement.
• Evaluation metrics:
  • Regression: RMSE = √((1/m) Σ (ŷ − y)²) and R² (fraction of variance explained).
  • Classification: accuracy, precision = TP/(TP+FP), recall = TP/(TP+FN), and F1 (their harmonic mean).
• Limitations: requires costly labelled data; label noise degrades the model; risk of overfitting when the model capacity exceeds the information in the data.

III. Unsupervised Learning — structure without labels

Unsupervised learning discovers structure in data that has features but no target labels, so the algorithm organises the data by intrinsic similarity rather than matching a known answer.

A. Definition and aims

With no y, there is no prediction error to minimise; the objective is instead to model the data's distribution or geometry.

• Input: an unlabelled set {x⁽ⁱ⁾} of m feature vectors.
• Goals: grouping similar samples (clustering), compressing dimensions (representation), or estimating density and detecting anomalies.
• Validation difficulty: with no ground truth, quality is judged by internal criteria (e.g. cluster compactness) rather than accuracy against labels.

B. Unsupervised

Unsupervised methods split by the kind of structure they extract.

1. Clustering: partitions samples into groups of high internal similarity.
   • k-means: minimises within-cluster variance J = Σₖ Σ_{x∈Cₖ} ‖x − μₖ‖², where μₖ is the centroid of cluster Cₖ. It alternates assigning each point to its nearest centroid and recomputing centroids until assignments stabilise.
   • Hierarchical clustering: builds a nested tree (dendrogram) by successively merging the closest pairs, needing no preset cluster count.
2. Dimensionality reduction: projects data onto fewer variables while preserving variance.
   • Principal component analysis (PCA): finds orthogonal axes (eigenvectors of the covariance matrix) capturing maximal variance, used for visualisation and noise removal.
   • Density and anomaly work: Gaussian mixture models fit soft, probabilistic clusters; points of low estimated density are flagged as outliers.
   • Applications and limitations: customer segmentation, image compression, exploratory analysis. Results are sensitive to feature scaling and the chosen distance metric, and interpreting clusters demands domain judgement.

C. Supervised versus unsupervised — the paradigm contrast

The two paradigms are defined by one variable: the presence of labels.

1. Supervised: labelled data; goal is to predict a known target; success is measured against ground truth (accuracy, RMSE); example task — credit-default prediction.
2. Unsupervised: unlabelled data; goal is to reveal hidden structure; success is judged by internal coherence; example task — grouping shoppers by behaviour.
   • Bridge: semi-supervised learning uses a small labelled set with a large unlabelled one, combining both signals when labels are scarce.

IV. Logistic Regression — probabilistic linear classification

Logistic regression is a supervised algorithm that predicts the probability of a binary outcome by passing a linear combination of features through the logistic (sigmoid) function, despite the word "regression" it is a classifier.

A. Model formulation

The purpose is to output a calibrated probability P(y = 1 | x) bounded in (0, 1).

• Linear score: z = θᵀx = θ₀ + θ₁x₁ + … + θₙxₙ.
• Sigmoid mapping: squashes z to a probability.

σ(z) = 1 / (1 + e^(−z))
h_θ(x) = σ(θᵀx) = 1 / (1 + e^(−θᵀx))

• Symbols: z = raw score (log-odds); σ = logistic function with range (0,1); e = Euler's number; h_θ(x) = estimated P(y=1|x).
• Decision rule: predict class 1 if h_θ(x) ≥ 0.5, i.e. when z ≥ 0, giving a linear decision boundary θᵀx = 0.

B. Logistic regression

This subsection sets out how parameters are learned and how the fitted model is read.

1. Cost function — why not squared error: the MSE cost is non-convex for the sigmoid, so training uses the convex log loss (binary cross-entropy).

J(θ) = −(1/m) Σ [ y⁽ⁱ⁾ log h_θ(x⁽ⁱ⁾) + (1 − y⁽ⁱ⁾) log(1 − h_θ(x⁽ⁱ⁾)) ]

• Reading it: when y = 1 the term −log h_θ(x) penalises low predicted probability; when y = 0 the term −log(1 − h_θ(x)) penalises high predicted probability. The penalty grows without bound as a confident prediction turns out wrong.
  2. Parameter update: gradient descent uses the compact gradient below, identical in form to linear regression but with the sigmoid h_θ.

θⱼ := θⱼ − α (1/m) Σ (h_θ(x⁽ⁱ⁾) − y⁽ⁱ⁾) x_j⁽ⁱ⁾

• α = learning rate; (h_θ(x⁽ⁱ⁾) − y⁽ⁱ⁾) = prediction error for example i; x_j⁽ⁱ⁾ = its j-th feature.
  • Interpreting coefficients: θⱼ is the change in the log-odds ln(p/(1−p)) per unit increase in xⱼ; hence e^{θⱼ} is the odds ratio. A θⱼ = 0.7 gives an odds ratio ≈ 2, so the odds of y=1 roughly double per unit of xⱼ.
  • Regularisation: adding (λ/2m) Σθⱼ² (L2) to J(θ) shrinks coefficients and curbs overfitting; λ controls the strength.
  • Multiclass extension: the softmax generalisation (multinomial logistic regression) handles K > 2 classes by outputting a probability per class that sums to one.

C. Worked example — pass/fail from study hours

A single feature illustrates the full prediction step.

• Setup: feature x₁ = hours studied; fitted parameters θ₀ = −4, θ₁ = 1.5; target y = 1 means "pass".
• Compute for x₁ = 3: z = −4 + 1.5·3 = 0.5, so h_θ(x) = 1/(1 + e^(−0.5)) ≈ 0.62. Since 0.62 ≥ 0.5, predict pass.
• Boundary: z = 0 at x₁ = 4 − ? → −4 + 1.5x₁ = 0 ⇒ x₁ ≈ 2.67 hours; below this the model predicts fail, above it predicts pass.
• Odds reading: θ₁ = 1.5 gives odds ratio e^{1.5} ≈ 4.5, so each extra hour multiplies the odds of passing by about 4.5.

D. Assumptions, strengths and limitations

Logistic regression trades flexibility for interpretability.

• Assumptions: a linear relationship between features and the log-odds, and roughly independent observations.
• Strengths: fast to train, outputs interpretable probabilities and odds ratios, and resists overfitting with regularisation on modest datasets.
• Limitations: the linear boundary cannot separate classes that are not linearly separable without engineered or polynomial features; performance suffers with strongly correlated features (multicollinearity) and heavy class imbalance.
• Typical uses: medical diagnosis (disease/no disease), credit scoring, and spam detection, wherever a transparent, probability-valued classifier is preferred over a black-box model.