Unit 5: Dimensionality Reduction and Neural Networks

I. Orientation

Predictive analytics converts observed variables into useful predictions by identifying structure, reducing irrelevant variation, and learning relationships between inputs and outputs. Dimensionality reduction simplifies high-dimensional data, while neural networks learn flexible nonlinear mappings from features to predictions. Both areas depend on careful preprocessing, appropriate model complexity, and evaluation on data not used for training.

• Governing principle: Preserve predictive information while reducing noise, redundancy, computational cost, and overfitting.
• Data representation: A dataset with (n) observations and (p) variables is commonly represented as a matrix (X \in \mathbb{R}^{n \times p}).
• Feature convention: Each row generally represents one observation; each column represents one feature or measured variable.
• Scaling assumption: Distance-based and variance-based methods are sensitive to measurement units, so numerical features often require standardization.
• Learning distinction: Unsupervised methods, such as PCA, learn structure without a target variable; supervised neural networks learn using a target (y).
• Generalization requirement: A model should perform well on unseen data, not merely memorize the training set.
• Complexity trade-off: More dimensions and more network parameters can represent richer patterns but can also increase variance and overfitting.

II. Dimensionality Reduction — Simplifying High-Dimensional Data

A. Orientation

Dimensionality reduction transforms data from a large number of original variables into a smaller set of informative representations. The transformation may retain the most important variation, preserve distances, or select a subset of original variables.

B. Dimensionality Reduction

Dimensionality reduction reduces the number of input dimensions while attempting to preserve information relevant to analysis or prediction.

• Purpose: Replace (p) original features with (k) components or selected features, where (k < p); for example, 100 sensor readings may be represented by 10 derived variables.
• Redundancy removal: Highly correlated variables often carry overlapping information; replacing temperature readings from several nearby sensors can reduce duplication.
• Noise control: Low-variance or unstable directions may contain measurement noise rather than useful signal.
• Computational benefit: Algorithms operating on (k=10) dimensions generally require less memory and fewer calculations than those operating on (p=1{,}000).
• Visualization: Data can be projected to two or three dimensions for plotting, although visual interpretability does not guarantee predictive usefulness.
• Feature selection versus extraction:
  • Selection: Retains original variables, such as choosing age, income, and account balance.
  • Extraction: Creates new variables, such as principal components formed from weighted combinations of all original features.
• Supervised versus unsupervised reduction: Supervised techniques use the target to preserve predictive separation; PCA is unsupervised and does not know which directions predict (y).
• Important limitation: A direction with large variance is not necessarily the direction with the strongest relationship to the target. Dimensionality reduction should therefore be validated against the prediction objective.
• Data leakage control: Fit the transformation using training data only, then apply the learned transformation to validation and test data.

III. Principal Component Analysis (PCA) — Orthogonal Variance-Preserving Projection

A. Orientation

Principal Component Analysis is an unsupervised linear transformation that creates orthogonal directions, called principal components, ordered by the amount of variance they explain. PCA is usually applied after centering and, when units differ substantially, standardizing the features.

B. Principal Component Analysis (PCA)

PCA finds directions in feature space that maximize projected variance, with each new direction constrained to be orthogonal to the preceding directions.

• Standardization: For feature (j), a standardized value is commonly computed as
  [
  z{ij}=\frac{x{ij}-\mu_j}{sj}
  ]
  where (x{ij}) is observation (i)'s value, (\mu_j) is the training mean, and (s_j) is the training standard deviation.
• Covariance structure: After centering, PCA analyzes the covariance matrix
  [
  S=\frac{1}{n-1}X_c^\top X_c
  ]
  where (X_c) is the centered data matrix and (n) is the number of observations.
• Eigenvalue problem: Principal directions satisfy
  [
  Sv_r=\lambda_r v_r
  ]
  where (v_r) is the (r)-th eigenvector and (\lambda_r) is its eigenvalue, representing variance along that direction.
• Component scores: Observations are projected using
  [
  Z=X_cV_k
  ]
  where (V_k=[v_1,\ldots,v_k]) contains the first (k) eigenvectors and (Z) contains the reduced coordinates.
• Ordering convention: Components are ordered so that (\lambda_1 \geq \lambda_2 \geq \cdots \geq \lambda_p); PC1 explains the greatest variance and PC2 explains the greatest remaining variance subject to orthogonality.
• Explained variance ratio: For component (r),
  [
  \text{EVR}_r=\frac{\lambdar}{\sum{j=1}^{p}\lambda_j}
  ]
  and cumulative explained variance is the sum of the first (k) ratios.
• Worked example: If eigenvalues are (6,3,1), total variance is (10). PC1 explains (60\%), PC2 explains (30\%), and the first two components retain (90\%) of total variance.
• Interpretation caution: A component is a weighted combination of original features. Large positive or negative loadings indicate contribution to that component, but the sign itself is arbitrary.
• Strengths: PCA reduces correlated dimensions, supports visualization, compresses data, and can improve model stability when predictors are numerous and redundant.
• Limitations: PCA is linear, sensitive to scaling and outliers, and may discard low-variance information that is highly predictive. Components can also be difficult to interpret because they combine many original variables.
• Pipeline rule: Compute means, standard deviations, and eigenvectors on the training partition only; otherwise information from the test set can influence the model.

IV. Feedforward Neural Networks — Layered Function Approximation

A. Orientation

A feedforward neural network maps inputs to outputs through directed layers, with information moving only forward from the input layer to hidden layers and then to the output layer. It learns weights and biases by minimizing a loss function.

B. Feedforward Neural Networks

A feedforward neural network represents a function by composing affine transformations with nonlinear activation functions.

• Input representation: For one observation, the feature vector is (x \in \mathbb{R}^{p}); numerical inputs are commonly scaled so optimization is not dominated by large measurement units.
• Neuron computation: A neuron first calculates
  [
  z=w^\top x+b
  ]
  then produces (a=\phi(z)), where (w) is a weight vector, (b) is a bias, and (\phi) is an activation function.
• Layer transformation: A layer can be written as
  [
  a^{(l)}=\phi^{(l)}\left(W^{(l)}a^{(l-1)}+b^{(l)}\right)
  ]
  where (W^{(l)}) and (b^{(l)}) are the weights and biases of layer (l), and (a^{(0)}=x).
• Forward-only structure: There are no cycles or recurrent connections; the output depends on the current input and learned parameters.
• Activation functions:
  • ReLU: (\phi(z)=\max(0,z)), efficient and widely used in hidden layers.
  • Sigmoid: (\phi(z)=1/(1+e^{-z})), useful for a binary probability output.
  • Softmax: For class (c), (P(y=c\mid x)=e^{z_c}/\sum_j e^{z_j}), producing probabilities that sum to one.
• Output design: Regression commonly uses one linear output; binary classification uses one sigmoid output; multiclass classification uses a softmax output with one unit per class.
• Loss functions:
  • Mean squared error: (\text{MSE}=\frac{1}{n}\sum_i(y_i-\hat y_i)^2) for continuous targets.
  • Binary cross-entropy: (-\frac{1}{n}\sum_i[y_i\log(\hat p_i)+(1-y_i)\log(1-\hat p_i)]) for binary labels.
• Parameter learning: Training adjusts weights to reduce loss, usually through gradient descent:
  [
  \theta_{t+1}=\thetat-\eta\nabla\theta L(\theta_t)
  ]
  where (\theta) denotes all parameters, (\eta) is the learning rate, and (L) is the loss.
• Backpropagation: The chain rule computes how each parameter contributes to the loss, allowing gradients to move backward through the network during training.
• Generalization controls: Validation monitoring, early stopping, dropout, weight decay, and suitable network size reduce overfitting.
• Limitations: Neural networks require tuning, can be computationally expensive, may be hard to interpret, and can be sensitive to feature scaling, initialization, and class imbalance.

V. Multi-layer Perceptron (MLP) — A Fully Connected Neural Architecture

A. Orientation

A Multi-layer Perceptron is a feedforward neural network containing an input layer, one or more fully connected hidden layers, and an output layer. With nonlinear hidden activations, an MLP can approximate complex nonlinear relationships.

B. Multi-layer Perceptron (MLP)

An MLP applies successive dense transformations, enabling combinations of features that cannot be represented by a single linear model.

• Architecture: An MLP with (p) inputs, hidden widths (h_1,\ldots,h_L), and (q) outputs has parameter matrices sized according to adjacent layers; the first weight matrix is (h_1 \times p).
• Single-hidden-layer form: A regression MLP can be expressed as
  [
  \hat y=W_2\phi(W_1x+b_1)+b_2
  ]
  where (W_1,b_1) define the hidden layer and (W_2,b_2) define the output layer.
• Nonlinearity requirement: If (\phi) is absent or linear in every layer, multiple layers collapse into one linear transformation. Nonlinear activations give the MLP its expressive power.
• Universal approximation: Under suitable conditions, an MLP with at least one hidden layer can approximate broad classes of continuous functions, but this does not guarantee efficient learning or good generalization.
• Training cycle: A typical iteration performs forward propagation, loss calculation, backpropagation, and parameter update. A batch is the subset of observations used for one update; an epoch is one complete pass through the training data.
• Worked example: With three inputs, one hidden layer of four ReLU neurons, and one linear output, (W_1) has shape (4\times3), (b_1) has four values, (W_2) has shape (1\times4), and (b_2) is one value. The network has (12+4+4+1=21) trainable parameters.
• Hyperparameters: Hidden-layer count, neurons per layer, activation, learning rate, batch size, epochs, regularization strength, and initialization affect training and predictive performance.
• Evaluation discipline: Select architecture and hyperparameters using validation data or cross-validation; reserve the test set for final performance estimation.
• Interpretability: Feature importance can be explored with permutation importance, partial dependence, or SHAP-style methods, but these explain model behavior rather than proving causation.
• Applications and limitations: MLPs are suitable for tabular regression, classification, forecasting inputs, and nonlinear pattern detection. They are less attractive when datasets are very small, explanations are mandatory, or simpler models achieve comparable accuracy.