Unit 5: Dimensionality Reduction and Neural Networks - Subjective Questions

Dimensionality reduction is the process of reducing the number of input variables or features in a dataset while retaining as much useful information as possible.

It is important because:

• It reduces computational cost and training time.
• It helps address the curse of dimensionality.
• It can remove redundant and noisy features.
• It improves model interpretability and visualization.
• It may reduce overfitting by simplifying the model.

Dimensionality reduction can be performed using feature selection, which keeps a subset of original variables, or feature extraction, which transforms the original variables into a smaller set of new variables.

The curse of dimensionality refers to the problems that arise when the number of features in a dataset becomes very large.

Its effects include:

• Data points become increasingly sparse in the feature space.
• A much larger training dataset is required to represent the feature space adequately.
• Distance-based methods become less reliable because distances tend to become similar.
• Models require more memory and computational time.
• The likelihood of overfitting increases because the model may learn noise.

Dimensionality reduction helps by representing the data with fewer meaningful variables while preserving important patterns.

Feature selection chooses a subset of the original variables, whereas feature extraction creates new variables from combinations or transformations of the original variables.

Feature selection is generally easier to interpret, while feature extraction can achieve stronger dimensional compression.

Principal Component Analysis (PCA) is an unsupervised dimensionality reduction technique that transforms correlated variables into a smaller set of uncorrelated variables called principal components.

Its main objective is to find directions in the feature space that maximize variance. The first principal component captures the greatest possible variance, the second captures the greatest remaining variance subject to being orthogonal to the first, and so on.

PCA is commonly used to:

• Compress high-dimensional data.
• Remove correlation among variables.
• Visualize data in two or three dimensions.
• Reduce noise and improve computational efficiency.

The main steps of PCA are:

1. Standardize the data: Center each feature by subtracting its mean and, when necessary, divide by its standard deviation.
2. Construct the covariance or correlation matrix: This identifies relationships among features.
3. Calculate eigenvalues and eigenvectors: Eigenvectors represent principal directions, while eigenvalues represent the variance captured by those directions.
4. Rank the components: Sort eigenvectors in descending order of their corresponding eigenvalues.
5. Select the required components: Choose the first components based on explained variance.
6. Project the data: Transform the original observations into the selected principal-component space using .

Let be a centered data matrix with observations and features. The covariance matrix is

PCA seeks a unit direction vector such that the projected data has maximum variance. The variance in this direction is

subject to the constraint .

Using a Lagrange multiplier, maximize

Differentiating and setting the derivative to zero gives

Thus, must be an eigenvector of the covariance matrix, and the corresponding eigenvalue equals the variance captured in that direction. The eigenvector with the largest eigenvalue is the first principal component, and subsequent components are selected in descending eigenvalue order.

The number of principal components is selected by balancing information retention, model complexity, and interpretability.

Common methods include:

• Explained variance ratio: Select the smallest number of components whose cumulative explained variance reaches a target such as or .
• Scree plot: Plot eigenvalues and identify the elbow point where additional components provide little benefit.
• Kaiser criterion: Retain components with eigenvalues greater than when PCA is applied to standardized variables.
• Cross-validation: Select the number that produces the best predictive performance in a downstream model.

Using too few components may discard useful information, while using too many reduces the benefit of dimensionality reduction.

Standardization is important because PCA is variance-based. If features are measured on different scales, variables with larger numerical units may dominate the principal components.

For a feature , the standardized value is

where is the feature mean and is its standard deviation.

Standardization:

• Gives features comparable influence.
• Prevents units such as income or population from dominating smaller-scale variables.
• Makes PCA based on the correlation matrix equivalent to PCA on standardized data.

However, if all variables are measured in comparable units and their original variance is meaningful, standardization may not always be necessary.

PCA and feature selection reduce dimensionality in different ways.

• Interpretability: Feature selection retains original variables, so the selected features remain easy to explain. PCA produces combinations such as , which may be harder to interpret.
• Information retention: Feature selection may discard variables that individually appear weak but jointly contain useful information. PCA combines variables and can retain a high proportion of total variance.
• Redundancy: PCA removes linear correlation by creating orthogonal components. Feature selection may leave correlated features.
• Practical use: Feature selection is preferable when domain interpretation is essential, while PCA is useful for compression, visualization, and handling correlated variables.

A feedforward neural network is a neural model in which information moves in one direction, from the input layer through one or more hidden layers to the output layer. It contains no feedback connections or cycles.

Its structure includes:

• Input layer: Receives feature values.
• Hidden layers: Apply weighted transformations and activation functions.
• Output layer: Produces predictions.

For a neuron, the weighted input is

and its output is

where are weights, is a bias, and is an activation function. Each layer passes its activations to the next layer until the final prediction is produced.

Weights determine the influence of input variables on a neuron. A large positive weight increases the neuron's activation, while a negative weight decreases it.

Biases shift the activation threshold and allow a neuron to produce useful outputs even when all input values are zero.

Activation functions introduce nonlinearity. Without them, multiple neural-network layers would be equivalent to a single linear transformation.

Examples include:

• ReLU:
• Sigmoid:
• Hyperbolic tangent:

Together, weights and biases define the transformation, while activation functions enable the network to learn complex relationships.

A Multi-layer Perceptron (MLP) is a fully connected feedforward neural network containing an input layer, one or more hidden layers, and an output layer.

A single-layer perceptron directly maps inputs to outputs and can learn only linearly separable relationships. An MLP includes hidden layers and nonlinear activation functions, allowing it to model nonlinear decision boundaries.

Key differences:

• A single-layer perceptron has no hidden layer; an MLP has at least one.
• A single-layer perceptron solves only linear classification problems; an MLP can solve nonlinear problems.
• MLPs have greater representational capacity but require more computation and careful training.

In forward propagation, the input is passed sequentially through each layer.

For input vector , the first hidden layer computes

The second hidden layer computes

The output layer computes

Here, and are the weights and biases of layer , is a hidden-layer activation function, and is selected according to the task. For example, may be sigmoid for binary classification, softmax for multiclass classification, or the identity function for regression.

Backpropagation calculates how much each weight contributes to the prediction error and uses this information to update the weights.

The process is:

1. Initialize weights and biases, usually with small random values.
2. Perform forward propagation to calculate the prediction .
3. Compute a loss function such as mean squared error or cross-entropy.
4. Apply the chain rule to calculate gradients of the loss with respect to each parameter.
5. Propagate the error backward from the output layer toward the input layer.
6. Update parameters using gradient descent:

where is the learning rate and is the loss.

This process is repeated for multiple batches and epochs until the loss decreases sufficiently.

Activation functions determine the output of a neuron and introduce nonlinearity.

• Sigmoid: . It produces values between and and is useful for binary output probabilities. It can suffer from vanishing gradients.
• Tanh: . It produces values between and and is zero-centered, but it can also have vanishing gradients.
• ReLU: . It is computationally simple and usually trains efficiently, but neurons may become permanently inactive for negative inputs.
• Softmax: Converts output scores into multiclass probabilities whose sum is .

ReLU is common in hidden layers, while sigmoid and softmax are commonly used in classification output layers.

During backpropagation, gradients are repeatedly multiplied across layers. This can create two problems:

• Vanishing gradients: Gradients become extremely small as they move toward earlier layers. The weights of those layers update very slowly, making learning difficult. This often occurs with sigmoid or tanh activations.
• Exploding gradients: Gradients become extremely large, causing unstable updates, numerical overflow, or divergence.

Possible solutions include:

• Using ReLU-family activation functions.
• Applying suitable weight initialization.
• Using batch normalization.
• Applying gradient clipping to limit large gradients.
• Using optimizers such as Adam.
• Reducing network depth when appropriate.

An MLP overfits when it learns the training data too closely, including noise, and performs poorly on unseen data.

Regularization methods include:

• L1 regularization: Adds to the loss and can encourage some weights to become zero.
• L2 regularization: Adds and discourages excessively large weights.
• Dropout: Randomly deactivates a proportion of neurons during training, reducing dependence on particular paths.
• Early stopping: Stops training when validation loss begins to increase.
• Data augmentation: Creates additional varied training examples where appropriate.
• Reducing network size: Limits the number of parameters and model capacity.

These methods encourage the network to learn general patterns instead of memorizing training examples.

MLP architecture should be selected according to the data, task, and available computational resources.

Important decisions include:

• Input size: Usually equals the number of input features after preprocessing or dimensionality reduction.
• Output size: One output may be used for regression or binary classification, while multiple outputs may be used for multiclass classification.
• Number of hidden layers: More layers can represent more complex functions but increase training difficulty.
• Neurons per layer: More neurons increase capacity but may cause overfitting.
• Activation functions: Nonlinear functions are generally used in hidden layers.
• Regularization: Dropout, weight penalties, and early stopping help control complexity.

The final architecture should be selected using validation performance rather than training performance alone.

The output layer and loss function must match the prediction task.

• Regression: Use a linear output activation and commonly use mean squared error:

• Binary classification: Use a sigmoid output that produces a probability between and , together with binary cross-entropy:

• Multiclass classification: Use a softmax output and categorical cross-entropy. Softmax converts logits into probabilities:

The selected loss guides the network to optimize an appropriate measure of prediction error.

Learning rate controls the size of each parameter update. A rate that is too large may cause divergence, while one that is too small may make training very slow.

Batch size is the number of training examples used to estimate a gradient update. Small batches require less memory and introduce useful gradient variation, while large batches provide more stable estimates but may require more memory.

An epoch is one complete pass through the training dataset. Too few epochs may cause underfitting, while too many may cause overfitting.

These hyperparameters are usually selected using validation data, learning-rate schedules, or systematic search methods.