Unit 4: Autoencoders and Pre-trained CNN - Subjective Questions

An autoencoder is an unsupervised neural network that learns a compact representation of input data and reconstructs the original input from that representation.

Its main components are:

• Encoder: Converts input into a latent representation :
  
• Latent space or bottleneck: Stores the compressed features of the input.
• Decoder: Reconstructs the input from :
  

The network is trained by minimizing a reconstruction loss such as mean squared error:

Because the desired output is the input itself, explicit class labels are not required. The bottleneck encourages the model to retain important information while discarding redundancy.

Autoencoders are needed to learn useful features from unlabeled data. They automatically discover structure in high-dimensional inputs and transform them into lower-dimensional representations.

Major applications include:

• Dimensionality reduction: Representing high-dimensional data using fewer latent variables.
• Data compression: Encoding data into compact representations.
• Denoising: Recovering clean data from corrupted inputs.
• Anomaly detection: Identifying samples with unusually high reconstruction error.
• Feature extraction: Producing latent features for classification or clustering.
• Data generation: Variational autoencoders can generate new samples.
• Missing-value reconstruction: Estimating missing portions of data from learned patterns.

Unlike fixed reduction methods, autoencoders can learn nonlinear transformations, making them useful for complex data such as images, audio, and sensor measurements.

A basic autoencoder consists of an input layer, an encoder, a bottleneck, a decoder, and an output layer.

For a single-layer encoder:

where and are encoder parameters and is an activation function.

The decoder produces the reconstruction:

where and are decoder parameters.

The architecture performs the following operations:

• The encoder reduces the input dimensions and extracts features.
• The bottleneck contains the most compact learned representation.
• The decoder expands the latent representation back to the input dimensions.
• The output is compared with input using a reconstruction loss.

In a deep autoencoder, both the encoder and decoder contain multiple hidden layers. The decoder is often approximately symmetric to the encoder.

Reconstruction loss measures the difference between the original input and reconstructed output . Training adjusts the encoder and decoder parameters to minimize this loss.

Mean squared error:

• Suitable for continuous-valued data.
• Commonly used when the decoder models a Gaussian output distribution.
• Penalizes large reconstruction errors strongly.

Binary cross-entropy:

• Suitable for binary data or values normalized to .
• Usually paired with a sigmoid output layer.
• Interprets each output value as a Bernoulli probability.

The choice of loss should match the nature and assumed probability distribution of the input data.

Undercomplete autoencoder:

• The latent dimension is smaller than the input dimension: .
• The bottleneck forces the model to compress the input.
• It learns the most informative features required for reconstruction.
• It is commonly used for dimensionality reduction and compression.

Overcomplete autoencoder:

• The latent dimension is equal to or larger than the input dimension: .
• It has enough capacity to learn a trivial identity mapping.
• Regularization is required to ensure useful feature learning.
• Sparsity constraints, noise injection, or weight penalties may be applied.

Thus, undercomplete autoencoders control learning through a small bottleneck, whereas overcomplete autoencoders depend mainly on regularization to prevent simple copying.

A sparse autoencoder encourages only a small number of hidden neurons to be active for any given input. It can have a large latent layer while still learning meaningful features.

Let be the average activation of hidden neuron , and let be a small desired activation value. Sparsity can be imposed using Kullback-Leibler divergence:

The complete objective may be written as:

Sparsity can also be introduced using an penalty on latent activations. The constraint prevents every neuron from responding to every input and promotes the learning of specialized, interpretable features.

A denoising autoencoder learns to reconstruct a clean input from a corrupted version of that input.

Its training procedure is:

1. Select a clean training sample .
2. Apply a corruption process to obtain , such as Gaussian noise or masking noise.
3. Encode the corrupted sample:
   
4. Decode the latent representation:
   
5. Compare with the original clean sample , not with .
6. Minimize using backpropagation.

Advantages:

• Prevents the network from merely learning the identity function.
• Produces features that are robust to noise.
• Learns the underlying structure of data.
• Can be used for image restoration, signal cleaning, and feature extraction.

A contractive autoencoder adds a penalty that discourages the latent representation from changing significantly when the input changes slightly.

If is the encoder output, its objective is:

where:

• is the Jacobian of the encoder.
• denotes the Frobenius norm.
• controls the strength of contraction.

A small Jacobian norm means that minor input perturbations produce only small changes in the latent code. Therefore, the model becomes insensitive to local noise while remaining sensitive to important variations along the data manifold. This leads to robust feature representations.

Basic autoencoder:

• Minimizes only reconstruction error.
• Usually uses an undercomplete bottleneck.
• May learn an identity mapping if its capacity is too high.

Sparse autoencoder:

• Adds a sparsity penalty to latent activations.
• Allows a large latent layer while activating only a few units.
• Learns specialized features.

Denoising autoencoder:

• Receives corrupted input but reconstructs the clean input.
• Learns noise-resistant representations.
• Is useful for restoration and robust feature extraction.

Contractive autoencoder:

• Penalizes the encoder's Jacobian with respect to its input.
• Makes latent codes insensitive to small input changes.
• Captures stable local structure around the data manifold.

All four use encoder-decoder structures, but they differ in the constraints used to obtain meaningful representations.

An autoencoder performs compression by mapping the original input into a lower-dimensional latent code.

Compression:

The encoder converts an input into , where .

Decompression:

The decoder reconstructs an approximation of the original data from the latent code.

The compression process has the following characteristics:

• It is generally lossy, because may not exactly equal .
• It is data-dependent, because the model learns compression from the training distribution.
• It can capture nonlinear relationships that traditional linear compression may miss.
• Its quality is evaluated using compression ratio, reconstruction loss, or domain-specific measures such as PSNR for images.

Only the latent vector and the trained decoder are required to reconstruct compressed data.

Benefits:

• Learn nonlinear compression functions from data.
• Can preserve task-relevant patterns better than generic methods.
• Work with images, audio, text representations, and sensor data.
• Allow compression rate to be controlled through latent-space size.
• Can be optimized using perceptual or domain-specific losses.

Limitations:

• Training may require a large and representative dataset.
• Encoding and decoding can require substantial computation.
• The method is generally lossy.
• Performance may degrade on data unlike the training distribution.
• The decoder must be available to recover the data.
• Latent values may require quantization and entropy coding for actual storage savings.

Therefore, a small latent vector alone does not guarantee an efficient compression system. Practical systems must also account for numerical precision, model size, transmission cost, and reconstruction quality.

Principal component analysis (PCA):

• Learns a linear projection onto orthogonal directions of maximum variance.
• Has a closed-form solution based on eigenvalue or singular-value decomposition.
• Is fast, deterministic, and comparatively easy to interpret.
• Has limited ability to model nonlinear data manifolds.

Autoencoders:

• Can learn nonlinear transformations using nonlinear activation functions.
• Are trained iteratively using gradient-based optimization.
• Can use deep architectures and specialized regularization.
• Require choices of architecture, hyperparameters, and training data.

A linear undercomplete autoencoder trained with mean squared error learns a subspace closely related to the principal subspace found by PCA, although its latent axes need not be identical to PCA's orthogonal components. Nonlinear autoencoders are more flexible but also more computationally expensive and prone to overfitting.

A variational autoencoder (VAE) is a probabilistic generative model that learns a distribution over latent variables instead of mapping each input to a single unrestricted latent code.

The encoder estimates parameters of an approximate posterior:

The decoder models the likelihood:

Key differences are:

• A conventional autoencoder maps to a deterministic code, while a VAE maps to distribution parameters and .
• A VAE regularizes its latent distribution toward a prior such as .
• A VAE can generate new samples by drawing from the prior and decoding it.
• Conventional autoencoders mainly optimize reconstruction loss, whereas VAEs optimize reconstruction and distribution-regularization terms.

The regularized latent space of a VAE is usually smoother and more suitable for sampling.

For a latent-variable model, the marginal log-likelihood is:

Because the true posterior is often intractable, a VAE introduces an approximate posterior . The log-likelihood can be decomposed as:

Since KL divergence is non-negative:

The evidence lower bound is:

• The expected log-likelihood term promotes accurate reconstruction.
• The KL-divergence term makes the approximate posterior close to the prior.

Training maximizes the ELBO, or equivalently minimizes:

This balances reconstruction quality with a smooth, sampleable latent space.

In a VAE, the encoder predicts and for the latent distribution:

Direct random sampling from this distribution interrupts ordinary backpropagation because sampling is a stochastic operation involving the learned parameters. The reparameterization trick rewrites the sample as:

Here, randomness is isolated in , while becomes a differentiable function of and .

This trick is necessary because it:

• Allows gradients to flow through the sampled latent variable.
• Enables end-to-end training with backpropagation.
• Provides a practical gradient estimator for the expected reconstruction term.
• Separates stochastic noise from the trainable encoder parameters.

In implementations, the encoder often predicts for numerical stability.

Assume the approximate posterior is:

and the prior is:

For a latent space with dimensions, the KL divergence is:

An equivalent form commonly used in code is:

The term is minimized when and for every dimension. Therefore, it encourages each approximate posterior to remain close to the standard normal prior, which creates a continuous latent space from which valid samples can be drawn.

After training, a VAE can generate data without providing an input sample.

Generation process:

1. Draw a latent vector from the prior:
   
2. Pass through the trained decoder.
3. Obtain the parameters of or a reconstructed sample .

Latent interpolation:

Given two latent vectors and , a linear interpolation is:

Decoding successive values of often produces gradual transitions between samples. This is possible because the KL-divergence term regularizes the latent codes and reduces large empty regions. However, strong regularization may reduce reconstruction sharpness, so VAE training requires a balance between latent continuity and reconstruction quality.

A pre-trained CNN is a convolutional neural network whose parameters have already been learned from a large dataset, such as ImageNet. Common examples include VGG, ResNet, Inception, and EfficientNet.

In transfer learning, knowledge learned for a source task is reused for a related target task. A typical procedure is:

• Load the pre-trained CNN and its learned weights.
• Remove or replace its original classification head.
• Add a new output layer appropriate for the target classes.
• Initially freeze some or all convolutional layers.
• Train the new classification head on the target dataset.
• Optionally unfreeze selected deeper layers and fine-tune them using a small learning rate.

Early CNN layers generally detect generic patterns such as edges and textures, while deeper layers represent more task-specific structures. Reusing these features reduces training time and data requirements.

Feature extraction:

• The convolutional base is kept frozen.
• Pre-trained activations are used as fixed features.
• Only a newly added classifier or regressor is trained.
• It is computationally efficient and suitable for small datasets.
• It reduces the risk of overfitting and catastrophic forgetting.

Fine-tuning:

• Some or all pre-trained layers are unfrozen.
• Their weights are updated using target-domain data.
• It can adapt high-level features to the new task.
• It requires more computation and careful learning-rate selection.
• It is more useful when sufficient target data exists or the target domain differs from the source domain.

A common strategy is to train the new head first and then fine-tune selected upper layers with a much smaller learning rate.

A suitable transfer-learning workflow is:

1. Prepare data: Split images into training, validation, and test sets using stratification where appropriate.
2. Match preprocessing: Resize and normalize images according to the requirements of the chosen pre-trained CNN.
3. Apply augmentation: Use realistic transformations such as flips, crops, or small rotations only on training data.
4. Load the model: Initialize a CNN with pre-trained weights and remove its original output head.
5. Add a task-specific head: Use pooling, optional dropout, and an output layer matching the number of classes.
6. Freeze the base: Train the new head while keeping convolutional weights fixed.
7. Fine-tune carefully: Unfreeze selected upper layers and continue training with a small learning rate.
8. Evaluate: Report suitable metrics such as accuracy, precision, recall, F1-score, and a confusion matrix.

Important precautions include avoiding data leakage, using the correct normalization, handling class imbalance, monitoring validation loss, applying early stopping, and ensuring that frozen batch-normalization layers behave consistently during training.