Unit 4: Autoencoders and Pre-trained CNN

I. Foundations and Governing Principle

An autoencoder is an artificial neural network that learns to reproduce its input at the output through a constrained internal representation. Introduced in early neural-network research and later strengthened by deep learning, it performs self-supervised representation learning because the input itself supplies the reconstruction target.

• Governing principle: Learn parameters that minimize the difference between an input (x) and its reconstruction (\hat{x}).
• Encoder–decoder structure: The encoder maps data into a latent code; the decoder maps that code back into the original data space.
• Information bottleneck: A restricted latent representation forces the network to retain important features rather than simply copy every input value.
• Training convention: For a dataset ({x^{(i)}}_{i=1}^{N}), both the network input and target are usually (x^{(i)}).
• Learned representation: Unlike manually designed compression rules, the latent features are discovered from training data.
• CNN connection: Autoencoders for images often use convolutional layers, while pre-trained CNNs reuse visual features learned previously on large datasets.
• Central distinction: Ordinary autoencoders learn deterministic codes; variational autoencoders learn probability distributions over latent variables.

II. Autoencoder Fundamentals — Purpose, Need, and Structure

A. Introduction to autoencoders

An autoencoder learns two functions whose composition approximates the identity function while passing data through a useful latent representation.

• Encoding function: The encoder transforms an input (x) into a latent code (z):

z = fθ(x)

Here, (x) is the input vector, (z) is the latent representation, (f) is the encoder, and (\theta) denotes encoder parameters.

• Decoding function: The decoder reconstructs the input from the code:

x̂ = gφ(z) = gφ(fθ(x))

Here, (\hat{x}) is the reconstruction, (g) is the decoder, and (\phi) denotes decoder parameters.

• Learning objective: Parameters are optimized by minimizing average reconstruction loss:

L(θ, φ) = (1/N) Σᵢ ℓ(x⁽ⁱ⁾, x̂⁽ⁱ⁾)

(N) is the number of examples and (\ell) measures reconstruction error.

• Common losses:
  • Mean squared error: Used for real-valued inputs such as normalized image intensities.

ℓMSE(x, x̂) = (1/d) Σⱼ (xⱼ − x̂ⱼ)²

• Binary cross-entropy: Appropriate when each feature is interpreted as a Bernoulli probability.

• Self-supervision: No external class label such as “cat” or “car” is required; each input provides its own target.

• Nonlinear learning: Activation functions such as ReLU allow deep autoencoders to capture relationships that linear techniques cannot. A single-layer linear autoencoder with MSE and suitable constraints learns a subspace closely related to principal component analysis.

B. Need for autoencoders

Autoencoders are needed when useful representations must be learned from abundant unlabelled data or when high-dimensional observations contain redundancy.

• Dimensionality reduction: An input (x\in\mathbb{R}^{d}) can be mapped to (z\in\mathbb{R}^{k}), where (k<d), reducing storage and computation.
• Feature learning: The code may capture edges, shapes, textures, or semantic structure without manually defining these features.
• Noise removal: A denoising autoencoder receives a corrupted input (\tilde{x}) but is trained to reconstruct clean (x), learning resistance to irrelevant disturbances.
• Anomaly detection: A model trained on normal samples generally reconstructs normal inputs well. A large error,

e(x) = ||x − x̂||²

can indicate an unusual transaction, defective product, or abnormal sensor record.

• Missing-value reconstruction: Patterns learned among observed variables can help estimate corrupted or absent entries, although reliability depends on training-data quality.
• Pretraining: Encoder weights can initialize a larger supervised network when labelled examples are scarce.
• Manifold learning: High-dimensional data may lie near a lower-dimensional manifold; the latent space provides coordinates for this structure.
• Necessary constraint: If the network has excessive capacity and no regularization, it may learn the trivial identity mapping (g(f(x))=x) without discovering meaningful features.

C. Architecture of autoencoder

The architecture consists of an encoder, a bottleneck or latent layer, and a decoder trained jointly through backpropagation.

• Encoder: Successive layers reduce or transform the input:

h¹ = σ(W¹x + b¹)
z  = σ(W²h¹ + b²)

(W¹,W²) are weight matrices, (b¹,b²) are bias vectors, (h¹) is a hidden activation, and (\sigma) is an activation function.

• Bottleneck: The code (z) contains the compressed representation. An undercomplete bottleneck has fewer dimensions than the input.
• Decoder: Decoder layers reverse the transformation:

h² = σ(W³z + b³)
x̂  = ψ(W⁴h² + b⁴)

The output activation (\psi) depends on the data: sigmoid for values in ([0,1]), linear activation for unrestricted continuous values.

• Image architecture: Convolutional autoencoders use convolution and downsampling in the encoder, followed by upsampling or transposed convolution in the decoder.
• Symmetry: The decoder often mirrors the encoder, but exact symmetry and tied weights are not mandatory.
• Training process:
  1. Pass a mini-batch through the encoder and decoder.
  2. Calculate reconstruction and regularization losses.
  3. Backpropagate gradients through both components.
  4. Update parameters using Adam, stochastic gradient descent, or a related optimizer.
• Capacity control: Bottleneck size, layer width, weight decay, sparsity penalties, dropout, and noise determine how much information the model can preserve.

III. Autoencoder Variants — Constraints and Learning Behaviour

A. Types of autoencoders

Different autoencoder types impose different constraints so that the latent representation captures useful structure rather than memorizing inputs.

• Undercomplete autoencoder: Uses (k<d), forcing compression through a smaller code. It is effective when essential information occupies fewer dimensions than the raw input.
• Sparse autoencoder: May use a wide hidden layer but penalizes frequent neuron activation:

Ltotal = Lreconstruction + λ Σⱼ |zⱼ|

Here, (\lambda) controls sparsity strength and (|z_j|) penalizes active latent units.

• Denoising autoencoder: Learns the mapping (\tilde{x}\rightarrow x), where (\tilde{x}) is produced by masking features or adding Gaussian noise. It must infer stable structure instead of copying corrupted values.
• Contractive autoencoder: Adds a penalty on the encoder Jacobian:

Ltotal = Lreconstruction + λ ||∂fθ(x)/∂x||²F

The Frobenius norm penalty encourages small code changes for small input perturbations.

• Convolutional autoencoder: Preserves spatial locality through shared convolutional filters, making it suitable for images and spatial signals.
• Stacked autoencoder: Contains multiple encoding and decoding layers, enabling hierarchical feature extraction. Earlier layers may detect simple patterns while deeper layers combine them.
• Sequence autoencoder: Uses recurrent units, one-dimensional convolutions, or transformers to encode variable-length text, audio, or time-series sequences.
• Adversarial autoencoder: Uses an adversarial discriminator to make encoded samples follow a chosen prior distribution.
• Comparison:
  1. Deterministic variants generate one code (z=f(x)) for each input.
  2. Probabilistic variants, especially VAEs, model uncertainty and support principled sampling.

IV. Compression — Encoding Data into Compact Representations

A. Data compression using autoencoders

Autoencoder compression stores or transmits the latent code instead of the original high-dimensional input and reconstructs an approximation when needed.

• Compression pipeline:

Original data x → Encoder → Latent code z
Latent code z   → Decoder → Reconstruction x̂

• Compression ratio: If an input has (d) stored values and its code has (k), a simple element-count ratio is:

Compression ratio = d/k

For a (28\times28) grayscale image, (d=784). A code with (k=32) gives (784/32=24.5), before accounting for numeric precision and model storage.

• Lossy nature: Because (k<d), exact reconstruction is generally impossible; the decoder preserves statistically important features while discarding detail.
• Quality measures: MSE and peak signal-to-noise ratio assess pixel accuracy, while structural similarity measures perceptual similarity.
• Learned advantage: The compressor adapts to a domain. A model trained on faces can exploit regularities such as repeated eye, nose, and contour structures.
• Practical limitation: Both sender and receiver need the trained decoder, so model size and inference cost must be included when evaluating compression efficiency.
• Generalization risk: Inputs unlike the training distribution may reconstruct poorly, making learned compression less predictable than standardized codecs.

V. Variational Autoencoders — Probabilistic Latent-Variable Models

A. Variational autoencoders

A variational autoencoder learns a continuous probability distribution in latent space, enabling reconstruction, interpolation, and generation of new samples.

• Probabilistic encoder: Instead of outputting one code, the encoder estimates parameters of an approximate posterior:

qφ(z|x) = N(z; μ(x), diag(σ²(x)))

Here, (\mu(x)) is the latent mean, (\sigma^2(x)) is the latent variance, and (q_\phi(z|x)) approximates the unknown posterior.

• Prior distribution: The latent variable usually follows:

p(z) = N(0, I)

(I) is the identity covariance matrix.

• Reparameterization trick: Differentiable sampling is performed as:

ε ~ N(0, I)
z = μ + σ ⊙ ε

(\epsilon) is random noise and (\odot) denotes element-wise multiplication.

• VAE objective: Training minimizes negative evidence lower bound:

LVAE = −Eqφ(z|x)[log pθ(x|z)] + DKL(qφ(z|x) || p(z))

The first term is reconstruction loss; (D_{KL}) is Kullback–Leibler divergence, which regularizes the encoded distribution toward the prior.

• Generation: A new code (z\sim N(0,I)) is sampled and passed through the decoder to produce a new observation.
• Continuous latent space: Nearby codes generally decode into similar outputs, supporting interpolation between learned examples.
• Limitation: Strong KL regularization can reduce detail or cause posterior collapse, where the decoder ignores (z). VAEs therefore often produce smoother but less sharp images than adversarial generative models.

VI. Pre-trained CNN — Transfer of Learned Visual Features

A. Definition and mechanism

A pre-trained convolutional neural network is a CNN whose parameters were learned previously on a large dataset and are reused for a related task.

• Transfer learning: Early convolutional layers commonly detect transferable patterns such as edges, corners, color transitions, and textures.
• Feature extraction: The convolutional base is frozen, and only a new task-specific output layer is trained.
• Fine-tuning: Selected upper layers are unfrozen and updated using a small learning rate after the new classifier has stabilized.
• Typical adaptation:

Input image → Pre-trained convolutional base
            → Global average pooling
            → New dense output layer

• Relationship to autoencoders: An encoder trained through reconstruction can supply reusable features, while a conventional pre-trained CNN usually learns from labelled classification data.
• Benefits: Pretraining reduces training time, lowers labelled-data requirements, and often improves generalization.
• Limitations: Negative transfer may occur when the source and target domains differ substantially; fine-tuning can also overfit small datasets or catastrophically alter useful pretrained features.