Unit 2: Convolutional Neural Networks and Training Techniques

I. Orientation — Learning Spatial Hierarchies

Convolutional neural networks (CNNs) are deep learning models designed for grid-structured data, especially images. Introduced through early systems such as the Neocognitron (1980) and LeNet-5 (1998), CNNs learn local visual patterns and combine them into increasingly abstract representations.

• Spatial structure: An image is represented as a tensor of height (H), width (W), and channels (C), commonly (H \times W \times 3) for RGB.
• Local connectivity: Each convolutional unit processes only a small spatial neighborhood rather than the complete image.
• Parameter sharing: One filter is applied at every spatial position, greatly reducing the number of trainable parameters.
• Hierarchical learning: Early layers detect edges and textures; deeper layers represent parts, objects, and semantic concepts.
• Translation response: Shifting an input shifts its convolutional feature map; pooling and augmentation can provide limited translation invariance.
• End-to-end training: Filters and classification parameters are learned jointly by minimizing a loss through backpropagation.

II. CNN Foundations — Spatial Feature Extraction

A. Convolutional neural networks

A CNN transforms an input image through convolutional, nonlinear, pooling, and task-specific output layers.

• Typical pipeline: A classifier may follow image -> convolution -> ReLU -> pooling -> convolution -> classifier.
• Learned parameters: Training adjusts filter weights and biases to minimize a loss such as cross-entropy.
• Parameter efficiency: A (3 \times 3) convolution from 32 to 64 channels uses (3 \times 3 \times 32 \times 64=18{,}432) weights, independent of image width and height.
• Common tasks: CNNs support image classification, object detection, semantic segmentation, face recognition, and medical-image analysis.

B. Convolution operation

Convolution computes a weighted local combination of input values at each spatial location.

Y[i,j,k] = b[k] + Σu Σv Σc K[u,v,c,k] X[i+u,j+v,c]

Here, (X) is the input, (K) is the kernel, (Y) is the output, (b[k]) is the bias for output channel (k), and (u,v,c) index kernel rows, columns, and input channels. Deep-learning libraries normally implement cross-correlation, omitting mathematical kernel reversal.

• Sliding computation: The kernel moves across the input and produces one output value at each valid position.
• Channel aggregation: A filter spans all input channels; a (3 \times 3) RGB filter therefore has (3 \times 3 \times 3=27) weights.
• Nonlinearity: ReLU, (f(z)=\max(0,z)), usually follows convolution so stacked layers can model nonlinear functions.

C. Filters

Filters are trainable kernels that specialize in detecting useful local patterns.

• Shape: With kernel size (K_h \times Kw), (C{in}) input channels, and (C_{out}) filters, the weight tensor is (K_h \times Kw \times C{in} \times C_{out}).
• Learned patterns: Early filters often respond to oriented edges or colors; deeper filters respond to textures and object parts.
• Depth control: The number of filters determines the output depth; 64 filters produce 64 output channels.

D. Feature maps

A feature map records how strongly a filter responds at different spatial positions.

• Activation meaning: A large value indicates that the learned pattern is present in the corresponding receptive region.
• Output tensor: Applying (C{out}) filters produces (H{out} \times W{out} \times C{out}).
• Interpretation: Spatial dimensions retain approximate location, while channels represent different learned features.

E. Padding

Padding adds values, usually zeros, around an input’s boundary to control output dimensions and preserve edge information.

Hout = floor((H + 2P - K) / S) + 1

Here, (H) is input height, (P) is padding, (K) is kernel height, and (S) is stride; the width formula is analogous.

• Valid padding: (P=0), so a (5 \times 5) input convolved by a (3 \times 3) kernel at stride 1 becomes (3 \times 3).
• Same padding: For odd (K) and stride 1, (P=(K-1)/2) preserves spatial size.
• Boundary effect: Padding allows border pixels to participate in more convolution windows.

F. Stride

Stride is the number of pixels by which a filter moves between successive positions.

• Stride one: Preserves dense spatial sampling and is common in feature-extraction layers.
• Larger stride: A stride of 2 approximately halves height and width, reducing computation and resolution.
• Trade-off: Larger strides expand effective coverage quickly but may discard fine details.

G. Pooling

Pooling summarizes local neighborhoods without learning kernel weights.

1. Max pooling: Returns the largest activation, emphasizing the strongest detected feature.
2. Average pooling: Returns the mean activation, preserving general response intensity.

• Downsampling: A (2 \times 2) pool with stride 2 changes a (28 \times 28) map to (14 \times 14).
• Benefits: Pooling lowers memory use and provides limited robustness to small translations.
• Limitation: Repeated pooling can remove precise spatial information needed for segmentation.

H. Receptive fields

A unit’s receptive field is the region of the original input capable of influencing that unit.

r_l = r_(l-1) + (k_l - 1)j_(l-1)
j_l = j_(l-1)s_l

Here, (r_l) is receptive-field size, (j_l) is the spacing between adjacent receptive-field centers, and (k_l,s_l) are layer (l)’s kernel size and stride.

• Layer stacking: Two stride-1 (3 \times 3) convolutions produce a (5 \times 5) receptive field.
• Context growth: Deeper layers integrate broader context while retaining fewer parameters than one very large kernel.
• Practical distinction: The theoretical receptive field includes all possible influencing pixels; actual influence is often concentrated near its center.

III. Landmark Architectures — Evolution of CNN Design

A. LeNet

LeNet-5 established the practical pattern of alternating convolution and subsampling for handwritten-digit recognition.

• Structure: The 1998 network processes (32 \times 32) inputs through convolution, subsampling, and fully connected layers.
• Contribution: Shared weights demonstrated efficient recognition of spatial patterns in documents.
• Limitation: Its small capacity suits simple grayscale images better than large natural-image datasets.

B. AlexNet

AlexNet demonstrated that deep CNNs trained on GPUs could dominate large-scale image classification.

• Milestone: It won the 2012 ImageNet Large Scale Visual Recognition Challenge by a substantial margin.
• Design: Five convolutional layers are followed by three fully connected layers.
• Techniques: ReLU accelerated optimization, dropout regularized dense layers, and data augmentation improved generalization.
• Impact: Its success triggered widespread adoption of deep learning in computer vision.

C. VGG

VGG networks showed that depth could be increased systematically using small, uniform convolution kernels.

• Design rule: VGG-16 uses 13 convolutional and 3 fully connected layers, primarily with (3 \times 3) kernels.
• Effective coverage: Stacking two (3 \times 3) layers gives a (5 \times 5) receptive field with extra nonlinearities.
• Strength: The regular architecture is easy to understand and useful for feature transfer.
• Limitation: VGG-16 has about 138 million parameters, making it memory- and computation-intensive.

D. ResNet

Residual networks use shortcut connections to train architectures far deeper than earlier CNNs.

y = F(x, W) + x

Here, (x) is the block input, (F) is the learned residual transformation with weights (W), and (y) is the output.

• Residual principle: The block learns a correction (F(x)) to the identity mapping instead of a complete transformation.
• Gradient flow: Identity shortcuts provide direct paths for signals and gradients.
• Depth: Standard variants include ResNet-18, ResNet-50, and ResNet-152.
• Projection shortcut: A (1 \times 1) convolution aligns dimensions when input and output shapes differ.

E. Inception networks

Inception networks process inputs through parallel branches to capture patterns at multiple spatial scales.

• Module branches: A module may combine (1 \times 1), (3 \times 3), and (5 \times 5) convolutions with pooling.
• Concatenation: Branch outputs are joined along the channel dimension.
• Bottlenecks: (1 \times 1) convolutions reduce channel counts before expensive operations.
• Efficiency: Later versions factorize large kernels, such as replacing (5 \times 5) with two (3 \times 3) convolutions.
• Trade-off: Multi-branch modules are efficient but structurally more complex than sequential VGG-style networks.

IV. Generalization and Optimization — Reliable CNN Training

A. Regularization techniques

Regularization reduces overfitting by limiting model complexity or introducing controlled variation.

• L2 regularization: Adds (\lambda\sum_i w_i^2) to the loss, discouraging excessively large weights.
• L1 regularization: Adds (\lambda\sum_i |w_i|), encouraging sparse parameters.
• Weight decay: During an update, weights are reduced in magnitude; decoupled weight decay is used by optimizers such as AdamW.
• Model-level methods: Dropout, augmentation, early stopping, and reduced capacity also improve generalization.

B. Batch normalization

Batch normalization standardizes intermediate activations and then learns a new scale and shift.

x_hat = (x - μB) / sqrt(σB² + ε)
y = γx_hat + β

Here, (\mu_B) and (\sigma_B^2) are mini-batch statistics, (\epsilon) ensures numerical stability, and (\gamma,\beta) are learned parameters.

• Training effect: More stable activation scales often permit larger learning rates and faster convergence.
• Inference: Running estimates of mean and variance replace mini-batch statistics.
• Placement: A common sequence is convolution, batch normalization, then ReLU.
• Limitation: Very small batches produce noisy statistics; group normalization may then be preferable.

C. Dropout

Dropout randomly suppresses activations during training to discourage dependence on particular units.

• Mechanism: Each activation is retained with probability (q=1-p), where (p) is the dropout rate.
• Inverted dropout: Retained activations are divided by (q) during training, so no rescaling is needed at inference.
• Usage: Rates near (p=0.5) are common in dense layers, while convolutional blocks often use smaller rates.
• Limitation: Excessive dropout causes underfitting and may be redundant in strongly augmented, normalized networks.

D. Data augmentation methods

Data augmentation creates label-preserving training variations to improve robustness and effective dataset diversity.

• Geometric methods: Random crops, flips, rotations, translations, and scaling alter spatial presentation.
• Photometric methods: Brightness, contrast, saturation, and color jitter vary imaging conditions.
• Erasing methods: Cutout or random erasing masks image regions, encouraging use of distributed evidence.
• Sample mixing: Mixup interpolates images and labels; CutMix replaces a region with a patch from another example and mixes labels by area.
• Constraint: Transformations must preserve meaning; a vertical flip may invalidate labels in street scenes.

E. Learning rate scheduling

Learning rate scheduling changes the optimizer’s step size as training progresses.

• Step decay: Multiplies the rate by a factor at fixed epochs, such as (0.1) every 30 epochs.
• Cosine decay: Smoothly reduces the rate toward a minimum following a cosine curve.
• Warm-up: Begins with a small rate and increases it over several iterations, stabilizing large-batch training.
• Plateau scheduling: Reduces the rate when validation performance stops improving.
• Role: Large early steps accelerate learning; smaller late steps refine parameters near a minimum.

F. Weight initialization

Weight initialization sets appropriate starting scales so activations and gradients neither vanish nor explode.

• Xavier initialization: Suitable for tanh-like activations, with variance based on both fan-in and fan-out.
• He initialization: Designed for ReLU networks, commonly using variance (2/n{in}), where (n{in}) is fan-in.
• Biases: Bias parameters are commonly initialized to zero.
• Symmetry requirement: Initializing all weights identically prevents hidden units from learning distinct features.
• Transfer learning: Pretrained weights can replace random initialization when source and target visual domains are related.

G. Early stopping

Early stopping ends training when validation performance ceases to improve, preventing continued fitting of training noise.

• Monitored quantity: Validation loss or a task metric is checked after each epoch.
• Patience: Training may stop after, for example, 10 epochs without sufficient improvement.
• Checkpointing: The parameters from the best validation epoch are restored rather than retaining the final epoch.
• Trade-off: Small patience may stop useful learning; excessive patience weakens regularization and wastes computation.