Unit 2: Convolutional Neural Networks and Training Techniques - Subjective Questions

A Convolutional Neural Network (CNN) is a deep neural network designed to process grid-structured data such as images.

Its major building blocks are:

• Convolutional layers: Apply learnable filters to detect local patterns such as edges, textures, and shapes.
• Activation functions: Introduce non-linearity, commonly using ReLU: .
• Pooling layers: Reduce spatial dimensions and make features less sensitive to small translations.
• Fully connected layers: Combine extracted features to perform classification or regression.
• Output layer: Produces class probabilities or task-specific predictions.

CNNs are effective because they use local connectivity, parameter sharing, and hierarchical feature learning. Early layers learn simple features such as edges, while deeper layers learn complex objects and semantic structures. Parameter sharing also gives CNNs far fewer parameters than fully connected networks operating directly on image pixels.

The convolution operation slides a small filter over an input and computes a weighted sum at each spatial location. In deep learning, the operation typically implemented is technically cross-correlation because the filter is not reversed.

For a two-dimensional single-channel input and filter , an output element is:

where:

• and are the filter height and width.
• is a learnable bias.
• is the response at position .

For an input with channels:

Each filter spans all input channels and generates one output feature map. During training, the filter weights are optimized through backpropagation to respond to useful visual patterns.

Filters, also called kernels, are small learnable weight matrices that scan across an input. A filter is trained to respond strongly to a particular pattern, such as a horizontal edge, texture, curve, or object part.

A feature map is the output produced by applying one filter across the complete input. Its values indicate where and how strongly the learned pattern occurs.

Their relationship can be summarized as follows:

• One filter generally produces one output feature map.
• A convolutional layer with filters produces output feature maps.
• Each filter has dimensions , where is the number of input channels.
• Early-layer feature maps represent low-level patterns.
• Deeper-layer feature maps represent increasingly abstract features.

Thus, filters act as pattern detectors, while feature maps preserve the spatial locations of detected patterns.

Padding adds pixels, commonly zeros, around the input boundary. It controls output size and allows filters to process border pixels. Stride is the number of positions by which a filter moves after each operation.

For input size , filter size , padding , and stride , the output size along one spatial dimension is:

For a two-dimensional input:

Important cases include:

• Valid convolution: , so spatial dimensions usually decrease.
• Same convolution: Padding is selected to preserve dimensions when ; for an odd kernel, .
• Larger stride: Reduces output resolution and computation but may discard spatial detail.

Pooling summarizes a local region of a feature map and reduces its spatial dimensions. It lowers computation, increases the effective receptive field, and gives limited robustness to small translations.

• Max pooling: Selects the largest value in each pooling window. It retains the strongest detected feature and is widely used in classification CNNs.
• Average pooling: Computes the mean value in each window. It produces smoother representations but may weaken prominent activations.
• Global average pooling: Averages every spatial value in each feature map, producing one value per channel. For channel :

Global average pooling can replace large fully connected layers, significantly reducing parameters and overfitting. Pooling has no learnable weights, although aggressive pooling can remove useful location and fine-detail information.

The receptive field of a neuron is the region of the original input that can influence that neuron's activation. A larger receptive field allows the neuron to use broader contextual information.

If layer has kernel size and stride , the receptive field and effective jump can be calculated recursively:

with and .

The receptive field increases through:

• Larger kernels, which observe wider local regions.
• Higher strides or pooling, which increase the spacing represented by later activations.
• Greater depth, since multiple small-kernel layers accumulate a large receptive field.
• Dilated convolution, which spreads kernel elements over a wider region.

For example, two stride-one convolutions have an effective receptive field of , while using fewer parameters and more non-linearities than one convolution.

LeNet-5, introduced by Yann LeCun and collaborators, is an early CNN designed primarily for handwritten digit recognition.

Its architecture follows this general sequence:

• Input image of approximately pixels.
• Convolutional layer with filters.
• Subsampling or average-pooling layer.
• Second convolutional layer followed by subsampling.
• Additional convolutional or fully connected processing.
• Fully connected output classifier.

LeNet-5 established several foundational CNN ideas:

• Local receptive fields.
• Shared convolutional weights.
• Alternating convolution and downsampling.
• End-to-end learning using gradient-based optimization.

Its major historical contribution was demonstrating that learned hierarchical image features could outperform manually designed features for document and digit recognition.

AlexNet was a deep CNN that achieved a major improvement in the 2012 ImageNet competition. It contained five convolutional layers followed by three fully connected layers.

Its important features included:

• ReLU activations, which trained faster than sigmoid or tanh functions.
• GPU-based training, making large-scale CNN optimization practical.
• Overlapping max pooling for spatial downsampling.
• Dropout in fully connected layers to reduce overfitting.
• Data augmentation, including image crops, translations, and horizontal reflections.
• A large number of filters and parameters suitable for the ImageNet dataset.

AlexNet was a breakthrough because it substantially reduced classification error compared with traditional computer vision methods. It demonstrated that deep CNNs trained on large datasets with powerful hardware could learn highly effective visual representations.

VGG networks use a simple and uniform architecture built mainly from stride-one convolutions, max pooling, and fully connected layers. VGG-16 and VGG-19 contain 16 and 19 learnable layers, respectively.

Stacking small filters has two major advantages. Two convolutions produce an effective receptive field. For equal input and output channel count , their approximate parameter count is:

A single convolution requires:

Therefore, stacked layers use fewer parameters. They also introduce an additional non-linear activation, increasing representational power.

VGG demonstrated that increasing depth with a consistent architecture improves feature learning. Its disadvantages are high memory consumption, expensive computation, and very large fully connected layers.

ResNet introduces a residual block that learns a residual mapping instead of directly learning a desired mapping . The block output is:

where is carried through an identity skip connection. If dimensions differ, a projection such as a convolution may be used:

Skip connections help because:

• They provide short paths for gradients during backpropagation.
• The derivative contains an identity component, reducing vanishing-gradient effects.
• A block can approximate an identity mapping by learning .
• Very deep networks can be optimized without the training-error degradation seen in plain networks.
• Earlier features can flow directly to later layers.

ResNet enabled practical training of networks with tens or hundreds of layers and became a standard backbone for classification, detection, and segmentation.

An Inception module processes the same input through multiple parallel branches and concatenates their outputs along the channel dimension. Typical branches include:

• A convolution.
• A convolution followed by a convolution.
• A convolution followed by a convolution or factorized alternatives.
• A pooling operation followed by a convolution.

Parallel branches capture patterns at different spatial scales. Small kernels detect local details, while larger kernels use broader context.

The convolutions:

• Reduce the number of channels before expensive convolutions.
• Decrease parameter count and computational cost.
• Mix information across channels.
• Add non-linear transformations when followed by activation functions.

If the branch outputs are , the module output is:

This multi-scale design achieves strong accuracy while controlling computation.

The architectures represent major stages in CNN development:

• LeNet: Established local connectivity, shared weights, and alternating convolution and subsampling for digit recognition.
• AlexNet: Scaled CNNs to ImageNet using ReLU, GPU training, dropout, max pooling, and data augmentation.
• VGG: Showed that deeper networks built from repeated convolutions could learn strong representations.
• Inception: Introduced parallel multi-scale branches and bottleneck convolutions to improve computational efficiency.
• ResNet: Introduced identity skip connections and residual learning, enabling extremely deep networks to train effectively.

In terms of trade-offs, VGG is simple but computationally expensive, Inception is efficient but architecturally complex, and ResNet offers strong optimization behavior and reusable residual blocks. Together, these models illustrate the progression from shallow CNNs to deeper, more efficient, and easier-to-optimize architectures.

Regularization refers to techniques that improve generalization by preventing a model from fitting noise or irrelevant details in the training data.

Common CNN regularization techniques include:

• L2 regularization or weight decay: Adds a parameter penalty to the loss:

• L1 regularization: Adds and can encourage sparse parameters.
• Dropout: Randomly removes activations during training.
• Data augmentation: Produces varied training examples through label-preserving transformations.
• Early stopping: Stops training when validation performance no longer improves.
• Batch normalization: Primarily stabilizes optimization but may also provide a mild regularizing effect due to mini-batch statistics.

The regularization strength must be balanced. Too little may cause overfitting, while too much can cause underfitting and prevent the network from learning useful patterns.

Batch normalization normalizes activations using mini-batch statistics and then applies a learnable scale and shift.

For a mini-batch :

Here, and are learned parameters. In CNNs, statistics are generally computed per channel across batch and spatial dimensions.

During training, batch statistics are used and running estimates are updated. During inference, stored running means and variances are used so predictions do not depend on the current batch.

Benefits include faster and more stable optimization, reduced sensitivity to initialization, support for larger learning rates, and a mild regularizing effect.

Dropout randomly sets selected activations to zero during training. If is sampled from a Bernoulli distribution with keep probability , inverted dropout computes:

The division by preserves the expected activation:

During inference, dropout is disabled and all activations are used without additional scaling when inverted dropout was used during training.

Dropout reduces overfitting by:

• Preventing units from relying excessively on particular other units.
• Encouraging distributed and robust feature representations.
• Training many implicit subnetworks that share parameters.
• Acting approximately like an ensemble at inference time.

Dropout is often used in fully connected layers. Very high dropout rates or unnecessary dropout in convolutional layers can slow convergence and cause underfitting.

Data augmentation increases training diversity by creating modified examples while preserving or appropriately transforming their labels.

Common label-preserving transformations include:

• Random cropping and resizing.
• Horizontal or vertical flipping when valid for the domain.
• Small rotations, translations, scaling, and affine transformations.
• Brightness, contrast, saturation, and hue adjustments.
• Noise injection, blur, random erasing, or cutout.

Geometric operations are not universally valid. For example, flipping a medical image or rotating a digit may alter its meaning, so augmentation must respect the task.

Sample-mixing methods include:

• Mixup: Combines two images and labels:

• CutMix: Replaces a region of one image with a region from another and mixes labels according to region area.

Augmentation reduces overfitting, improves invariance, and helps the model generalize to realistic input variations.

Learning rate scheduling changes the optimizer's learning rate during training. A relatively large learning rate supports rapid early progress, while a smaller later rate enables fine convergence.

• Step decay: Multiplies the rate by a factor after fixed intervals:

• Exponential decay: Reduces the rate smoothly:

• Cosine annealing: Follows a cosine curve toward a minimum value:

• Warm-up: Gradually raises the learning rate during initial iterations before applying another schedule. It is useful for large batches or deep networks where a high initial rate may destabilize training.

Schedules may also respond to validation plateaus. Their purpose is to balance optimization speed, stability, and final generalization.

Weight initialization influences how activation and gradient variances change across layers. Poor initialization can cause activations or gradients to vanish or explode, making deep networks difficult to train. Initializing every weight to the same value is also unsuitable because neurons remain symmetric and learn identical features.

Xavier or Glorot initialization is commonly used with sigmoid or tanh activations. A typical variance is:

He or Kaiming initialization is designed for ReLU-family activations, which discard many negative activations:

where and are the input and output fan sizes.

Both methods aim to maintain stable signal magnitudes through the network. The correct choice depends primarily on the activation function and layer structure.

Early stopping is a regularization method that terminates training when performance on held-out validation data stops improving.

A practical procedure is:

• Divide the available data into training, validation, and test sets.
• Train the model while evaluating a validation metric after each epoch.
• Save a checkpoint whenever the monitored metric improves.
• Stop training if there is no meaningful improvement for a fixed patience period.
• Restore the checkpoint with the best validation result.
• Evaluate the restored model once on the untouched test set.

A minimum delta may be specified so that insignificant changes do not reset patience. Validation loss is often monitored because training loss can continue decreasing even after generalization begins to worsen.

Early stopping saves computation and limits overfitting, but very short patience may stop training during a temporary plateau.

A robust training strategy may contain the following stages:

• Data preparation: Split data into training, validation, and test sets using stratification when class balance matters. Estimate normalization statistics from training data only.
• Augmentation: Apply task-valid random crops, flips, color changes, and optional Mixup or CutMix to training images. Use deterministic resizing and normalization for validation and test images.
• Initialization: Use He initialization for layers followed by ReLU or related activations.
• Normalization: Add batch normalization after convolutions and before or according to the architecture's activation convention.
• Regularization: Use moderate weight decay and, where appropriate, dropout. Avoid excessive simultaneous regularization that causes underfitting.
• Optimization: Start with a suitable optimizer and learning rate. Use warm-up if training is initially unstable, followed by cosine decay or step decay.
• Monitoring: Track training and validation loss, accuracy, and class-sensitive metrics when the data are imbalanced.
• Early stopping: Save the best validation checkpoint and stop after a reasonable patience interval.
• Final evaluation: Restore the best checkpoint and report results on the test set only once model selection is complete.

This strategy controls overfitting while preserving stable gradient flow and efficient convergence. Hyperparameters should be selected using validation results rather than test performance.