Unit 3: Classifying Images with Deep Convolutional Neural Networks - Subjective Questions

A convolutional neural network (CNN) is a deep neural network designed to process grid-structured data such as images. It learns spatial hierarchies of features automatically.

The main building blocks are:

• Convolutional layer: Applies learnable filters to local regions of the input to generate feature maps.
• Activation function: Introduces non-linearity. ReLU is commonly defined as .
• Pooling layer: Reduces the spatial dimensions of feature maps and makes representations less sensitive to small translations.
• Batch normalization: Normalizes intermediate activations, improving training stability and speed.
• Dropout: Randomly disables neurons during training to reduce overfitting.
• Fully connected layer: Combines extracted features to perform classification.
• Output layer: Produces class probabilities, commonly through the softmax function.

Early convolutional layers generally detect edges and textures, while deeper layers learn object parts and high-level semantic features.

• Local receptive field: Each output neuron is connected only to a small spatial region of the input. For example, a filter processes nine neighboring input values at a time.
• Sparse connectivity: Unlike a fully connected layer, a convolutional neuron does not connect to every input value. This substantially reduces the number of parameters.
• Parameter sharing: The same filter weights are applied at every valid spatial location. A feature can therefore be detected regardless of where it appears in the image.
• Feature map: The output produced by sliding one filter over the input is called a feature map. Different filters generate different feature maps.

If a layer contains filters, it produces output channels. These properties make CNNs computationally efficient and well suited to extracting spatial patterns from images.

For an input of spatial size , kernel size , padding , and stride , the output dimensions are

For the given values:

Each filter produces one feature map, and there are filters. Therefore, the final output shape is

The first two dimensions are spatial dimensions, while the last dimension is the number of output channels.

Valid padding and same padding determine how a convolution handles image boundaries.

• Valid padding: No zeros are added around the input. The filter is placed only where it fits completely inside the input. For stride , the output size is , so the feature map becomes smaller.
• Same padding: Zeros are added around the input so that the output spatial size is preserved when stride is . For an odd kernel of size , the padding on each side is usually .

For a input and a kernel with stride :

• Valid padding gives a output.
• Same padding with gives a output.

Same padding helps preserve boundary information and supports deeper networks, whereas valid padding avoids introducing artificial zero values.

A discrete two-dimensional convolution combines a small matrix called a kernel with local image regions. The kernel moves across the input, and at each position, corresponding values are multiplied and summed.

For input and kernel , mathematical convolution can be written as

This definition flips the kernel in both spatial directions. Deep-learning libraries usually compute cross-correlation instead:

The difference does not limit learning because CNN kernel values are trainable.

The kernel's role is to detect a particular local pattern. For example:

• Edge kernels detect intensity changes.
• Blur kernels smooth an image.
• Learned CNN kernels identify useful textures, shapes, and object components.

A bias is commonly added to each output channel before applying an activation function.

A input processed by a kernel with valid padding and stride produces a output.

For the top-left region:

For the top-right region:

For the bottom-left region:

For the bottom-right region:

Therefore, the output feature map is

This is cross-correlation because the kernel is applied directly without being flipped.

Subsampling reduces the spatial dimensions of a feature map while retaining important information. Pooling is a common subsampling method.

For the region

• Max pooling selects the largest value: .
• Average pooling computes the mean: .

Comparison:

• Max pooling preserves the strongest activation and is effective when the presence of a feature is more important than its exact location.
• Average pooling preserves the average response and produces smoother representations.
• Both reduce computation, memory consumption, and sensitivity to small translations.
• Excessive pooling may discard useful spatial information.

A pooling window with stride normally halves the height and width of a feature map.

Pooling-based subsampling:

• Applies a fixed operation such as maximum or average over local regions.
• Introduces no trainable parameters.
• Is computationally inexpensive and provides limited translation invariance.
• May discard information because the reduction rule cannot adapt to the data.

Strided convolution:

• Uses a convolution with stride greater than to learn features while reducing spatial size.
• Contains trainable parameters and can learn a task-specific downsampling operation.
• Usually requires more computation than pooling.
• Can introduce aliasing if high-frequency information is sampled without suitable filtering.

Pooling is simple and effective in conventional CNNs. Strided convolution gives the model more flexibility and is common in modern architectures. Both approaches should be used carefully when precise spatial details are required, especially in image segmentation.

A typical image-classification CNN follows these stages:

1. Input: An image is represented as a tensor of height, width, and channels.
2. Convolution: Filters generate feature maps from local image regions.
3. Activation: ReLU introduces non-linearity using .
4. Pooling or strided convolution: Spatial dimensions are reduced while useful responses are retained.
5. Repeated feature extraction: Additional convolutional blocks learn increasingly abstract patterns.
6. Flattening or global average pooling: Feature maps are converted into a compact feature vector.
7. Dense classification layer: Features are mapped to class scores called logits.
8. Output activation: Softmax converts logits into class probabilities:

During training, backpropagation adjusts all trainable filters and weights to minimize a loss such as categorical cross-entropy.

Each convolutional filter spans all input channels. The number of weights per filter is

Including one bias gives parameters per filter. For filters:

A fully connected layer receives input values. For outputs, including one bias per output:

Thus, the convolutional layer uses only parameters compared with in the dense layer. This reduction results from local connectivity and parameter sharing, which also allow the convolutional layer to exploit image structure.

A TensorFlow CNN can be implemented with the following workflow:

• Load images and divide them into training, validation, and test sets.
• Resize images, normalize pixel values, and batch the data using tf.data.
• define a tf.keras.Sequential model or use the Functional API.
• Add layers such as Conv2D, BatchNormalization, ReLU, MaxPooling2D, Dropout, GlobalAveragePooling2D, and Dense.
• Compile the model with an optimizer, loss function, and metrics.
• Train it with model.fit() and evaluate it with model.evaluate().

A typical model structure is:

Input -> Conv2D -> ReLU -> MaxPooling2D -> Conv2D -> ReLU -> GlobalAveragePooling2D -> Dense

For multiclass classification, one may use the Adam optimizer and sparse categorical cross-entropy. Callbacks such as early stopping and model checkpoints help control overfitting and preserve the best model. Final performance should be reported on test data that was not used for model selection.

• Batch normalization: Normalizes intermediate activations using batch statistics, followed by learnable scale and shift parameters. It improves numerical stability and often permits faster training.
• Dropout: Randomly sets a fraction of activations to zero during training. This discourages neurons from depending excessively on one another and reduces overfitting.
• Early stopping: Monitors validation performance and stops training when improvement has ceased for a specified number of epochs. It prevents unnecessary training and limits overfitting.
• Model checkpoint: Saves model weights when a monitored quantity, such as validation loss, improves. It ensures that the best-performing version can be restored.

These techniques serve different purposes. Batch normalization primarily stabilizes optimization, while dropout and early stopping act as regularizers. Checkpointing provides reliable model recovery and selection.

Transfer learning reuses features learned by a model trained on a large source dataset, such as ImageNet, for a new target task.

The procedure is:

1. Load a pre-trained model such as ResNet, EfficientNet, or MobileNet without its original classification head.
2. Apply the input size and preprocessing expected by that architecture.
3. Freeze the convolutional base by setting its layers as non-trainable.
4. Add a new task-specific head, often consisting of global average pooling, dropout, and a dense output layer.
5. Train the new head on the target dataset.
6. Optionally unfreeze some deeper layers and fine-tune them using a small learning rate.
7. Recompile after changing layer trainability and evaluate on an independent test set.

Transfer learning generally reduces training time and data requirements because early and intermediate CNN features are useful across many visual tasks.

Feature extraction:

• The pre-trained convolutional base remains frozen.
• Only a newly added classification head is trained.
• It is computationally efficient and reduces overfitting.
• It is appropriate when the target dataset is small or similar to the pre-training dataset.

Fine-tuning:

• Some or all pre-trained layers are unfrozen and updated on target data.
• It can adapt high-level features to the new domain.
• It requires more computation and carries a greater overfitting risk.
• It is useful when sufficient target data is available or when the target domain differs from the source domain.

Fine-tuning should normally begin only after training the new head. A low learning rate is used to avoid destroying useful pre-trained representations, a problem known as catastrophic forgetting.

Data augmentation creates varied training examples by applying label-preserving transformations to existing images. It increases effective data diversity and improves generalization.

Common operations include:

• Horizontal or vertical flipping
• Small rotations and translations
• Random cropping, resizing, and zooming
• Brightness, contrast, saturation, or hue adjustment
• Random noise, blur, erasing, MixUp, or CutMix

Precautions include:

• Transformations must preserve the correct label. For example, vertical flipping may be invalid for handwritten digits or road scenes.
• Validation and test images should receive only deterministic preprocessing, not random augmentation.
• Geometric transformations must also update masks or bounding boxes in segmentation and detection tasks.
• Excessively strong augmentation can generate unrealistic samples and cause underfitting.
• Preprocessing must remain compatible with any pre-trained model being used.

In TensorFlow, augmentation can be implemented with Keras preprocessing layers or a tf.data pipeline.

A TensorFlow pipeline commonly uses tf.data.Dataset and performs the following operations:

• Decode and resize: Convert image files into tensors of a consistent shape.
• Normalization: Scale values, for example from to , or apply the preprocessing required by a pre-trained network.
• Shuffling: Randomize training-example order to reduce correlations between successive batches.
• Augmentation: Apply random, label-preserving transformations only to training data.
• Batching: Group examples into tensors that can be processed efficiently by the accelerator.
• Caching: Store decoded or preprocessed data in memory or on disk to avoid repeating expensive work.
• Prefetching: Prepare future batches while the model processes the current batch, reducing input bottlenecks.

A typical conceptual sequence is load -> shuffle -> decode/resize -> augment -> normalize -> batch -> prefetch. Caching may be inserted where memory capacity and randomness requirements permit. The exact order should preserve fresh random augmentation across epochs.

Image segmentation assigns a label to each image pixel, producing a spatially detailed understanding of a scene.

• Semantic segmentation: Every pixel receives a class label, but separate objects of the same class are not distinguished. All cars, for example, share one class mask.
• Instance segmentation: Each object instance receives its own mask. Two cars are represented as distinct objects even though they have the same class.
• Panoptic segmentation: Combines semantic and instance segmentation. Countable objects are separated into instances, while background regions such as sky or road receive semantic labels.

Segmentation differs from image classification because classification predicts one or more labels for the whole image. It also differs from object detection, which generally represents objects with bounding boxes rather than pixel-accurate masks.

An encoder-decoder segmentation network transforms an image into a pixel-level output mask.

• The encoder applies convolution and downsampling to learn high-level contextual features. Spatial resolution decreases while channel depth usually increases.
• The decoder restores spatial resolution using interpolation, transposed convolution, or other learned upsampling operations.
• The final layer predicts class logits for every pixel.

Upsampling is necessary because the encoder's feature maps are smaller than the input image. It reconstructs an output with the required spatial dimensions.

Skip connections, as used in U-Net, transfer higher-resolution encoder features directly to corresponding decoder stages. They restore edge and location information that may be lost during downsampling. This combination allows the model to use both high-level semantic context and low-level spatial detail, producing more accurate object boundaries.

Pixel-wise cross-entropy treats every pixel as a classification problem. For one-hot target and predicted probability , it is

It may be weighted to address class imbalance.

Intersection over Union (IoU) measures overlap between predicted mask and ground-truth mask :

Dice coefficient is defined as

Both IoU and Dice range from to , with representing perfect overlap. Dice-based loss is useful when foreground pixels are rare. Mean IoU averages IoU across classes. Pixel accuracy can also be reported, but it may be misleading when a large background class dominates the image.

Important NVIDIA tools include:

• nvidia-smi: Displays GPU models, driver versions, CUDA compatibility, memory usage, utilization, temperature, power consumption, and active processes. Options such as -l provide periodic monitoring.
• nvcc --version: Reports the installed CUDA compiler toolkit version.
• nvidia-smi dmon: Continuously reports device-level utilization and performance statistics.
• nvidia-smi pmon: Monitors GPU usage by individual processes.
• Compute Sanitizer: Detects memory-access errors, race conditions, initialization errors, and synchronization problems in CUDA applications.
• Nsight Systems: Profiles system-wide execution, including CPU activity, CUDA kernels, data transfers, and synchronization.
• Nsight Compute: Provides detailed kernel-level performance metrics.

These utilities help verify that TensorFlow can access the expected GPU, detect out-of-memory conditions, locate performance bottlenecks, and identify competing processes. Driver, CUDA, framework, and library compatibility must also be checked when diagnosing GPU problems.