Unit 3: Classifying Images with Deep Convolutional Neural Networks

I. Orientation — The CNN Principle

A convolutional neural network (CNN) is a deep learning model designed to process grid-structured data such as images. CNNs learn spatial hierarchies: early layers detect edges and textures, intermediate layers combine them into shapes, and deeper layers recognize object-level patterns.

• Input convention: An image is represented by height (H), width (W), and channels (C); an RGB image has shape (H\times W\times3).
• Local connectivity: Each convolutional neuron reads only a small receptive field, such as (3\times3), rather than the entire image.
• Parameter sharing: One filter uses the same weights at every spatial location, greatly reducing the parameter count.
• Translation-related response: Shifting a feature in the input shifts its feature-map activation; pooling and global aggregation can add approximate translation invariance.
• Learned features: Filter values are optimized through backpropagation rather than manually specified.
• Typical pipeline: Input (\rightarrow) convolution (\rightarrow) activation (\rightarrow) subsampling (\rightarrow) deeper features (\rightarrow) classifier.
• Classification output: For (K) mutually exclusive classes, softmax converts logits (z_k) into probabilities:
  
  TEXT

  p_k = exp(z_k) / Σ_j exp(z_j)

  
  Here, (p_k) is the probability of class (k), and (z_k) is its unnormalized score.

II. CNN Components — Local Feature Extraction

Convolutional networks are assembled from specialized layers that extract, transform, regularize, and classify visual features.

A. Building blocks of convolutional neural networks

The main CNN building blocks progressively convert image pixels into discriminative class scores.

• Convolutional layer: Applies (F) learnable kernels to produce (F) feature maps; a (3\times3) kernel on an RGB input contains (3\times3\times3=27) weights per filter.
• Activation function: Introduces nonlinearity; ReLU is commonly defined as:
  
  TEXT

  ReLU(x) = max(0, x)

  
  Without nonlinear activations, stacked convolutions would remain equivalent to one linear transformation.
• Pooling layer: Reduces spatial dimensions using operations such as maximum or average selection over a local window.
• Batch normalization: Normalizes intermediate activations and learns scale and shift parameters, often stabilizing and accelerating optimization.
• Dropout: Randomly sets a proportion of activations to zero during training; a rate of (0.5) drops half the selected units on average.
• Dense layer: Connects extracted features to output units but can introduce many parameters after flattening.
• Global average pooling: Replaces each feature map by its spatial mean, reducing parameters compared with a large dense layer.
• Loss function: Sparse categorical cross-entropy is suitable when labels are integer class indices:
  
  TEXT

  L = -log(p_y)

  
  Here, (p_y) is the predicted probability of the correct class (y).

III. Spatial Dimensions — Shape Calculation

Output-shape calculation determines whether layers are compatible and how rapidly spatial resolution decreases.

A. Determining the size of the convolution output

A convolution’s output size depends on input size, kernel size, padding, stride, and dilation.

• General one-dimensional formula:
  
  TEXT

  O = floor((N + 2P - D(K - 1) - 1) / S) + 1

  
  (N) is input size, (K) kernel size, (P) padding per side, (S) stride, (D) dilation, and (O) output size.
• Two-dimensional use: Apply the formula separately to height and width; output depth equals the number of filters (F).
• Valid padding: Uses (P=0), so the filter is placed only where it lies completely inside the input.
• Same padding: With stride (1), padding is chosen to preserve height and width; an odd (K) usually uses (P=(K-1)/2).
• Worked example: For (N=32), (K=5), (P=2), (S=1), and (D=1):
  
  TEXT

  O = floor((32 + 4 - 4 - 1) / 1) + 1 = 32

  
  With 16 filters, the output shape is (32\times32\times16).

IV. Convolution Operation — Computing Feature Maps

Discrete two-dimensional convolution combines neighboring pixel values with kernel weights to detect local visual patterns.

A. Performing a discrete convolution in 2D

At each location, the kernel and corresponding image patch are multiplied element by element and summed.

• Mathematical convolution:
  
  TEXT

  Y[i,j] = Σ_m Σ_n X[i-m, j-n] K[m,n]

  
  (X) is the input, (K) the kernel, and (Y) the output feature map.
• Cross-correlation in CNNs: Deep learning libraries usually omit kernel reversal:
  
  TEXT

  Y[i,j] = b + Σ_m Σ_n X[i+m, j+n] K[m,n]

  
  Because (K) is learned, this convention does not reduce model capability.
• Multiple channels: For an RGB image, every filter spans all three channels; products are summed across height, width, and channel dimensions.
• Multiple filters: Each filter creates one output channel, so 64 filters produce a depth of 64.
• Bias and activation: A scalar bias (b) is added before applying an activation such as ReLU.
• Learned detectors: During training, kernels may become responsive to vertical edges, color contrasts, corners, textures, or more complex patterns.

V. Resolution Reduction — Compact Feature Representations

Subsampling reduces feature-map size, computation, and sensitivity to small positional changes.

A. Subsampling

Subsampling summarizes local neighborhoods through pooling or strided operations.

1. Max pooling:
   • Operation: Returns the largest value in each window.
   • Effect: Preserves strong feature evidence; (2\times2) pooling with stride 2 usually halves height and width.
2. Average pooling:
   • Operation: Returns the arithmetic mean of each window.
   • Effect: Produces smoother summaries but may weaken sharply localized activations.

• Strided convolution: A convolution with (S>1) performs learned downsampling and may replace pooling.
• Global average pooling: Converts an (H\times W\times C) tensor into a (C)-element vector by averaging every channel spatially.
• Trade-off: Downsampling lowers memory use and enlarges the effective receptive field, but excessive reduction destroys fine spatial detail.

VI. CNN Architecture — End-to-End Classification

A complete CNN connects feature extraction to a classifier and is trained jointly using gradient-based optimization.

A. Putting everything together to build a CNN

CNN design typically increases channel depth while decreasing spatial resolution.

• Example architecture:
  
  TEXT

  Input 64×64×3
  → Conv(32, 3×3) + ReLU
  → MaxPool(2×2)
  → Conv(64, 3×3) + ReLU
  → MaxPool(2×2)
  → GlobalAveragePooling
  → Dense(K, softmax)

• Feature progression: The first convolution learns simple patterns; deeper layers combine them into class-specific structures.
• Parameter planning: A convolution with kernel (K_h\times Kw), (C{in}) input channels, and (C_{out}) filters has:
  
  TEXT

  Parameters = (K_h K_w C_in + 1) C_out

  
  The added (1) represents one bias per filter.
• Training cycle: Forward propagation computes predictions, loss measures error, backpropagation computes gradients, and an optimizer updates parameters.
• Evaluation: Validation data guides model selection; test data estimates final generalization.
• Overfitting control: Data augmentation, dropout, weight decay, and early stopping reduce memorization.

VII. TensorFlow Workflow — Model Construction and Training

TensorFlow and Keras provide layers, automatic differentiation, GPU execution, and high-level training utilities.

A. Implementing a deep convolutional neural network using TensorFlow

A TensorFlow CNN can be defined sequentially and trained with mini-batch gradient descent.

import tensorflow as tf

model = tf.keras.Sequential([
    tf.keras.layers.Input(shape=(64, 64, 3)),
    tf.keras.layers.Rescaling(1./255),
    tf.keras.layers.Conv2D(32, 3, padding="same", activation="relu"),
    tf.keras.layers.MaxPooling2D(),
    tf.keras.layers.Conv2D(64, 3, padding="same", activation="relu"),
    tf.keras.layers.MaxPooling2D(),
    tf.keras.layers.GlobalAveragePooling2D(),
    tf.keras.layers.Dense(10, activation="softmax")
])

model.compile(
    optimizer=tf.keras.optimizers.Adam(learning_rate=1e-3),
    loss="sparse_categorical_crossentropy",
    metrics=["accuracy"]
)

model.fit(train_ds, validation_data=val_ds, epochs=20)

• Input scaling: Rescaling(1./255) maps 8-bit pixel values from (0!-!255) to (0!-!1).
• Dataset pipeline: tf.data.Dataset supports batching, shuffling, caching, and prefetching.
• Inference: model.predict(images) returns class probabilities for each image.
• Callbacks: EarlyStopping and ModelCheckpoint can preserve the best validation model and limit overfitting.

VIII. Reusing Learned Features — Efficient Adaptation

Transfer learning adapts representations learned from a large source dataset to a smaller target task.

A. Transfer learning with pre-trained CNN

A pre-trained network serves as a feature extractor or as an initialization for fine-tuning.

• Base model: Architectures such as VGG, ResNet, EfficientNet, and MobileNet are commonly available with ImageNet-trained weights.
• Feature extraction: Remove the original classifier, freeze convolutional layers, and train a new task-specific output head.
• Fine-tuning: Unfreeze selected upper layers after the new head stabilizes, then train with a small learning rate such as (10^{-5}).
• Input requirement: Images must follow the model’s expected size and preprocessing function.
• Benefit: Useful edge, texture, and shape features can substantially reduce data and training requirements.
• Limitation: Transfer is weaker when the target domain differs greatly from natural images; careless fine-tuning may cause catastrophic forgetting.

IX. Dataset Expansion — Improving Generalization

Data augmentation creates varied training examples while preserving their intended labels.

A. Data augmentation

Augmentation exposes a model to plausible transformations without collecting new images.

• Geometric transformations: Random flipping, rotation, translation, zoom, and cropping vary object position and orientation.
• Photometric transformations: Brightness, contrast, saturation, and color jitter simulate illumination changes.
• Regularizing methods: Cutout masks image regions; MixUp interpolates image-label pairs; CutMix replaces one region with a patch from another image.
• Pipeline rule: Apply random transformations only during training, not validation or testing.
• Label preservation: Transformations must fit the domain; horizontal flipping may be unsuitable for text, directional signs, or asymmetric medical anatomy.
• Segmentation requirement: Geometric transformations must be applied identically to an image and its pixel mask.

X. Pixel-Level Prediction — Dense Visual Understanding

Image segmentation assigns a category or identity to every pixel rather than producing one label for the entire image.

A. Image segmentation

Segmentation architectures preserve or reconstruct spatial detail to generate an output mask.

• Semantic segmentation: Labels all pixels by class, such as road, vehicle, or pedestrian, without separating individual objects.
• Instance segmentation: Distinguishes separate objects of the same class, such as three individual vehicles.
• Encoder–decoder design: The encoder extracts low-resolution features; the decoder upsamples them to the original image resolution.
• U-Net principle: Skip connections transfer high-resolution encoder features to corresponding decoder stages, restoring boundary detail.
• Output layer: Binary tasks commonly use one sigmoid channel; multiclass tasks use (K) softmax channels per pixel.
• Metrics: Intersection over Union is:
  
  TEXT

  IoU = |Prediction ∩ Target| / |Prediction ∪ Target|

  
  Dice score is (2|P\cap T|/(|P|+|T|)), where (P) is the predicted pixel set and (T) the target set.
• Challenge: Class imbalance, small objects, and uncertain boundaries can make pixel accuracy misleading.

XI. GPU Operations — Monitoring and Development

NVIDIA command-line utilities support GPU inspection, CUDA compilation, debugging, and performance analysis.

A. NVIDIA Command Line Tools and Utilities

These tools help verify hardware availability and diagnose deep learning workloads.

• nvidia-smi: Displays GPU model, driver version, utilization, temperature, power, memory consumption, and active processes.
  
  BASH

  nvidia-smi
  nvidia-smi -l 2

  
  The second command refreshes the report every two seconds.
• Process monitoring: nvidia-smi pmon reports GPU activity by process, helping identify competing workloads.
• CUDA compiler: nvcc --version displays the installed CUDA compiler version; nvcc file.cu -o app compiles CUDA source code.
• Device listing: nvidia-smi -L lists available GPUs and their identifiers.
• GPU selection:
  
  BASH

  CUDA_VISIBLE_DEVICES=0 python train.py

  
  This exposes GPU 0 to the training process.
• Profiling tools: nsys from Nsight Systems analyzes CPU–GPU timelines, while ncu from Nsight Compute examines CUDA kernel performance.
• Operational distinction: Driver-supported CUDA capability, installed toolkit version, and TensorFlow’s required CUDA/cuDNN versions must be compatible.
• Safety: Terminating GPU processes or resetting devices can interrupt other users and should be done only with appropriate permissions.