Unit 5: Image Segmentation and Advanced Vision Architectures - Subjective Questions

Semantic segmentation is a dense prediction task in which every pixel of an image is assigned a semantic class label such as road, person, vehicle, or background.

For an image , a semantic segmentation model produces an output:

where is the number of classes.

Difference from image classification:

• Image classification predicts one or more labels for the complete image.
• Semantic segmentation predicts a class label for each pixel.
• Classification mainly identifies what is present, whereas segmentation identifies both what is present and where it is located.
• Semantic segmentation does not distinguish different objects belonging to the same class.

For example, all pixels belonging to multiple cars receive the same car label in semantic segmentation.

Instance segmentation detects individual objects and produces a separate pixel-level mask for every detected object instance.

Comparison:

• Semantic segmentation: Assigns a class label to every pixel but merges objects of the same class.
• Object detection: Predicts a class label and bounding box for each object but does not identify its exact boundary.
• Instance segmentation: Predicts a class label, bounding box, and binary mask for each object instance.

For example, if an image contains three people:

• Semantic segmentation labels all person pixels as one connected semantic region.
• Object detection returns three rectangular bounding boxes.
• Instance segmentation returns three independent masks, one for each person.

Instance segmentation is useful in autonomous driving, medical image analysis, robotic manipulation, and image editing, where the precise shape and identity of each object are required.

U-Net is an encoder-decoder convolutional network originally developed for biomedical image segmentation. Its architecture resembles the letter U.

Contracting path or encoder:

• Repeated convolution and activation operations extract image features.
• Pooling or strided convolution reduces spatial resolution.
• The number of feature channels generally increases with depth.
• This path captures semantic and contextual information.

Bottleneck:

• Connects the encoder and decoder.
• Contains high-level features at the lowest spatial resolution.

Expanding path or decoder:

• Upsampling or transposed convolution increases spatial resolution.
• Convolution layers refine the reconstructed feature maps.
• The final layer produces pixel-level class scores.

Skip connections:

• Encoder feature maps are concatenated with corresponding decoder feature maps.
• They restore fine spatial details lost during downsampling.
• They improve boundary localization and gradient flow.

Thus, U-Net combines high-level contextual features with low-level localization information to produce precise segmentation masks.

Skip connections transfer feature maps directly from an encoder stage to the decoder stage having the corresponding spatial resolution.

Their main benefits are:

• Spatial detail preservation: Early encoder features retain edges, textures, and object boundaries.
• Improved localization: Decoder predictions are aligned more accurately with input pixels.
• Feature fusion: High-resolution appearance features are combined with deep semantic features.
• Better gradient flow: Direct paths reduce optimization difficulties in deep networks.
• Recovery of small objects: Fine structures that disappear during pooling can be recovered.

If skip connections are removed:

• Segmentation masks may become coarse or blurry.
• Object boundaries may be inaccurate.
• Small objects may be missed.
• The decoder must reconstruct spatial details only from compressed bottleneck features.
• Training may become more difficult.

Therefore, skip connections are a central reason for U-Net's effectiveness, especially when precise pixel-level localization is required.

The DeepLab family improves semantic segmentation by preserving feature resolution and capturing context at multiple scales.

Its main ideas include:

• Atrous convolution: Enlarges the receptive field without proportionally increasing parameters or reducing feature-map resolution.
• Atrous Spatial Pyramid Pooling (ASPP): Applies parallel atrous convolutions with different dilation rates to capture multi-scale context.
• Pretrained backbone: Networks such as ResNet or Xception extract strong visual features.
• Encoder-decoder refinement: Later DeepLab versions use a decoder to recover boundaries and spatial details.

Evolution:

• DeepLabv1: Introduced atrous convolution and used a fully connected conditional random field for boundary refinement.
• DeepLabv2: Introduced ASPP.
• DeepLabv3: Improved ASPP using batch normalization and image-level features.
• DeepLabv3+: Added an encoder-decoder structure and commonly used depthwise separable convolution.

These techniques allow DeepLab models to recognize objects at different scales while maintaining comparatively accurate segmentation boundaries.

Atrous convolution, also called dilated convolution, inserts spaces between adjacent kernel elements. It expands the receptive field without increasing the number of kernel parameters.

For an input , filter , and dilation rate , the one-dimensional operation is:

For a kernel containing elements, there are intervals between adjacent elements. Each interval has an effective width of . Therefore, the effective kernel size is:

which simplifies to:

For example, when and :

Thus, a -element kernel behaves as if it covered a region of size , while still using only three learned weights. This is useful in segmentation because it captures wider context without excessive downsampling.

Atrous Spatial Pyramid Pooling (ASPP) is a multi-branch module that processes the same feature map using filters with different receptive-field sizes.

A typical ASPP module contains:

• A convolution branch.
• Multiple atrous convolution branches with different dilation rates.
• An image-level pooling branch for global context.
• Concatenation of all branch outputs.
• A projection convolution that fuses the concatenated features.

Importance of ASPP:

• Objects can appear at very different sizes in an image.
• Small dilation rates capture local details.
• Large dilation rates capture wider contextual information.
• Parallel branches avoid selecting only one fixed receptive-field size.
• Image-level pooling provides global scene information.

The fused ASPP representation enables DeepLab to classify pixels using both local appearance and multi-scale context, improving performance on objects with varying sizes and complex surroundings.

U-Net and DeepLabv3+ are both encoder-decoder segmentation models, but they use different mechanisms to preserve detail and capture context.

U-Net:

• Uses a symmetric contracting and expanding architecture.
• Transfers features through skip connections at multiple resolutions.
• Recovers fine boundaries effectively from high-resolution encoder features.
• Is widely used in medical and scientific imaging.
• Can work well with relatively limited training data and suitable augmentation.

DeepLabv3+:

• Uses a powerful classification backbone as its encoder.
• Applies atrous convolution to preserve spatial resolution.
• Uses ASPP to capture multi-scale contextual information.
• Employs a relatively lightweight decoder for boundary refinement.
• Is commonly used for natural-scene and large-scale semantic segmentation.

Key distinction:

U-Net emphasizes repeated encoder-decoder feature fusion, whereas DeepLabv3+ emphasizes atrous convolution and multi-scale context aggregation. The better choice depends on dataset size, object-scale variation, computational limits, and the required boundary precision.

Mask R-CNN extends Faster R-CNN by adding a parallel branch that predicts an object mask for every detected instance.

The pipeline consists of:

1. Backbone: A convolutional network such as ResNet extracts feature maps.
2. Feature Pyramid Network: Multi-scale feature maps are constructed to detect objects of different sizes.
3. Region Proposal Network: Candidate object regions are generated with objectness scores and approximate bounding boxes.
4. RoIAlign: Fixed-size, spatially aligned features are extracted for each proposal.
5. Classification branch: Predicts the class of each region.
6. Bounding-box branch: Refines the proposal coordinates.
7. Mask branch: A small fully convolutional network predicts a binary mask for each class.

During inference, low-confidence detections are removed, non-maximum suppression reduces duplicate boxes, and the mask corresponding to the predicted class is selected. The selected mask is resized and placed within the final bounding box to obtain the instance-level segmentation output.

RoIPool converts region coordinates to discrete feature-map positions using quantization. This rounding can misalign extracted features with the original object, which is especially harmful for pixel-accurate mask prediction.

RoIAlign operation:

• Proposal coordinates remain continuous rather than being rounded.
• Each proposal is divided into a fixed number of bins.
• Sampling points are selected inside every bin.
• Feature values at fractional coordinates are calculated using bilinear interpolation.
• Sampled values are aggregated to form a fixed-size feature map.

Bilinear interpolation can be expressed as:

where the four neighboring feature values are weighted according to their distances from .

Effect:

• Preserves spatial correspondence between the proposal and feature map.
• Produces more accurate object boundaries.
• Improves mask quality, particularly for small or thin objects.

Thus, RoIAlign removes the coordinate quantization errors introduced by RoIPool.

Mask R-CNN is trained using a combined multi-task objective:

The components are:

• Classification loss : Measures whether each proposed region is assigned the correct object class, usually using cross-entropy loss.
• Bounding-box loss : Measures the error in predicted box coordinates, commonly using Smooth loss or a related regression loss.
• Mask loss : Measures pixel-wise binary classification error between the predicted mask and ground-truth mask.

The mask branch predicts one mask for each possible class. During training, the mask loss is computed only for the channel corresponding to the proposal's ground-truth class. This avoids competition among classes within the mask branch.

The combined loss allows shared backbone features to support object recognition, localization, and precise instance-mask prediction simultaneously.

Self-attention allows every token in a sequence to aggregate information from other tokens in the same sequence according to learned relevance scores.

Given an input matrix , query, key, and value matrices are computed as:

The similarity between each query and all keys is obtained through dot products:

To control the magnitude of these values, the scores are divided by , where is the key dimension. Softmax converts them into normalized attention weights:

The final output is:

The scaling factor prevents large dot products from pushing softmax into saturated regions with very small gradients. In vision, tokens usually represent image patches or local windows, so self-attention models relationships among spatial regions.

Multi-head attention performs several attention operations in parallel using different learned projections of the same input.

For head :

The outputs are concatenated and projected:

Advantages of multiple heads:

• Different heads can learn different spatial or semantic relationships.
• One head may focus on local edges, while another captures distant object parts.
• Multiple representation subspaces improve model expressiveness.
• Long-range and short-range dependencies can be modeled simultaneously.
• The total embedding dimension can be distributed across heads, keeping computation manageable.

A single attention head produces only one attention pattern for each token. Multi-head attention provides several complementary patterns, allowing the model to form a richer contextual representation.

Self-attention alone is largely insensitive to token order because it processes tokens as a set. A vision transformer must therefore receive positional information to distinguish patches located at different image coordinates.

Sinusoidal positional encoding:

For position and dimension index :

It is deterministic, introduces smooth positional patterns, and may generalize to sequence lengths not seen during training.

Learned positional encoding:

• Assigns a trainable vector to every token position.
• The vectors are optimized with the rest of the model.
• It can adapt specifically to the training data.
• It may require interpolation when the input resolution changes.

Vision models may also use two-dimensional or relative positional encodings. Relative encoding represents the displacement between patches and is particularly useful in architectures such as the Swin Transformer.

A Vision Transformer (ViT) treats an image as a sequence of fixed-size patches.

For an image of size and square patch size , the number of patches is:

Each patch contains values. The processing steps are:

1. Divide the image into non-overlapping patches.
2. Flatten each patch into a vector.
3. Project each vector into a -dimensional embedding using a learned linear layer.
4. Prepend a learnable classification token, commonly called the token.
5. Add positional embeddings to preserve patch-location information.
6. Pass the sequence through transformer encoder blocks containing multi-head self-attention, feed-forward layers, normalization, and residual connections.
7. Feed the final representation to a classification head.

For dense prediction tasks, patch representations can instead be reshaped into spatial feature maps and processed by a segmentation decoder.

Convolutional neural networks and Vision Transformers learn visual representations differently.

Inductive bias:

• CNNs include locality and translation-equivariance through convolution and weight sharing.
• ViTs have weaker built-in visual assumptions and learn relationships mainly from data.

Receptive field:

• CNN receptive fields grow gradually with network depth.
• Global self-attention can connect all patch tokens in a single layer.

Data requirements:

• CNNs often perform well on smaller datasets because of their visual inductive biases.
• Standard ViTs generally benefit from large-scale pretraining or strong data augmentation.

Computation:

• Convolution cost depends mainly on kernel size, channels, and spatial resolution.
• Global attention has approximately complexity in the number of tokens .

Representation:

• CNNs naturally build hierarchical local features.
• Original ViT architectures maintain a relatively uniform token resolution.

Modern systems often combine convolutional and transformer ideas to obtain both efficient local processing and strong long-range modeling.

The Swin Transformer is a hierarchical vision transformer designed to produce multi-scale feature maps efficiently.

Its major stages are:

• Patch partitioning: The input image is divided into small non-overlapping patches.
• Linear embedding: Patch pixels are projected into feature vectors.
• Swin Transformer blocks: Self-attention is computed within local windows rather than across the entire image.
• Patch merging: Neighboring patches are combined, reducing spatial resolution and increasing channel dimension.
• Repeated stages: Successive stages construct a feature hierarchy.

The hierarchy has properties similar to a CNN feature pyramid:

• Early stages contain high-resolution local details.
• Deeper stages contain lower-resolution semantic features.
• Multi-scale outputs can be connected to detection and segmentation heads.

This design makes the Swin Transformer suitable as a general-purpose backbone for classification, object detection, semantic segmentation, and instance segmentation while avoiding the full cost of global attention at high resolution.

Window-based multi-head self-attention (W-MSA) partitions the feature map into non-overlapping windows and computes attention independently inside each window.

Its benefits are:

• Lower computational cost than global self-attention.
• Efficient processing of high-resolution feature maps.
• Strong modeling of local relationships.

However, fixed windows prevent direct communication between tokens located in different windows. The next block therefore uses shifted-window multi-head self-attention (SW-MSA):

• The window partition is shifted by a fraction of the window size.
• Tokens that were in separate windows can now appear in the same window.
• An attention mask prevents invalid interactions caused by cyclic shifting.

Alternating W-MSA and SW-MSA allows information to propagate across window boundaries while retaining the efficiency of local attention. Without shifted windows, separate regions could remain poorly connected; without local windows, the computational cost would grow significantly for large images.

Consider a feature map containing tokens, each with embedding dimension .

For global self-attention, every token interacts with every other token. The attention matrix contains entries, giving an attention-related complexity of approximately:

This becomes expensive for high-resolution images.

For window-based self-attention, assume each window contains tokens. There are approximately windows. Attention within one window costs , so the total attention cost is:

When is fixed, this cost grows approximately linearly with the number of image tokens.

Therefore:

• Global attention provides immediate long-range interactions but has quadratic spatial cost.
• Window attention is substantially more efficient but initially restricts interactions to local regions.
• Shifted windows restore cross-window communication across successive layers.

Intersection over Union (IoU):

For predicted pixel set and ground-truth set :

Mean IoU averages the class-wise IoU values and is widely used for semantic segmentation.

Dice coefficient:

Dice emphasizes overlap and is common in medical segmentation, particularly when foreground and background classes are imbalanced. Its relation to IoU is:

Pixel accuracy:

• Measures the fraction of correctly classified pixels.
• Can be misleading when the background class dominates.

Average Precision (AP):

• Commonly used for instance segmentation.
• Matches predicted instances to ground-truth instances using an IoU threshold.
• Computes precision-recall performance across confidence levels.
• COCO-style mask AP averages results across multiple IoU thresholds, commonly from to .

Thus, semantic segmentation focuses on class-wise pixel overlap, while instance segmentation metrics must also evaluate object detection and instance-mask quality.