Unit 5: Image Segmentation and Advanced Vision Architectures

I. Foundations of Dense Prediction and Modern Vision Models

Image segmentation assigns labels to individual pixels, while advanced vision architectures learn spatial relationships through convolution, attention, or combinations of both. These methods support scene understanding, medical imaging, autonomous navigation, remote sensing, and image editing.

• Dense prediction: Unlike image classification, which produces one label per image, segmentation produces an output associated with every pixel.
• Spatial resolution: Downsampling captures high-level context but removes detail; successful segmentation models recover boundaries through upsampling, skip connections, dilation, or multiscale fusion.
• Feature representation: CNNs encode locality and translation equivariance, whereas transformers model relationships between potentially distant image regions.
• Training target: Ground-truth masks may describe semantic classes, distinct object instances, or both.
• Evaluation conventions:
  • Intersection over Union: (\mathrm{IoU}=|P\cap G|/|P\cup G|), where (P) is the predicted pixel set and (G) is the ground-truth set.
  • Dice coefficient: (\mathrm{Dice}=2|P\cap G|/(|P|+|G|)).
  • Mean IoU: Class-wise IoU averaged across all classes.

II. Semantic Segmentation — Pixel-Level Class Prediction

Semantic segmentation classifies every pixel according to a fixed set of semantic categories.

A. Semantic segmentation

The central purpose is to produce a dense class map without distinguishing separate objects belonging to the same class.

• Output structure: For an image of size (H\times W) and (C) classes, the model predicts logits (Z\in\mathbb{R}^{H\times W\times C}).
• Pixel classification: A softmax converts logits into class probabilities:

p(i,j,c) = exp(Z(i,j,c)) / sum_k exp(Z(i,j,k))

Here, (p(i,j,c)) is the probability that pixel ((i,j)) belongs to class (c), and (k) indexes all classes.

• Loss function: Pixel-wise cross-entropy is commonly used; Dice or focal loss helps when foreground classes occupy few pixels.
• Typical pipeline: An encoder reduces spatial size and extracts context, while a decoder upsamples features into a full-resolution mask.
• Limitation: Two adjacent cars receive the same “car” label and are not represented as distinct entities.

III. Instance-Level Recognition — Separating Individual Objects

Instance segmentation combines object detection with pixel-accurate foreground prediction.

A. Instance segmentation

Instance segmentation assigns both a semantic category and a unique identity to every detected object.

• Output components: Each instance has a class label, confidence score, bounding box, and binary mask.
• Semantic contrast:
  1. Semantic segmentation: Labels all person pixels as one class region.
  2. Instance segmentation: Produces a separate mask for each person.
• Evaluation: Mask Average Precision computes detection precision across IoU thresholds, commonly from (0.50) to (0.95).
• Challenges: Occlusion, overlapping objects, small instances, and ambiguous boundaries make instance separation difficult.
• Applications: Robotic manipulation requires individual object masks, while traffic analysis requires separate vehicle identities.

IV. U-Shaped Encoder–Decoder Networks

U-Net was introduced for biomedical image segmentation and is designed to combine contextual recognition with precise localization.

A. UNet

UNet uses a symmetric contracting path and expanding path connected by feature-preserving skip connections.

• Encoder: Repeated convolutions and pooling increase receptive field while reducing feature-map resolution.
• Bottleneck: The lowest-resolution representation captures high-level semantic context.
• Decoder: Upsampling restores spatial dimensions and combines coarse semantic information with fine details.
• Skip connections: Encoder features are concatenated with decoder features at corresponding resolutions:

D_l = Conv(Concat(Up(D_(l+1)), E_l))

Here, (El) is the encoder feature at level (l), (D{l+1}) is the deeper decoder feature, and (D_l) is the refined decoder output.

• Strength: Skip connections preserve edges that would otherwise be lost through pooling.
• Limitation: High-resolution feature concatenation increases memory use, especially for 3D medical volumes.

V. Multiscale Segmentation with Atrous Convolution

DeepLab is a family of semantic segmentation architectures that expands receptive fields without excessive loss of resolution.

A. DeepLab architectures

DeepLab architectures combine atrous convolution, multiscale context, and boundary refinement.

• Atrous convolution: A dilation rate (r) inserts spaces between kernel samples, enlarging the receptive field without adding parameters.
• ASPP: Atrous Spatial Pyramid Pooling applies parallel convolutions with different dilation rates to capture objects at multiple scales.
• DeepLab versions:
  • DeepLab v1/v2: Introduced atrous convolution and ASPP, with dense CRF refinement.
  • DeepLab v3: Improved ASPP using image-level context and batch normalization.
  • DeepLab v3+: Added an encoder–decoder structure for sharper boundaries.
• Backbones: ResNet and Xception commonly provide feature extraction.
• Limitation: Large dilation rates may sample sparse grid positions and create gridding artifacts.

VI. Region-Based Mask Prediction

Mask R-CNN extends a two-stage object detector by adding a parallel branch for binary mask prediction.

A. Instance segmentation using Mask R-CNN

Mask R-CNN detects candidate objects and predicts a class-specific mask for each retained region.

• Backbone and FPN: A CNN backbone extracts features, while a Feature Pyramid Network represents objects at several scales.
• Region proposals: The Region Proposal Network predicts candidate boxes and objectness scores.
• RoIAlign: Bilinear interpolation extracts aligned fixed-size features without coordinate rounding, preserving mask boundaries.
• Parallel heads: Each proposal is processed by classification, box-regression, and mask branches.
• Multitask objective:

L = L_cls + L_box + L_mask

Here, (L{\text{cls}}) is classification loss, (L{\text{box}}) is bounding-box regression loss, and (L_{\text{mask}}) is per-pixel binary cross-entropy.

• Limitation: Its two-stage proposal pipeline is accurate but computationally heavier than many one-stage models.

VII. Content-Dependent Feature Interaction

Attention dynamically weights information according to relationships between feature elements.

A. Self-attention

Self-attention allows each token to aggregate information from every token in the same input sequence.

• Projections: Input tokens (X) are transformed into queries (Q), keys (K), and values (V).
• Scaled dot-product attention:

Attention(Q,K,V) = softmax(QK^T / sqrt(d_k))V

Here, (d_k) is key dimensionality; scaling prevents large dot products from saturating softmax.

• Interpretation: Query–key similarity determines how strongly each value contributes to an output token.
• Vision role: A patch depicting a wheel can attend to distant vehicle patches, supporting global object recognition.
• Cost: For (N) tokens, full attention requires (O(N^2)) pairwise scores, making high-resolution images expensive.

VIII. Parallel Attention Subspaces

Multiple attention heads capture different types of relationships at the same network layer.

A. Multi-head attention

Multi-head attention runs several learned attention operations in parallel and combines their outputs.

• Head computation:

head_h = Attention(XW_h^Q, XW_h^K, XW_h^V)
MHA(X) = Concat(head_1,...,head_H)W^O

Here, (H) is the number of heads; (W_h^Q), (W_h^K), (W_h^V), and (W^O) are learned projection matrices.

• Representation benefit: One head may emphasize local texture, while another links distant but semantically related regions.
• Dimensional convention: With model width (d), each of (H) heads often uses width (d/H).
• Limitation: Different heads are not guaranteed to learn interpretable or distinct relationships.

IX. Encoding Spatial Order

Because attention alone is permutation-equivariant, transformer tokens require explicit information about their positions.

A. Positional encoding

Positional encoding injects spatial order into patch or pixel representations before attention is applied.

• Absolute encoding: A position vector is added to each token embedding; it may be learned or fixed.
• Sinusoidal encoding:

PE(pos,2i)   = sin(pos / 10000^(2i/d))
PE(pos,2i+1) = cos(pos / 10000^(2i/d))

Here, (pos) is token position, (i) indexes frequency pairs, and (d) is embedding dimension.

• Two-dimensional images: Row and column encodings can be combined to represent patch coordinates.
• Relative encoding: Attention scores receive a bias based on displacement between tokens, which is useful for local windows.
• Limitation: Learned absolute embeddings may require interpolation when inference resolution differs from training resolution.

X. Attention-Based Visual Representation

Vision transformers convert images into token sequences and process them with transformer blocks.

A. Transformer-based models

Transformer-based models alternate attention and feed-forward transformations with residual connections and normalization.

• Tokenization: Images may be represented as fixed-size patches, feature-map locations, or multiscale region tokens.
• Transformer block:

Y = X + MSA(LN(X))
Z = Y + MLP(LN(Y))

Here, (LN) is layer normalization, (MSA) is multi-head self-attention, and (MLP) is a feed-forward network.

• Advantages: Attention supports long-range dependencies and flexible interaction among image regions.
• Hybrid designs: CNN stems or convolutional decoders can supply locality and efficient dense prediction.
• Limitation: Transformers often need substantial training data, augmentation, or pretraining because they possess weaker image-specific inductive biases than CNNs.

XI. Vision Transformer

ViT applies the original transformer encoder architecture directly to sequences of image patches.

A. ViT

ViT represents each image patch as a token and uses global self-attention for image classification.

• Patch embedding: An image (x\in\mathbb{R}^{H\times W\times C}) is divided into (P\times P) patches, producing (N=HW/P^2) tokens.
• Projection: Each flattened patch of length (P^2C) is linearly mapped to the model dimension.
• Classification token: A learned [CLS] token is prepended; its final representation feeds the classifier.
• Global context: Every patch can attend directly to every other patch in each encoder layer.
• Concrete scale: A (224\times224) image with (16\times16) patches produces (14\times14=196) patch tokens.
• Limitation: Global (O(N^2)) attention becomes costly as patch size decreases or image resolution increases.

XII. Hierarchical Vision Transformers

Swin Transformer introduces local shifted-window attention and hierarchical feature maps suitable for dense vision tasks.

A. Swin Transformer

Swin Transformer restricts attention to windows and shifts those windows between successive blocks to enable cross-window communication.

• Window attention: Tokens attend within non-overlapping (M\times M) windows, reducing cost from global quadratic attention to approximately linear growth with image size for fixed (M).
• Shifted windows:
  1. Regular block: Attention operates inside fixed window boundaries.
  2. Shifted block: Windows move, commonly by (M/2), connecting tokens previously placed in different windows.
• Patch merging: Neighboring token groups are combined as depth increases, lowering resolution and increasing channel dimension.
• Hierarchical outputs: Multiscale features resemble CNN feature pyramids and integrate naturally with detection or segmentation decoders.
• Relative position bias: Learned displacement-dependent terms encode spatial relationships inside each window.
• Limitation: Windowing improves efficiency but provides indirect rather than immediate global interaction.