Unit 3: Transfer Learning and Fine-Tuning for Vision Tasks

I. Foundations — Reusing Visual Knowledge

Transfer learning reuses representations learned from a source task or dataset to improve learning on a related target task. In computer vision, models commonly begin with weights pretrained on ImageNet, whose large-scale object classification task teaches reusable visual features such as edges, textures, shapes, and object parts.

• Governing principle: Features learned from sufficiently large and diverse image collections often generalize beyond the original labels.
• Source domain: The dataset and task used for pretraining, such as ImageNet classification with 1,000 classes.
• Target domain: The new dataset and task, such as classifying retinal scans or detecting manufacturing defects.
• Backbone: The convolutional network that transforms an image into feature maps; examples include ResNet, EfficientNet, and MobileNet.
• Task-specific head: The final layers adapted to the target output, such as a classifier with (K) logits for (K) classes.
• Training modes:
  • Feature extraction: Freeze the backbone and train only the new head.
  • Fine-tuning: Update some or all pretrained backbone parameters.
• Central assumption: Transfer is most effective when source and target domains share useful visual structure.
• Main risk: Negative transfer occurs when unsuitable source representations hinder target performance.

II. Transfer Learning — Knowledge Reuse Across Tasks

A. Transfer learning

Transfer learning initializes a target model with knowledge obtained from a source task instead of learning every parameter from random initialization.

• Mathematical view: Let pretrained parameters be (\theta_s), target data be (D_t), and target loss be (L_t); training begins from (\theta_s):
  
  TEXT

    theta_t = argmin_theta L_t(theta; D_t), initialized with theta = theta_s

  
  Here, (\theta_t) denotes the adapted target parameters.
• Typical workflow: Select a pretrained backbone, replace its output head, train the head, and then selectively fine-tune deeper layers.
• Advantages: Pretraining reduces data requirements, accelerates convergence, and usually improves generalization on small datasets.
• Task compatibility: Image classification weights can support classification, detection, segmentation, pose estimation, and metric learning because early visual features are shared.
• Input requirements: Images must match the model’s expected channel ordering, normalization statistics, and minimum spatial dimensions.
• Negative transfer: A model pretrained on natural photographs may transfer poorly to highly specialized modalities unless its higher layers are adapted.
• Evaluation rule: Compare transferred models against a randomly initialized baseline using the same train-validation split and augmentation pipeline.

III. ResNet — Residual Learning for Deep Networks

A. ResNet

ResNet uses residual connections to make very deep convolutional networks easier to optimize.

• Residual block: Instead of directly learning a mapping (H(x)), a block learns a residual function (F(x)):
  
  TEXT

    y = F(x, W) + x

  
  Here, (x) is the input tensor, (W) represents learned weights, and (y) is the block output.
• Shortcut connection: The identity path allows information and gradients to bypass convolutional layers, reducing degradation as network depth increases.
• Dimension matching: When spatial size or channel count changes, a (1 \times 1) projection can replace the identity shortcut:
  
  TEXT

    y = F(x, W) + Ws*x

  
  (W_s) is the projection matrix or convolution.
• Common variants: ResNet-18 and ResNet-34 use basic blocks; ResNet-50, ResNet-101, and ResNet-152 use bottleneck blocks containing (1 \times 1), (3 \times 3), and (1 \times 1) convolutions.
• Transfer behavior: Early stages capture edges and textures, while later residual stages encode source-task-specific object structure.
• Practical use: ResNet-50 is a common accuracy-compute baseline, but its size may be unsuitable for strict mobile latency constraints.

IV. EfficientNet — Compound Model Scaling

A. EfficientNet

EfficientNet improves computational efficiency by scaling network depth, width, and input resolution together.

• Compound scaling: Given scaling coefficient (\phi), dimensions are expanded approximately as:
  
  TEXT

    depth = alpha^phi
    width = beta^phi
    resolution = gamma^phi

  
  Here, (\alpha), (\beta), and (\gamma) are constants selected under a computational budget.
• Compute constraint: Because convolutional cost grows roughly with depth, squared width, and squared resolution, scaling is chosen so that:
  
  TEXT

    alpha * beta^2 * gamma^2 ≈ 2

• Model family: EfficientNet-B0 is the baseline; B1 through B7 progressively increase capacity and resolution.
• Core block: Mobile inverted bottleneck convolution uses expansion, depthwise convolution, projection, and often squeeze-and-excitation channel attention.
• Transfer benefit: Strong pretrained accuracy per parameter makes EfficientNet useful when memory and training resources are limited.
• Input sensitivity: Larger variants expect higher image resolutions; using unnecessarily large inputs increases memory use without guaranteed target-domain gains.
• Limitation: Theoretical operation counts do not always predict hardware latency because depthwise operations may be poorly optimized on some devices.

V. MobileNet — Lightweight Vision Backbones

A. MobileNet

MobileNet reduces computation through factorized convolutions designed for mobile and embedded inference.

• Depthwise separable convolution: A standard convolution is divided into a spatial depthwise convolution per input channel and a (1 \times 1) pointwise convolution that mixes channels.
• Cost comparison: For kernel size (D_k), feature-map size (D_f), (M) input channels, and (N) output channels:
  
  TEXT

    Standard cost  = Dk^2 * M * N * Df^2
    Separable cost = Dk^2 * M * Df^2 + M * N * Df^2

• MobileNetV2: Inverted residual blocks expand narrow inputs, apply depthwise convolution, and project back to a linear bottleneck.
• MobileNetV3: Adds hardware-aware architecture search, squeeze-and-excitation modules, and efficient nonlinearities such as hard-swish.
• Deployment value: Small parameter counts and reduced multiply-accumulate operations support phones, cameras, and edge processors.
• Transfer trade-off: MobileNet may sacrifice some peak accuracy compared with larger ResNet or EfficientNet variants, but often provides better latency and memory usage.
• Deployment check: Measure real latency, memory, and energy on the target device; parameter count alone is insufficient.

VI. Feature Extraction — Training a New Prediction Head

A. Feature extraction

Feature extraction treats a pretrained backbone as a fixed transformation and learns only target-specific output layers.

• Representation pipeline: A backbone (f{\theta}) produces feature vector (z), and a trainable head (g{\phi}) predicts the target:
  
  TEXT

    z = f_theta(x)
    y_hat = g_phi(z)

  
  Here, (x) is an input image, (\theta) is frozen, (\phi) is trainable, and (\hat{y}) is the prediction.
• Head replacement: An ImageNet layer with 1,000 outputs is replaced by a layer with (K) outputs for the target classes.
• Optimization scope: Gradients may pass through the backbone computationally, but frozen parameters are excluded from optimizer updates.
• Best conditions: This method works well when the target dataset is small and visually similar to the pretraining data.
• Advantages: Training is fast, requires less memory, and reduces overfitting because relatively few parameters change.
• Limitations: Fixed high-level features may be unsuitable for domains such as radiographs, satellite imagery, or microscopy.
• Batch normalization: Frozen backbones are commonly placed in evaluation mode so running means and variances do not drift.

VII. Fine-Tuning — Adapting Pretrained Representations

A. Fine-tuning

Fine-tuning updates pretrained layers on target data so their representations become more task-specific.

• Starting procedure: First train a newly initialized head; then unfreeze selected backbone layers after the head reaches stable performance.
• Learning rate: Use a smaller rate for pretrained weights because large updates can destroy useful representations, a problem called catastrophic forgetting.
• Loss function: For multiclass classification, cross-entropy is commonly used:
  
  TEXT

    L = -sum(k=1 to K) y_k * log(p_k)

  
  (K) is the number of classes, (y_k) is the target indicator, and (p_k) is the predicted probability.
• Full fine-tuning: Updating the entire network offers maximum adaptability but increases memory use and overfitting risk.
• Partial fine-tuning: Updating only later stages preserves general low-level features while adapting semantic features.
• Regularization: Data augmentation, weight decay, dropout, early stopping, and label smoothing can control overfitting.
• Monitoring: Validation loss should guide checkpoint selection because training accuracy alone may conceal representation over-specialization.

VIII. Layer Control — Progressive Adaptation

A. Managing freezing and unfreezing of layers

Freezing controls which parameters remain fixed, while unfreezing determines how much pretrained knowledge can adapt.

1. Frozen phase: Set the backbone parameters as non-trainable and optimize only the new head.
2. Unfrozen phase: Enable gradients for later blocks, recreate or update the optimizer, and continue with a lower learning rate.

• Progressive unfreezing: Unfreeze the final stage first and move toward earlier stages only when validation performance justifies additional capacity.
• Layer hierarchy: Early layers detect generic edges and color contrasts; deeper layers encode task-dependent shapes and object parts.
• Optimizer state: Newly trainable parameters must be included in optimizer parameter groups; changing a framework flag alone may not update them.
• Normalization layers: Batch-normalization statistics and affine parameters can be frozen separately, especially with small batches where estimates are noisy.
• Decision evidence: A persistent train-validation underfit may justify more unfreezing, while rapid validation degradation suggests excessive adaptation.

IX. Differential Optimization — Controlling Update Magnitudes

A. Applying layer-wise learning rates

Layer-wise learning rates assign smaller updates to general early features and larger updates to task-specific later layers.

• Update rule: For layer (l), gradient descent applies:
  
  TEXT

    theta_l = theta_l - eta_l * gradient(L, theta_l)

  
  Here, (\theta_l) denotes layer parameters and (\eta_l) is that layer’s learning rate.
• Discriminative rates: A head might use (10^{-3}), late backbone blocks (10^{-4}), and early blocks (10^{-5}).
• Rationale: Randomly initialized heads require substantial learning, whereas pretrained early filters need only small corrections.
• Parameter groups: Optimizers can group parameters by stage, each with its own learning rate and possibly weight decay.
• Scheduling: Warm-up reduces instability at the start; cosine decay or step decay lowers rates as optimization converges.
• Caution: Excessive rate differences can prevent early layers from adapting, while large backbone rates can erase pretrained knowledge.
• Selection method: Tune rates using validation performance, gradient norms, and stability rather than relying only on fixed ratios.

X. Domain Shift — Adapting Data and Representations

A. Domain adaptation and dataset

Domain adaptation addresses differences between the source-data distribution and the target-data distribution.

• Domain shift: If source samples follow (P_s(X,Y)) and target samples follow (P_t(X,Y)), transfer becomes difficult when these distributions differ substantially.
• Dataset design: Target splits should be stratified where appropriate and separated by real-world unit, such as patient, location, device, or video, to prevent leakage.
• Preprocessing alignment: Resize and normalize inputs according to pretrained-model requirements, while preserving domain-relevant details such as lesion boundaries or fine defects.
• Target augmentation: Flips, crops, color transformations, blur, or noise should reflect plausible target conditions rather than create invalid examples.
• Class imbalance: Weighted losses, balanced sampling, or targeted augmentation can prevent majority classes from dominating optimization.
• Adaptation methods:
  • Supervised adaptation: Fine-tune using labeled target examples.
  • Unsupervised adaptation: Align source and target features using discrepancy losses, adversarial training, or pseudo-labels.
• Dataset bias: Backgrounds, acquisition devices, watermarks, or demographic imbalance can become unintended shortcuts.
• Evaluation: Report target-domain metrics on an untouched test set; accuracy may be supplemented by precision, recall, F1 score, area under the ROC curve, calibration, latency, or memory according to deployment needs.