Unit 3: Transfer Learning and Fine-Tuning for Vision Tasks - Subjective Questions

Transfer learning is the process of using knowledge learned by a model on one task or dataset to solve a different but related task. In computer vision, a convolutional neural network (CNN) trained on a large dataset such as ImageNet can be adapted to a target vision problem.

Its usefulness with small datasets follows from these points:

• Early layers learn general features such as edges, corners, textures, and simple shapes.
• Later layers learn more task-specific structures and object representations.
• Reusing pretrained features reduces the amount of labeled target data required.
• It speeds up training and usually provides better generalization than training from scratch.
• It reduces the risk of overfitting because the model begins with useful visual representations.

A pretrained network can be used either as a fixed feature extractor or as the starting point for fine-tuning.

The two main transfer learning strategies are:

• Feature extraction: The convolutional base of a pretrained model is kept frozen. Its output is used as a feature representation, and a new classification or regression head is trained for the target task. This approach is computationally efficient and suitable when the target dataset is small or similar to the source dataset.
• Fine-tuning: Some or all layers of the pretrained model are unfrozen and trained together with the new task-specific head. A small learning rate is generally used so that useful pretrained features are not destroyed.

Feature extraction has lower computational cost and fewer trainable parameters. Fine-tuning is more adaptable and can provide better performance when the target domain differs from the source domain or when sufficient target data is available.

ResNet, or Residual Network, uses residual blocks to make very deep neural networks easier to train. Instead of directly learning a transformation , a residual block learns a residual function:

The block then produces:

Here, is passed through a shortcut or skip connection and added to the output of the convolutional layers.

Major advantages include:

• Skip connections reduce the vanishing-gradient problem.
• Very deep models such as ResNet-50 and ResNet-101 can be trained effectively.
• The identity mapping allows useful information to flow directly through the network.
• Pretrained ResNet models provide strong general-purpose visual features.
• The architecture is widely supported and can be adapted by replacing its final classification layer.

These properties make ResNet a reliable baseline for image classification, detection, and segmentation tasks.

Consider a residual block whose output is:

where is the learned residual transformation and is the shortcut input. For a loss function , the chain rule gives:

where is the identity matrix associated with the shortcut connection.

The important term is . Even if becomes very small, the gradient still contains a direct identity contribution:

Therefore, the gradient can flow directly from deeper layers to shallower layers. This reduces gradient attenuation caused by repeatedly multiplying small derivatives. The residual block does not require every layer to learn a complete transformation; it can learn a small correction to the identity mapping. This makes optimization easier and enables the successful training of very deep networks.

EfficientNet improves the trade-off between accuracy and computational cost through compound scaling. Instead of increasing only depth, width, or input resolution, it scales all three dimensions in a balanced manner.

The three dimensions are:

• Depth: The number of layers in the network.
• Width: The number of channels in each layer.
• Resolution: The spatial size of the input image.

EfficientNet uses a compound scaling rule such as:

subject to a constraint of the form:

where controls the overall model size and , , and are scaling coefficients.

It is effective for transfer learning because it provides strong accuracy with fewer parameters and lower computational cost. Its pretrained representations can be adapted to target tasks while requiring less memory and training time than many similarly accurate models.

ResNet is often selected for its simplicity, robustness, and broad ecosystem. EfficientNet is useful when accuracy and efficiency must be balanced. MobileNet is preferable when latency, memory, and energy consumption are strict constraints. The best choice depends on the target dataset, hardware, inference-time requirement, and acceptable accuracy.

A standard convolution applies a filter across all input channels to produce output channels. If the input has channels, the output has channels, and the feature map has spatial size , its approximate computation is:

MobileNet separates this operation into two stages:

• Depthwise convolution: Applies one filter independently to each input channel. Its cost is:

• Pointwise convolution: Uses convolutions to combine the channels. Its cost is:

The total cost is:

Relative to standard convolution, the approximate cost ratio is:

This significantly reduces computation and parameters while preserving useful spatial and channel information, making MobileNet suitable for resource-constrained devices.

A typical feature extraction procedure is:

• Select a pretrained CNN such as ResNet, EfficientNet, or MobileNet.
• Remove its original task-specific classification head.
• Add a new head suitable for the target number of classes.
• Freeze the convolutional base by setting its layers as non-trainable.
• Preprocess target images using the input normalization expected by the pretrained model.
• Train only the new classification head using the target dataset.
• Evaluate the model using validation and test data.

The pretrained base computes a feature vector for an image :

The new head then predicts the class probabilities:

Only the parameters of the new head are updated. This approach is fast, uses limited memory, and is less likely to overfit on a small dataset.

Fine-tuning is the process of unfreezing selected layers of a pretrained model and updating their parameters on the target dataset. It should be preferred when:

• The target dataset is sufficiently large to support additional trainable parameters.
• The target images differ substantially from the source dataset.
• Feature extraction produces inadequate validation performance.
• The target task requires specialized visual patterns not represented well by the pretrained features.
• The model needs to adapt to differences in image style, resolution, lighting, or object appearance.

Fine-tuning is usually performed after first training the new head with the base frozen. The learning rate should be small because the pretrained weights already contain useful information. Unfreezing too many layers too early can cause overfitting or catastrophic forgetting.

Layer freezing is commonly managed in stages:

• Initially freeze all layers of the pretrained backbone.
• Replace and train the task-specific head until it learns a reasonable mapping from features to labels.
• Unfreeze the final block or a selected group of deeper layers.
• Recompile the model so the optimizer recognizes the newly trainable parameters.
• Continue training with a much smaller learning rate.
• Monitor validation loss and use early stopping to avoid overfitting.
• Unfreeze additional layers only if the target domain requires more adaptation.

Early layers generally learn generic features, while deeper layers learn source-domain-specific features. Therefore, deeper layers are usually unfrozen first. Batch normalization layers require special care because updating their moving statistics on a small dataset can destabilize training. The freezing schedule should be determined using validation performance rather than a fixed rule alone.

Layer-wise learning rates assign different learning rates to different groups of layers. A typical arrangement is:

• Very small learning rates for early layers.
• Moderate learning rates for middle layers.
• Larger learning rates for deeper pretrained layers.
• The largest learning rate for a newly initialized task-specific head.

If layer has learning rate , an update can be written as:

Early layers contain general edge and texture detectors, so large updates may destroy useful representations. Deeper layers are more task-specific and may need greater adaptation. The new head has no pretrained knowledge and therefore usually requires the largest learning rate. Layer-wise learning rates improve stability, reduce catastrophic forgetting, and allow the model to adapt at different speeds across its hierarchy.

A possible schedule is:

• Freeze the entire ResNet backbone and train the new classification head with learning rate .
• Unfreeze the final residual block and train it with learning rate , while keeping the earlier backbone layers frozen.
• If validation performance improves, unfreeze the preceding residual block and use for earlier blocks.
• Keep the new head at approximately or , depending on validation stability.
• Use early stopping, weight decay, and data augmentation to control overfitting.

For layer group , the update is:

The head receives a larger rate because it is randomly initialized. The final residual block receives a small rate because it must adapt to medical image patterns without losing useful pretrained features. Earlier blocks receive the smallest rate because their low-level representations are likely to remain useful. The exact values should be tuned using validation data.

The source domain is the dataset and distribution used to train the original pretrained model. The target domain is the dataset and distribution associated with the new task.

A domain shift occurs when the source and target distributions differ. If represents the source distribution and represents the target distribution, then domain shift exists when:

Examples include differences in:

• Image color, illumination, or resolution.
• Object scale, viewpoint, or background.
• Class frequencies and label definitions.
• Sensor type or acquisition conditions.
• Visual styles, such as natural photographs versus X-ray images.

A model trained only on the source domain may produce poor target-domain predictions because its learned features and decision boundaries do not match the new distribution. Fine-tuning, domain-specific preprocessing, data augmentation, and domain adaptation methods can reduce this performance degradation.

Domain adaptation adjusts a model trained on a source domain so that it performs well on a different target domain.

• Supervised domain adaptation: Uses labeled examples from both the source and target domains. The target labels directly guide model adaptation and usually provide the strongest supervision.
• Semi-supervised domain adaptation: Uses a small number of labeled target examples together with many unlabeled target examples. It combines supervised target loss with methods such as pseudo-labeling or consistency regularization.
• Unsupervised domain adaptation: Uses labeled source data and unlabeled target data. The model attempts to learn domain-invariant features by reducing the discrepancy between source and target feature distributions.

A common objective can be written as:

where preserves task performance and encourages source and target representations to become similar. The parameter controls the contribution of domain alignment.

The choice depends on the relationship between the source and target datasets:

• High similarity and small target dataset: Use feature extraction first because pretrained features are likely to transfer well and fine-tuning may overfit.
• High similarity and large target dataset: Fine-tune deeper layers to obtain additional task-specific adaptation.
• Low similarity and small target dataset: Use strong regularization and gradually fine-tune only selected deeper layers. Full fine-tuning may be unstable.
• Low similarity and large target dataset: Fine-tuning many layers or training a model from scratch may be appropriate.
• Noisy or imbalanced data: Improve data quality, use class-aware sampling, and apply suitable augmentation before increasing model complexity.

Dataset quality often matters more than dataset size alone. Incorrect labels, leakage, duplicate images, and inconsistent preprocessing can cause misleading validation results and poor transfer performance.

Preprocessing should be consistent with the assumptions used during pretraining. Important steps include:

• Resize or crop images to the model's expected spatial dimensions.
• Convert image values to the expected numerical range.
• Apply the model-specific channel normalization using the appropriate mean and standard deviation.
• Preserve the correct channel order, such as RGB.
• Apply the same deterministic preprocessing to validation and test images.

Training augmentation may include random crops, horizontal flips, rotations, color changes, or affine transformations. Augmentations should preserve the target label and reflect realistic variations in the application domain. Excessive augmentation can remove important diagnostic or semantic information.

For a target image , an augmentation function produces , where is sampled from a set of valid transformations. Proper preprocessing reduces distribution mismatch, while augmentation improves generalization and reduces overfitting during fine-tuning.

Overfitting occurs when the model performs well on training data but poorly on unseen validation or test data. Catastrophic forgetting occurs when aggressive updates cause the model to lose useful representations learned from the source task.

Common signs include:

• Training loss continues to decrease while validation loss increases.
• Training accuracy becomes much higher than validation accuracy.
• Performance drops sharply after unfreezing many layers.
• Predictions become unstable or overly specialized to the target training set.

Control methods include:

• Use a smaller learning rate for pretrained layers.
• Unfreeze layers gradually.
• Keep early layers frozen when possible.
• Apply weight decay, dropout, and early stopping.
• Use data augmentation and class-balanced sampling.
• Monitor both source-related and target-domain validation behavior when source data is available.
• Reinitialize or replace only the task-specific head instead of unnecessarily changing the backbone.

These methods preserve transferable representations while allowing controlled adaptation.

Batch normalization transforms activations using batch statistics and learned parameters. For a batch activation , it is commonly expressed as:

During transfer learning, batch normalization needs special treatment because a small target batch may produce unreliable estimates of and .

Recommended practices include:

• Keep batch normalization layers in inference mode when the target dataset is small.
• Freeze their learned scale and shift parameters if updating them causes instability.
• Avoid changing moving averages using very small batches.
• If the target dataset is large and substantially different, allow careful adaptation of batch normalization statistics.
• Ensure that training and evaluation modes are correctly configured.

Incorrect handling can cause a mismatch between training and inference behavior, unstable fine-tuning, or degradation of pretrained features.

A complete EfficientNet transfer learning pipeline can be organized as follows:

• Collect and inspect labeled target images.
• Remove duplicates, correct invalid labels, and divide the data into training, validation, and test subsets without leakage.
• Apply EfficientNet-compatible resizing and normalization.
• Add realistic training augmentation.
• Load an ImageNet-pretrained EfficientNet backbone without its original classification head.
• Add global average pooling, optional dropout, and a new dense output layer with the required number of classes.
• Freeze the backbone and train the new head.
• Evaluate validation loss, accuracy, precision, recall, and class-wise performance.
• Unfreeze the final EfficientNet blocks and fine-tune using a small learning rate.
• Use early stopping and save the best validation checkpoint.
• Evaluate once on the held-out test set.

For a -class problem, the final softmax probability for class can be written as:

This pipeline combines efficient pretrained representations with controlled target-domain adaptation.

Flattening converts a feature tensor of shape into a vector of length . A dense layer following flattening can therefore contain many parameters:

where is the number of units in the dense layer.

Global average pooling (GAP) averages each feature map over its spatial dimensions and produces a vector of length :

Advantages of GAP include:

• Far fewer trainable parameters.
• Lower risk of overfitting on small datasets.
• Greater tolerance to spatial translations.
• A direct connection between feature maps and semantic channels.

Flattening may preserve more spatial information but can create a very large classification head. GAP is usually preferred for transfer learning unless the target task requires detailed spatial relationships.