Unit 4: Object Detection and Localization - Subjective Questions

Object detection is a computer vision task that identifies the objects present in an image and determines their locations. It performs two related tasks:

• Classification: Assigning a class label to each detected object.
• Localization: Predicting the spatial position of each object using a bounding box.

Image classification assigns one or more labels to an entire image, while object detection produces multiple labeled regions. Image segmentation assigns a class label to individual pixels. Therefore, detection provides more spatial information than classification but less precise shape information than segmentation.

A detector usually learns a function that maps an image to a set of detections:

where is a bounding box, is a class label, and is the confidence score.

A bounding box is commonly represented in either of the following forms:

• Corner representation: .
• Center representation: , where is the center, is the width, and is the height.

The conversion from corner coordinates to center coordinates is:

The reverse conversion is:

Center-based coordinates are frequently used by detectors such as YOLO, whereas corner coordinates are convenient for calculating overlap.

Intersection over Union measures the overlap between a predicted bounding box and a ground-truth bounding box. It is defined as:

For predicted box and ground-truth box :

The union area is calculated as:

IoU ranges from to . A value of indicates no overlap, while indicates identical boxes. During training, IoU can help assign positive and negative samples and can be included in localization losses. During evaluation, a detection is usually considered correct when its class is correct and its IoU with a ground-truth box exceeds a chosen threshold such as .

Non-maximum Suppression, or NMS, removes duplicate detections of the same object. A detector may produce several overlapping boxes because multiple anchors or locations respond to one object.

The procedure is:

1. Filter out boxes whose confidence scores are below a predefined score threshold.
2. Select the remaining box with the highest confidence score.
3. Add this box to the final detection set.
4. Compute its IoU with every remaining box.
5. Suppress boxes whose IoU exceeds the NMS threshold.
6. Repeat the process until no boxes remain.

For example, if two boxes have an IoU of and one has a higher confidence score, the lower-scoring box is usually removed. NMS is necessary because duplicate predictions would reduce precision and cause one object to be counted multiple times. Its main limitation is that it can suppress valid neighboring objects when they overlap heavily.

A classical detection pipeline generally consists of the following stages:

• Image preprocessing: Resize, normalize, or construct an image pyramid to handle different object scales.
• Window generation: Slide a fixed-size window across the image at multiple locations and scales.
• Feature extraction: Compute hand-crafted features such as Haar features, Histogram of Oriented Gradients, or local binary patterns.
• Classification: Apply a classifier such as a support vector machine or AdaBoost to determine whether the window contains an object.
• Post-processing: Combine overlapping detections using methods such as NMS.

The main advantage is that the pipeline is interpretable and can work with limited training data. However, it is computationally expensive because many windows must be evaluated. Hand-crafted features are also less robust to changes in pose, illumination, occlusion, and background. Deep learning detectors improve performance by learning features and detection behavior jointly from data.

R-CNN, or Regions with Convolutional Neural Network features, is a two-stage detection method.

Its operation is:

1. Generate approximately 2,000 region proposals using selective search.
2. Warp each proposal to a fixed image size.
3. Pass every proposal through a convolutional neural network to extract features.
4. Classify the extracted features using class-specific support vector machines.
5. Refine the proposal coordinates using bounding-box regression.
6. Apply NMS to remove duplicate detections.

It is called a two-stage detector because it first generates candidate regions and then classifies and refines those regions. R-CNN achieved a major improvement over classical pipelines because it learned powerful visual features. However, it was slow and memory-intensive because the CNN had to process each proposal separately. This limitation motivated improvements such as Fast R-CNN and Faster R-CNN.

The three methods improve progressively in efficiency:

R-CNN extracts features independently for each region, leading to repeated computation. Fast R-CNN computes a single feature map and uses RoI pooling to extract proposal features, making it substantially faster. Faster R-CNN replaces selective search with a trainable Region Proposal Network that predicts object-like regions directly from the feature map.

All three are two-stage detectors and generally provide strong localization accuracy. Faster R-CNN is more efficient than its predecessors but is usually slower than modern single-stage detectors.

Single-stage detectors predict object classes and bounding-box coordinates directly from an image or its feature maps. They do not use a separate region proposal stage. Examples include YOLO, SSD, and RetinaNet.

Differences from two-stage detectors:

• Two-stage detectors first generate region proposals and then classify and refine them.
• Single-stage detectors perform classification and localization in one integrated prediction step.
• Single-stage methods generally have lower latency and simpler inference pipelines.

Advantages:

• High inference speed.
• Suitable for real-time applications.
• Simpler deployment and end-to-end processing.
• Efficient use of shared convolutional features.

Disadvantages:

• Earlier versions struggled with small objects and crowded scenes.
• Extreme class imbalance can make training difficult.
• Localization accuracy may be lower than that of some two-stage models.

Architectural improvements, feature pyramids, anchor design, and focal loss have significantly reduced these disadvantages.

YOLO, meaning You Only Look Once, treats object detection as a single regression problem. The entire image is processed by one neural network in a single forward pass.

In the original formulation, the image is divided into an grid. Each grid cell predicts:

• Bounding-box coordinates.
• Objectness confidence.
• Class probabilities.

For a box prediction, the output may be written as:

where represents the box center relative to the cell, represents its dimensions, and is the confidence score. The final class-specific confidence can be expressed as:

Predictions are filtered by confidence and duplicate boxes are removed using NMS. YOLO is fast because the image is examined globally and the network performs detection in one pass.

The YOLO family evolved through several important improvements:

• YOLOv1: Introduced single-pass grid-based detection. It was very fast but had difficulty with small objects, multiple objects in one grid cell, and precise localization.
• YOLOv2: Added batch normalization, anchor boxes, higher-resolution training, dimension clustering, and a stronger backbone. These changes improved recall and localization.
• YOLOv3: Used a deeper Darknet-53 backbone and predictions at multiple scales. It also improved objectness and class prediction, which helped small-object detection.
• Later YOLO variants: Added improved feature pyramids, better label assignment, stronger data augmentation, advanced loss functions, anchor-free options in some versions, and more efficient backbone designs.

The general trend was to improve the balance among speed, accuracy, and robustness. Multi-scale prediction and better feature extraction were especially important for detecting objects with different sizes.

SSD is a single-stage detector that predicts object classes and bounding-box offsets from multiple feature maps of different spatial resolutions.

Its main components are:

• A base convolutional network for extracting image features.
• Additional convolutional layers that progressively reduce spatial resolution.
• Default boxes, also called anchor boxes, associated with each feature-map location.
• Separate prediction layers for class scores and box coordinate offsets.

High-resolution feature maps contain detailed spatial information and are useful for small objects. Lower-resolution maps have larger receptive fields and are useful for large objects. For each default box, SSD predicts offsets such as:

along with class probabilities and a confidence score. All predictions are generated in one forward pass, after which confidence filtering and NMS are applied. SSD is efficient, although its performance can decrease for very small objects when feature maps do not preserve sufficient detail.

Anchor boxes are predefined bounding boxes with different sizes and aspect ratios. They provide reference shapes from which a detector predicts offsets to match ground-truth objects.

At each feature-map location, several anchors may be placed. For an anchor with center , width , and height , a detector can predict transformed coordinates such as:

During training, anchors are matched to ground-truth boxes using IoU thresholds. Matched anchors become positive examples, while poorly overlapping anchors become negative examples or are ignored. Anchors allow a detector to handle objects with different shapes, but they introduce hyperparameters and can make label assignment and prediction output more complex.

RetinaNet is a single-stage detector designed to achieve high accuracy while handling the severe class imbalance produced by dense anchors. It commonly uses a backbone network, a Feature Pyramid Network, and two subnetworks: one for classification and one for box regression.

For binary classification, let be the predicted probability of the true class. The standard cross-entropy loss is:

Focal loss adds a modulating factor that reduces the contribution of easy examples:

Here, is the probability assigned to the true class, balances positive and negative examples, and controls the strength of focusing. When an example is easy, is close to , so is small. Hard examples retain a larger loss. This prevents the large number of easy negative anchors from dominating training.

An object detector usually optimizes several loss components:

• Classification loss: Measures whether the predicted class is correct. Cross-entropy or focal loss is commonly used.
• Confidence or objectness loss: Measures whether a predicted box contains an object. It distinguishes foreground boxes from background boxes.
• Localization loss: Measures the difference between the predicted box and the ground-truth box. Coordinate losses, smooth loss, IoU loss, GIoU loss, or CIoU loss may be used.

A general objective can be written as:

where the values control the relative importance of each component. Classification and objectness determine what the box represents, while localization determines how accurately it is positioned and sized. Balancing these terms is important because their numerical scales and effects on training can differ.

For a chosen class and IoU threshold:

• Precision is the fraction of predicted positives that are correct:

• Recall is the fraction of ground-truth objects that are detected:

A prediction is a true positive only when its class is correct and its IoU with an unmatched ground-truth box satisfies the evaluation threshold.

Average Precision (AP) summarizes the precision-recall curve for one class, typically by integrating or numerically approximating precision over recall. Mean Average Precision (mAP) averages AP across all classes:

Some benchmarks report mAP at one IoU threshold, such as , while others average AP over multiple thresholds, such as .

To construct a precision-recall curve for one class:

1. Collect all predicted boxes for that class across the evaluation dataset.
2. Sort predictions in descending order of confidence.
3. Match each prediction to an unmatched ground-truth box using the selected IoU threshold.
4. Label each prediction as a true positive or false positive.
5. At every rank, compute cumulative precision and recall.
6. Plot precision against recall.

The resulting curve shows the trade-off between detecting more objects and introducing more false positives. Average Precision is the area under this curve:

In practical implementations, the integral is approximated using interpolation or a finite set of recall points. A detector with both high precision and high recall produces a larger AP. AP is computed separately for every class, and their average gives mAP.

An IoU threshold determines how much overlap is required between a predicted box and a ground-truth box.

During training, an anchor or proposal may be:

• Marked positive if its IoU is above a positive threshold.
• Marked negative if its IoU is below a negative threshold.
• Ignored when its IoU lies between the two thresholds.

During evaluation, a prediction is normally a true positive only if its class is correct and its IoU exceeds the chosen threshold. A lower threshold, such as , is more tolerant of localization errors and often produces higher recall. A higher threshold demands more accurate localization and usually lowers recall.

Threshold choice therefore affects reported AP and mAP. Reporting performance at multiple thresholds gives a more complete picture because a detector may identify the correct objects but still produce imprecise boxes.

The predicted box has coordinates , and the ground-truth box has coordinates .

The intersection coordinates are:

Thus, the intersection width and height are and , so:

The predicted-box area is:

The ground-truth area is:

Therefore, the union area is:

The IoU is:

Since , the prediction is not a true positive at the specified threshold, even if its class label is correct.

Two-stage detectors, such as Faster R-CNN, first generate region proposals and then classify and refine them. One-stage detectors, such as YOLO, SSD, and RetinaNet, directly predict classes and boxes from dense feature locations.

Two-stage methods may be preferred when accuracy is more important than latency. One-stage methods are preferred when rapid inference and low deployment complexity are essential. The distinction is no longer absolute because advanced one-stage models can achieve accuracy close to two-stage systems.

Small objects occupy only a few pixels, so their visual features can be lost as the image passes through successive downsampling layers. They are also more affected by localization errors, occlusion, and background clutter.

Multi-scale representations address this problem by combining features with different resolutions:

• High-resolution feature maps preserve fine spatial details and help detect small objects.
• Low-resolution feature maps provide larger receptive fields and help detect large objects.
• Feature Pyramid Networks combine semantic information from deep layers with spatial detail from shallow layers.

SSD predicts from several progressively smaller feature maps. YOLO variants use multi-scale detection heads. RetinaNet uses an FPN to generate pyramid levels and predicts at each level. This arrangement assigns objects of different sizes to appropriate feature-map scales, improving recall and localization across object sizes.