Unit 4: Object Detection and Localization

I. Orientation — From Image Classification to Spatial Prediction

Object detection identifies what objects are present and where they occur in an image. Unlike image classification, which produces one label for an entire image, detection returns a variable-sized set of class labels, confidence scores, and spatial locations.

A. Object detection principles

Object detection combines recognition and localization within a single prediction task.

• Detection output: Each detected object is represented by a tuple such as ((c, s, b)), where (c) is the predicted class, (s) is the confidence score, and (b) is the bounding box.
• Localization: The model predicts the spatial extent of each object, usually as a rectangular box surrounding visible object pixels.
• Classification: Every proposed region is assigned an object class or the background class.
• Multiple instances: An image may contain zero, one, or many objects, including several instances of the same class.
• Variable output size: The number of detections is not fixed; confidence filtering and non-maximum suppression determine the final set.
• Training targets:
  • Classification loss: Penalizes incorrect class predictions, commonly using cross-entropy or focal loss.
  • Localization loss: Penalizes inaccurate box coordinates, using smooth (L_1), (L_1), generalized IoU, or related losses.
• Core difficulties: Scale variation, occlusion, clutter, illumination changes, unusual viewpoints, class imbalance, and overlapping objects all complicate detection.
• Speed–accuracy trade-off: Two-stage detectors generally emphasize accuracy, while single-stage detectors are designed for lower-latency inference.

II. Detection Formulation and Spatial Operations

Detection systems require a consistent box representation, an overlap measure, and a procedure for removing duplicate predictions.

A. Bounding box representation

A bounding box is an axis-aligned rectangle that approximates an object's location and spatial extent.

• Corner format: A box can be written as ((x{\min},y{\min},x{\max},y{\max})), where the minimum and maximum coordinates define opposite corners.
• Center format: A box can instead be written as ((x_c,y_c,w,h)), where ((x_c,y_c)) is the center and (w,h) are width and height.
• Conversion:

x_min = x_c - w/2       x_max = x_c + w/2
y_min = y_c - h/2       y_max = y_c + h/2

• Normalized coordinates: Dividing horizontal coordinates by image width (W) and vertical coordinates by image height (H) maps values approximately into ([0,1]).
• Anchor-relative regression: Given anchor ((x_a,y_a,w_a,h_a)), a detector may learn:

t_x = (x - x_a)/w_a        t_y = (y - y_a)/h_a
t_w = log(w/w_a)           t_h = log(h/h_a)

Here ((x,y,w,h)) is the target box and (t_x,t_y,t_w,t_h) are regression targets.

B. Intersection over union

Intersection over union measures the spatial overlap between a predicted box and a ground-truth box.

• Definition:

IoU(A, B) = area(A ∩ B) / area(A ∪ B)
          = area(A ∩ B) / [area(A) + area(B) - area(A ∩ B)]

Here (A) and (B) are two boxes; IoU ranges from (0), meaning no overlap, to (1), meaning identical boxes.

• Role in training: Anchors or proposals may be labelled positive when their IoU with a ground-truth box exceeds a chosen threshold.
• Role in evaluation: A predicted box is normally considered correctly localized only if its IoU exceeds the evaluation threshold.
• Worked example: If two boxes have areas (100) and (80), with intersection area (40), their union is (100+80-40=140), giving (\operatorname{IoU}=40/140\approx0.286).
• Limitation: Ordinary IoU is zero for all non-overlapping boxes, regardless of how far apart they are; generalized IoU and distance IoU provide richer optimization signals.

C. Non-maximum suppression

Non-maximum suppression removes repeated detections of the same object while retaining the strongest prediction.

• Input: NMS operates on boxes, their confidence scores, and usually one class at a time.
• Procedure:

sort boxes by descending confidence
while boxes remain:
    select and keep the highest-scoring box M
    remove M from the candidate set
    discard candidates whose IoU with M exceeds threshold T

Here (M) is the selected box and (T) is the NMS IoU threshold.

• Threshold effect: A low (T) suppresses boxes aggressively and may remove nearby objects; a high (T) retains more duplicate detections.
• Class-wise NMS: Predictions from different classes are processed separately so that overlapping objects with different labels can survive.
• Soft-NMS: Instead of deleting overlapping boxes, Soft-NMS reduces their scores according to overlap.
• Limitation: In crowded scenes, two genuine objects may overlap strongly, causing conventional NMS to suppress one incorrectly.

III. Detection Families and Early Pipelines

Detection algorithms differ mainly in how they generate candidate regions and how many processing stages they use.

A. Object detection algorithms

Modern detection algorithms can be organized by their treatment of candidate locations.

1. Two-stage algorithms: First generate class-independent region proposals, then classify and refine those proposals; Faster R-CNN is the standard example.
2. Single-stage algorithms: Predict classes and boxes directly over dense feature-map locations; YOLO, SSD, and RetinaNet belong to this family.

• Anchor-based models: Predefined boxes of several scales and aspect ratios are regressed toward target objects.
• Anchor-free models: Predict centers, corners, or distances from feature-map points to box boundaries, reducing anchor design.
• Feature pyramids: Multi-resolution feature maps improve detection across object sizes; high-resolution maps help small objects, while deeper maps represent large objects.
• Loss composition:

L = L_cls + λL_box

Here (L{\text{cls}}) is classification loss, (L{\text{box}}) is localization loss, and (\lambda) controls their relative contribution.

B. Classical detection pipelines

Classical pipelines detect objects using manually engineered features and a sliding-window classifier.

• Sliding windows: A fixed-size window scans many image positions; an image pyramid repeats this process at multiple scales.
• Handcrafted descriptors: Haar-like features, histograms of oriented gradients, and related descriptors encode edges or local appearance.
• Classifier stage: Support vector machines, boosted classifiers, or cascades determine whether each window contains the target class.
• Canonical example: The Viola–Jones face detector combines Haar-like features, AdaBoost feature selection, and a cascade that quickly rejects easy negative windows.
• Post-processing: Overlapping positive windows are merged or suppressed because neighboring windows often detect the same object.
• Limitations: Exhaustive search is computationally expensive, handcrafted features transfer poorly across varied classes, and separate pipeline components are not optimized end to end.

IV. Region-Based Convolutional Detection

Region-based CNN methods established the two-stage pattern of proposing candidate regions and applying learned recognition.

A. RCNN

R-CNN applies a convolutional classifier independently to externally generated region proposals.

• Original R-CNN pipeline:
  • Generate about 2,000 proposals using selective search.
  • Warp every proposal to a fixed input size.
  • extract CNN features for each warped region.
  • classify regions with class-specific SVMs and refine boxes using regressors.
• Main weakness: Repeating CNN computation for overlapping proposals makes training and inference slow, while multi-stage training requires substantial storage.
• Fast R-CNN: Computes one convolutional feature map for the entire image, then uses region-of-interest pooling to extract a fixed-sized feature for each proposal.
• Faster R-CNN: Replaces selective search with a trainable region proposal network sharing backbone features with the detection head.
• Region proposal network: At each feature-map location, anchors receive an objectness score and box-coordinate offsets.
• Two-stage advantage: The second stage examines a relatively small set of proposals, supporting accurate classification and localization.
• Limitations: Proposal processing and per-region operations generally produce higher latency than dense single-stage prediction.

V. Dense Single-Stage Detection

Single-stage models perform classification and localization directly across feature maps without a separate proposal-classification stage.

A. Single-stage detectors

Single-stage detectors treat detection as dense prediction over spatial locations, anchors, or object centers.

• Dense outputs: Each feature-map position predicts class probabilities and box parameters for one or more candidates.
• Efficiency: A single forward pipeline avoids a separate region proposal stage, making real-time inference practical.
• Class imbalance: Most candidate locations represent background, so negative examples can dominate the classification loss.
• Multi-scale design: Predictions from several feature levels allow one model to handle both small and large objects.
• Trade-off: Dense prediction improves speed but requires careful label assignment, loss weighting, and suppression of duplicate boxes.

B. YOLO variants

YOLO frames object detection as a unified regression problem evaluated in one neural-network pass.

• Original YOLO: Divides the image into an (S\times S) grid; a cell predicts boxes, confidence values, and class probabilities for objects whose centers fall inside it.
• YOLOv2: Introduced anchor boxes, dimension clustering, batch normalization, and multi-scale training.
• YOLOv3: Added multi-scale predictions and independent logistic class predictions using a Darknet-53 backbone.
• Later variants: YOLOv4 and subsequent families combine stronger backbones, feature pyramids, improved augmentation, better label assignment, and optimized detection heads.
• Strength: Global image processing and unified computation support high throughput.
• Limitation: Dense grid assignment can make crowded scenes and very small objects difficult, although multi-scale features substantially reduce this problem.

C. SSD

The Single Shot MultiBox Detector predicts class scores and box offsets from multiple feature maps in one pass.

• Default boxes: Each feature-map cell is associated with boxes having predefined scales and aspect ratios.
• Multi-scale prediction: Earlier, higher-resolution maps target small objects; later, lower-resolution maps target larger objects.
• Matching: Ground-truth boxes are assigned to default boxes using IoU, including each target's best-matching default box.
• Hard-negative mining: High-loss background predictions are retained to control the imbalance between negative and positive examples.
• Loss: SSD combines a confidence classification loss with smooth (L_1) localization loss.
• Limitation: The original SSD is fast but often less accurate on small objects because shallow prediction maps have weaker semantic features.

D. RetinaNet with focal loss

RetinaNet is a one-stage detector that addresses severe foreground–background imbalance using focal loss.

• Architecture: A backbone and feature pyramid network produce multi-scale features, followed by separate classification and box-regression subnetworks.
• Focal loss:

FL(p_t) = -α_t(1 - p_t)^γ log(p_t)

Here (p_t) is the predicted probability of the correct class, (\alpha_t) balances classes, and (\gamma\geq0) focuses learning on difficult examples.

• Focusing effect: If (p_t=0.9) and (\gamma=2), the easy example's cross-entropy contribution is multiplied by ((1-0.9)^2=0.01).
• Purpose: Numerous easy background anchors receive very small weights, preventing them from overwhelming rare positive detections.
• Significance: RetinaNet demonstrated that a properly trained single-stage detector could match the accuracy of contemporary two-stage systems.

VI. Detection Evaluation

Detection evaluation must measure both correct classification and sufficiently accurate localization across confidence levels.

A. Mean average precision

Mean average precision summarizes precision–recall performance across object classes and, in some protocols, multiple IoU thresholds.

• True positive: A prediction is correct when its class matches and it pairs with an unmatched ground-truth box at the required IoU.
• False positive: Wrong classes, duplicate detections, background detections, and poorly localized boxes count as false positives.
• Precision and recall:

Precision = TP / (TP + FP)
Recall    = TP / (TP + FN)

Here (TP), (FP), and (FN) denote true positives, false positives, and false negatives.

• Average precision: Predictions are sorted by confidence; AP is the area under the class's precision–recall curve.
• Mean AP:

mAP = (1/C) Σ AP_c

Here (C) is the number of classes and (AP_c) is average precision for class (c).

• Interpretation: A model needs accurate boxes, correct labels, suitable confidence ranking, and few duplicates to achieve high mAP.

B. IoU thresholds

IoU thresholds determine how strictly a predicted box must align with ground truth to count as correct.

• PASCAL VOC convention: AP is commonly reported at (\operatorname{IoU}=0.50), written AP50.
• COCO convention: Primary AP averages results over thresholds (0.50,0.55,\ldots,0.95), producing a stricter localization measure.
• Threshold contrast:
  1. AP50: Rewards detections with moderate overlap and mainly reflects object discovery and classification.
  2. AP75 or higher: Requires tighter localization and exposes inaccurate box boundaries.
• Matching rule: Each ground-truth instance can normally match only one prediction; additional overlapping predictions are false positives.
• Practical implication: Two models can have similar AP50 but different COCO AP if one produces consistently tighter bounding boxes.