Object Detection & Instance Segmentation

Learning Objectives

• ✓Distinguish classification, localization, detection, semantic segmentation, and instance segmentation
• ✓Represent boxes correctly and interpret IoU, precision, recall, AP, mAP, and NMS
• ✓Explain the efficiency progression from R-CNN to Fast and Faster R-CNN
• ✓Compare two-stage detectors, one-stage detectors, and DETR
• ✓Fine-tune a torchvision Faster R-CNN model on a custom detection dataset

Five Vision Tasks, Five Different Outputs

Image classification returns one label for the whole image. Localization adds one box. Object detection returns a variable-size set of class, box, and confidence tuples. Semantic segmentation assigns a class to every pixel but does not separate two objects of the same class. Instance segmentation returns a separate mask for each object. This output contract determines the annotation format, loss, model head, and metric, so identify the task before choosing an architecture.

Boxes, IoU & Detection Metrics

Torchvision uses boxes as $[x_{min},y_{min},x_{max},y_{max}]$ with positive width and height. Intersection over Union measures overlap between prediction and ground truth. A prediction becomes a true positive only when its class is correct, its IoU meets the threshold, and that ground-truth object has not already been matched. Precision penalizes false alarms; recall penalizes misses. Average Precision summarizes a precision-recall curve, while COCO mAP averages AP across classes and IoU thresholds from 0.50 to 0.95.

From Sliding Windows to R-CNN

A brute-force detector classifies crops at many positions, scales, and aspect ratios, which repeats expensive CNN computation. R-CNN reduces the search with roughly two thousand region proposals, warps each proposal, runs the CNN separately, classifies with SVMs, and regresses box corrections. It established the region-based recipe but remains slow because every region needs an independent forward pass. Selective Search is outside the learned network, creating a second bottleneck.

Fast R-CNN & Faster R-CNN

Fast R-CNN runs the backbone once on the full image, then crops and resizes features for every proposal before joint classification and box regression. Faster R-CNN learns the proposal stage too. Its Region Proposal Network places anchors of multiple scales and aspect ratios on feature maps, predicts objectness plus four box offsets, ranks candidates, and keeps the strongest proposals. ROI Align turns variable-size proposal features into a fixed shape for the second-stage head. A Feature Pyramid Network supplies high-resolution maps for small objects and deeper semantic maps for large ones.

NMS, YOLO & the One-Stage Trade-off

Non-Maximum Suppression sorts boxes by confidence, keeps the best, and removes highly overlapping lower-scored boxes. A low NMS threshold removes duplicates aggressively but can erase nearby objects; a high threshold improves recall but keeps more duplicates. One-stage detectors such as YOLO, SSD, and RetinaNet predict boxes, objectness, and classes densely without a separate proposal-and-ROI stage. They are typically easier to run in real time, while two-stage models are often selected when proposal quality and accuracy matter more than latency.

DETR: Detection as Set Prediction

DETR uses a Transformer to output a fixed-size set of object predictions directly. Instead of anchors and hand-designed matching rules, bipartite matching assigns predictions to ground truth, and unmatched queries learn a no-object class. This produces an elegant end-to-end formulation and removes conventional NMS, but the training behavior and compute profile differ from CNN detectors. The conceptual shift is important: detection can be posed as permutation-invariant set prediction.

Mask R-CNN Adds an Instance Mask Head

Mask R-CNN extends Faster R-CNN with a small fully convolutional branch for each aligned ROI. The classification head predicts $C$ scores, the box head predicts class-specific offsets, and the mask head predicts one binary mask per class, commonly at $28\times28$. During training, only the mask channel for the ground-truth class contributes to the mask loss. ROI Align avoids the quantization error that would blur pixel-level mask boundaries.

Fine-tuning Faster R-CNN on Penn-Fudan

Penn-Fudan contains 170 pedestrian images and instance masks. The masks can be converted to boxes, but training Faster R-CNN remains a detection task. Start from COCO weights, replace the classification predictor for background plus one foreground class, pass a list of images and a list of target dictionaries, and monitor the four returned losses. Keep the evaluation score threshold low enough for mAP calculation; use a higher display threshold only for visualization.

from torchvision.models.detection import (
    FasterRCNN_ResNet50_FPN_Weights,
    fasterrcnn_resnet50_fpn,
)
from torchvision.models.detection.faster_rcnn import FastRCNNPredictor

model = fasterrcnn_resnet50_fpn(
    weights=FasterRCNN_ResNet50_FPN_Weights.DEFAULT,
    trainable_backbone_layers=3,
)

in_features = model.roi_heads.box_predictor.cls_score.in_features
model.roi_heads.box_predictor = FastRCNNPredictor(
    in_features,
    num_classes=2,  # background + pedestrian
)

# train mode: loss_dict = model(list_of_images, list_of_targets)
# eval mode:  predictions = model(list_of_images)