Deep Learning for Object detection & localization

R-CNN, Fast R-CNN, Faster R-CNN, YOLO, GAP, CAM, MSROI

Aaditya Prakash�Sep 25, 2018

Image classification

Image classification

• Whole of image is classified as one class.
• Every pixel is used to determine the class.
• Does not know/care if there is another object in the image.
• No separation of background/foreground, hence ‘tank-problem’.

Localization and Detection

Source: cs231n

Classification vs Localization vs Detection vs Segmentation

Source: cs231n

Detection - Fast and Accurate

MobilEye (video)

Nikon S60

Source: Ross Girshick

Detection - Fast and Accurate

MobilEye (video)

Nikon S60

Source: Ross Girshick

Demo

Source: YOLO

Before deep learning

• Deformable Parts Model
• low-level features based on HOG, SVM classifier
• mAP: 33%

​

Source: Pedro Felzenszwalb

Before deep learning

• Deformable Parts Model
• low-level features based on HOG, SVM classifier
• mAP: 33%
• Selective Search
• Bag-of-words with SIFT features, merge similar segments on segmentation
• mAP: 35%

​

Source: Uijlings et al

Before deep learning

• Deformable Parts Model
• low-level features based on HOG, SVM classifier
• mAP: 33%
• Selective Search
• Bag-of-words with SIFT features, merge similar segments found by segmentation
• mAP: 35%
• Regionlets
• HOG, LBP, Covariance features, relative position of sub-parts
• mAP: 40%

Source: Wang et al

Naive way to use CNN for localization - Regression

Source: cs231n

Naive way to use CNN for localization - Regression

Source: cs231n

Naive way to use CNN for localization - Regression

Source: cs231n

Naive way to use CNN for localization - Regression

• Limited to ‘one’ class per image
• Limited to ‘one’ instance of object per image
• Cannot separate object parts - like ear, head, eyes
• Limited to ‘rectangle’ object boundaries

​

Source: cs231n

Better way - Sliding window

Source: cs231n

Better way - Sliding window

Source: cs231n

Better way - Sliding window

Source: cs231n

Better way - Sliding window

Source: cs231n

Better way - Sliding window

Source: cs231n

Imagenet results (Localization)

AlexNet: Localization method not published

Overfeat: Multiscale convolutional regression with box merging

VGG: Same as Overfeat, but fewer scales and locations; simpler method, gains all due to deeper features

ResNet: Different localization method (RPN) and much deeper features

Can we do detection as regression?

Can we do detection as classification?

• Four corners for every object, many numbers to regress.
• No idea on how many objects to expect.

• How many boxes should we check? All of them?
• At what scale? Some dogs are smaller (or far away)

Can we do detection as regression?

Can we do detection as classification?

• Four corners for every object, many numbers to regress.
• No idea on how many objects to expect.

• How many boxes should we check? All of them?
• At what scale? Some dogs are smaller (or far away)

Better solution - Region proposals

• Find ‘regions-of-interest’ that are likely to contain some objects
• A simple detector which just says there is something of interest and not necessarily about the class
• These region proposals could be just blob of pixels or ‘super-pixels’.
• There are lot of techniques, like BING, CPMC, EdgeBoxes, Geodesic, MCG etc

Region Proposals - Selective Search

Intersection over Union

If IoU > 0.5 ⇒ Success

Region based CNN (R-CNN)

Source: Ross Girshick

Source: Ross Girshick

R-CNN Training

1. [offline] M ⃪ Pre-train a ConvNet for ImageNet classification

2. M’ ⃪ Fine-tune M for object detection (softmax classifier + log loss) [add background class]

3. F ⃪ Cache feature vectors to disk using M’

4. Train post hoc linear SVMs on F (hinge loss) [binary classifier]

5. Train post hoc linear bounding-box regressors on F (squared loss)

​

Source: E Mercan, R Girshick

R-CNN Results/Analysis

1. Very slow (upto 20 seconds per image on K40 GPU).�
2. Needs to run full CNN pass on each proposed region.�
3. Not end-to-end training.�
4. CNN features are not updated with results of training SVM�
5. Better results than any other system�

mAP - mean Average Precision� - Average of Average Precision for each class.� - True positive if IoU with ground-truth box > 0.5

Source: E Mercan

R-CNN Results/Analysis

• Very slow (upto 20 seconds per image on K40 GPU).�
• Needs to run full CNN pass on each proposed region.�
• Not end-to-end training.�
• CNN features are not updated with results of training SVM�
• Better results than any other system�

mAP - mean Average Precision� - Average of Average Precision for each class.� - True positive if IoU with ground-truth box > 0.5.� - AP is area under the Precision-Recall curve.

​

Source: E Mercan

Fast R-CNN

• Share computation of convolutional layers between ‘regions of proposal’

​

Source: Ross Girshick

Fully connected to fully convolutional

Source: Coursera

Fully connected to fully convolutional

Source: Coursera

Fully connected to fully convolutional

Source: Coursera

Convolutional implementation of sliding window

Source: Coursera

Convolutional implementation of sliding window

Source: Coursera

Fast R-CNN

• Share computation of convolutional layers between ‘regions of proposal’
• Train the whole system end-to-end
• no more post-hoc classifiers
• No caching features to disk�

Source: Ross Girshick, Kaiming He

Fast R-CNN

• Share computation of convolutional layers between ‘regions of proposal’
• Train the whole system end-to-end
• no more post-hoc classifiers
• No caching features to disk�

• Positive samples are those with IoU overlap with ground-truth bounding box > 0.5.
• Negative samples are in interval [0.1,0.5)
• 25%/75% ratio of Positive to Negative samples

Source: Ross Girshick, Kaiming He

Fast R-CNN: Region of Interest Pooling

Fast R-CNN: Region of Interest Pooling

Fast R-CNN: Region of Interest Pooling

Fast R-CNN: Region of Interest Pooling

Fast R-CNN: Region of Interest Pooling

Region of interest pooling - visualized

Fast R-CNN: Region of Interest Pooling

Fast R-CNN: Spatial Pyramid Pooling

Fast R-CNN: Spatial Pyramid Pooling

ROI Pooling is a special case of SPP pooling

Faster R-CNN

• In R-CNN and Fast R-CNN, bottleneck is ‘region proposal’ (selective search, CPMC, MCG).
• Replace external proposals with Region Proposal Network (RPN), trained with model directly.
• After RPN use ROI Pooling like Fast R-CNN

Region Proposal Network

• Slide a small window (anchor boxes) on the feature map.
• Have multiple such (k) anchor boxes.
• Build small network for
• binary classification of object in an anchor. (👍)
• regression of bounding box (anchor->proposal)
• Position of sliding window provides localization information with reference to the image.
• Binary classification gives probability that each anchor shows an object.
• Regression score gives offset from anchor boxes
• Anchors are selected for different scales and different aspect ratio, generally 3 each, total 9.

Faster R-CNN: RPN

Anchor is labelled positive if:� - anchor is the one with highest IoU overlap with ground truth box� - anchor has IoU overlap with a ground-truth box higher than 0.7�Anchor is labelled negative if IoU lower than 0.3 for all ground-truth boxes.�50%/50% ratio of positive to negative samples

Source: A Salvador

Faster R-CNN: Comparison accuracy/speed

Non-max Supression

1. Find all the regions that have high IOU
2. Pick the one with the highest confidence score

0.6

0.7

0.5

You Only Look Once (YOLO)

• Divide image into S x S grid
• Within each grid cell predict:
• B Boxes (4 coordinates + confidence)
• Class scores: C scores
• Regression from image to 7 x 7 x (5 * B + C) tensor

​

​

You Only Look Once (YOLO)

• Divide image into S x S grid
• Within each grid cell predict:
• B Boxes (4 coordinates + confidence)
• Class scores: C scores
• Regression from image to 7 x 7 x (5 * B + C) tensor
• Direct prediction using a CNN
• Faster than Faster R-CNN but not as good
• Struggles with small objects and different aspect ratio

​

Can you do this without spatial ground-truth?

Weakly supervised object localization

• If you look at the pixels which give maximum activation for a given class, it is no surprise that pixels corresponding to actual object have higher activation.
• But how do you control of actual object pixels? Remember the ‘tank problem’?
• Also this is not reliable in a cluttered scene containing many objects, and partially occluded objects or cropped objects.
• Oquab et al,
• removed the Fully Connected layer at the end of a CNN model to prevent losing any more localization information and,
• Added global max-pooling (GMP) to output layer to find highest scoring object position in the image.

​

​

​

Can you do this without spatial ground-truth?

Weakly supervised object localization

• If you look at the pixels which give maximum activation for a given class, it is no surprise that pixels corresponding to actual object have higher activation.
• But how do you control of actual object pixels? Remember the ‘tank problem’?
• Also this is not reliable in a cluttered scene containing many objects, and partially occluded objects or cropped objects.
• Oquab et al,
• removed the Fully Connected layer at the end of a CNN model to prevent losing any more localization information and,
• Added global max-pooling (GMP) to output layer to find highest scoring object position in the image.
• GMP enforces correspondence between feature maps and categories, and thus they can be interpreted as ‘confidence maps’.
• Best part - it is parameter free.

​

​

Max Pooling vs Global Max Pooling

Max Pooling vs Global Max Pooling

Global Max Pooling - Results

• Localization is only about finding ‘center’ point, cannot draw bounding box
• Helps improve the classification score
• Masked pooling is where you minimize the object score outside the masked area (requires location GT)

Class activation map

• Replace Global Max Pooling (GMP) with Global Average Pooling (GAP).
• The use of average pooling encourages the network to identify the complete extent of the object
• Intuition behind this is that the loss for average pooling benefits when the network identifies all discriminative regions of an object as compared to max pooling
• Class activation map is when you take the output of GAP and add a ‘fully connected layer’ (without nonlinearity). Use the weights of this layer to weight the activation at unit k, to generate a spatial map.

• For training, you maximize the cross entropy between predicted class and true distribution�
• This means it can “highlight” only one salient class in a given image. Gives more than center but not bounding box.

Class activation map

References

Questions?

1. Tools for efficient Object Detection [pdf]
2. R-CNN for Object Detection [pdf]
3. Segmentation as Selective Search (Poster) [pdf]
4. Object Detection, Lecture at UPC [pdf]
5. Stanford CS231n Lecture 8 slides [pdf]
6. Faster R-CNN: Towards real-time object detection [pdf]
7. Is localization for free? [pdf]
8. Learning Deep Features for Discriminative Localization [pdf]
9. Coursera: Convolutional Neural Networks [link]

​