1 of 31

Introduction to DeepStream, YOLOv3, and 3D Point Cloud Object Detection

Smart Intersection Project

Team Meeting Presentation - Friday June 12

2020 WINLAB Summer Internship

Nicholas Meegan

Kevin Zhang

Bryan Zhu

https://bzz3ru.wixsite.com/smartintersection | http://www.winlab.rutgers.edu/

2 of 31

NVIDIA DeepStream Overview

3 of 31

What is NVIDIA DeepStream?

Definition: A software development toolkit (SDK) for machine learning, image processing and detection, and video processing and detection
Develop in C, C++, or Python languages for ease of use
Uses object detection from various plugins

SSD
YOLO
FasterRCNN

4 of 31

Source: https://news.developer.nvidia.com/new-nvidia-deepstream-sdk-3-0-removes-boundaries-of-video-analytics/

5 of 31

How does DeepStream work?

Built on the top of NVCODEC and NVIDIA TensorRT, among various other lower level layers

NVCODEC - SDK used for video decoding, which utilizes the NVIDIA GPU to do faster than real-time processing
TensorRT - SDK for deep learning

Inference optimizer
Low latency and high-throughput
Optimize neural net models, lower precision and high accuracy

CUDA - parallel computing platform and application programming interface model

Gives the developer the ability to use a CUDA GPU for general purpose processing

6 of 31

How does DeepStream work? (Cont.)

DeepStream then employs third-party plugins to real-time image classification and object detection

GoogleNet, YOLOv3, etc.

7 of 31

DeepStream Installation

Installation on a Ubuntu x86_64 system with a discrete NVIDIA GPU
Install necessary packages:

All installation instruction sourced from: https://docs.nvidia.com/metropolis/deepstream/4.0/dev-guide/index.html

8 of 31

DeepStream Installation (Cont.)

2. Install CUDA Toolkit and TensorRT at the respective links:

https://developer.nvidia.com/cuda-downloads

https://developer.nvidia.com/nvidia-tensorrt-download

3. Install librdkafka for data processing (Kafka message broker), and build the library

$ git clone https://github.com/edenhill/librdkafka.git

$ cd librdkafka

$ git reset --hard 7101c2310341ab3f4675fc565f64f0967e135a6a

./configure

$ make

$ sudo make install

$ sudo mkdir -p /opt/nvidia/deepstream/deepstream-4.0/lib

$ sudo cp /usr/local/lib/librdkafka* /opt/nvidia/deepstream/deepstream-4.0/lib

9 of 31

DeepStream Installation (Cont.)

4. Install DeepStream SDK

$ sudo apt-get install ./deepstream-4.0_4.0-1_amd64.deb

To run, use deepstream-app:

$ deepstream-app -c <path_to_config_file>

The config file is a set of key-value pairs, used to set properties based on configuration

10 of 31

Using NVIDIA DeepStream with YOLOv3

To Use YOLOV3 with NVIDIA Deepstream, all we have to do is modify some lines of C/C++ code in the DeepStream Application to fit our needs

The next slide shows a step by step implementation of YOLOV3 in NVIDIA DeepStream

To summarize the next slide, all we need is a good understanding of YOLOV3 from Joseph Redmon’s YOLO Configuration and the code for NVIDIA DeepStream, which can be found on GitHub (Link Provided)

https://github.com/Kwull/deepstream-4.0.1/tree/master/sources/objectDetector_Yolo

11 of 31

Using NVIDIA DeepStream with YOLOv3

(Cont.)

Using NVIDIA DeepStream with YOLOv3

Source: https://docs.nvidia.com/metropolis/deepstream/Custom_YOLO_Model_in_the_DeepStream_YOLO_App.pdf

12 of 31

YOLOv3 Overview

Logo: https://pjreddie.com/darknet/yolo/

13 of 31

What is YOLO?

object detection system
compared to other object detection methods, notable for its…

detection speed (fast enough for real-time applications)
consideration of objects in the context of the whole image (global reasoning, reduced background error rate)
learning of generalizable representations of objects (models less likely to fail when viewing novel class inputs)

YOLO treats object detection as a single regression problem, where the entire image is fed into a single neural network (hence, You Only Look Once)

Image: https://pjreddie.com/darknet/yolo/

14 of 31

Things to Keep in Mind about YOLOv3

YOLOv3 can run on images of any dimension

however, if you want to batch process images (feed more than one into the model at once), they must be of equal widths and heights

YOLOv3 can only detect classes that were present during training

can't detect new objects/classes that weren't labelled in the training dataset

15 of 31

How YOLOv3 Works - Input Tensor

the input to the YOLOv3 detector should be of shape (m,w,h,3) where…

m = the number of images we want to process (batch size)
w = the pixel width of images in the batch
h = the pixel height of images in the batch
the last dimension of size 3 is to hold information for all three RGB color channels

other YOLOv3 parameters…

C = number of classes recognized by network (from training)
various prediction thresholds for things such as objectness scores and class confidences

16 of 31

How YOLOv3 Works - Darknet-53 CNN

inputs fed into the Darknet-53 Convolutional Neural Network (CNN)

Darknet-53 is a Fully Convolutional Network (FCN)

Convolutional layers are all...

batch normalized

layer outputs are first normalized
normalized output multiplied by γ and summed with β (denormalization); both params trainable
preserves stability of network (weights no longer constantly change)

activated by Leaky ReLU layers

Table: https://arxiv.org/abs/1804.02767

17 of 31

How YOLOv3 Works - Darknet-53 CNN

feature maps downsampled with convolutional layers with stride 2

stride determines the amount of units a filter is shifted by��
5 layers in total, resulting in a network stride of 32

final softmax activation layer produces probability distribution

Images: https://adeshpande3.github.io/A-Beginner%27s-Guide-To-Understanding-Convolutional-Neural-Networks-Part-2/

Table: https://arxiv.org/abs/1804.02767

18 of 31

How YOLOv3 Works - CNN Output

the output of Darknet-53 will be of shape (fw,fh,B,5+C) where…

fw and fh are the dimensions of the feature map

e.g. an image of size 416x416 run through a network with stride 32 will have detections made on a feature map of size (416/32)x(416/32) → 13x13
if the center of an object lies on a particular grid cell of the feature map, that cell is responsible for detecting that specific object

B = number of bounding boxes to be predicted for each grid cell (YOLOv3 uses 3)
the last vector is of the form (po,tx,ty,tw,th,c1,c2,...,cC) where...

po = the objectness score of the bounding box
tx, ty, tw, and th will be used to determine the position and size of the bounding box
c1 through cC are the class scores for the bounding box (one score for each class)

19 of 31

How YOLOv3 Works - Bounding Boxes

predict log-space transforms from sets of predefined anchor boxes (also called priors)

avoid directly predicting bounding box sizes, since doing so leads to unstable training gradients

anchor boxes for a class represent ideal locations, shapes, and sizes for that class
anchor boxes can be calculated from a training dataset using k-means clustering

unsupervised algorithm
given parameter k, tries to group inputs into k clusters

clusters defined by their centroids (centers)
continually reevaluate clusters and centroids according to a chosen distance metric until centroids have stabilized

20 of 31

How YOLOv3 Works - Bounding Boxes

tx, ty, tw, and th transforms are applied on anchor boxes to produce actual bounding box coordinates bx, by, bw, and bh
box positions predicted using sigmoid function σ(x) = 1/(1 + exp^(-x))

bx = σ(tx) + cx
by = σ(ty) + cy
cx and cy are the top-left coordinates of the grid
sigmoid function keeps bounding box predictions within their respective grid cell

box widths and heights are predicted as offsets from cluster centroids

bw = pw*exp(tw)
bh = ph*exp(th)
pw and ph are the anchor dimensions for the box

21 of 31

How YOLOv3 Works - Scoring

objectness score

probability that an object is contained inside a bounding box
passed through sigmoid

class confidences (class scores)

probabilities of the detected object belonging to a particular class
also passed through sigmoid

22 of 31

How YOLOv3 Works - Prediction Scales

predictions are generated �across 3 different scales �(i.e. 3 different feature maps)

done to improve small object detection rate (something previous YOLO versions were bad at)

feature maps with strides 32, 16, and 8 are used for feature detection

e.g. an image of size 416x416 will have detections made on feature maps of size 13x13, 26x26, and 52x52

anchor boxes differ between each scale
our final outputs are selected from the predicted bounding boxes from all 3 scales

e.g. if 3 bounding boxes are predicted for each grid cell, we end up with a total of�3*(13*13 + 26*26 + 52*52)) = 10647 bounding boxes to filter our best outputs from

Image: https://pylessons.com/YOLOv3-introduction/

23 of 31

How YOLOv3 Works - Output Filtering

bounding boxes with low objectness scores are ignored (YOLO is not confident an object was detected in this box)
bounding boxes with low class scores are also ignored (YOLO is not confident the object in this box is of any specific class)
of the remaining, there will likely be multiple overlapping boxes detecting the same objects

choose the best representative box for that object using NMS

Image: https://pylessons.com/YOLOv3-introduction/

24 of 31

How YOLOv3 Works - Output Filtering

Intersection over Union (IoU) is defined as the common area between two regions divided by their union (hence the name)
Non-Maximum Suppression (NMS) uses IoU to collapse bounding boxes

IoUs are computed of the box with the highest score with all other boxes
boxes with low scores when computed with the highest scoring box are removed
the above steps are repeated until no box has a lower score than the currently selected box

what remains are the best boxes for all detected classes in the image (FINAL OUTPUT)

Images: https://pylessons.com/YOLOv3-introduction/

25 of 31

YOLOv3 Further Reading

You Only Look Once: Unified, Real-Time Object Detection�https://arxiv.org/abs/1506.02640
YOLO9000: Better, Faster, Stronger�https://arxiv.org/abs/1612.08242
YOLOv3: An Incremental Improvement�https://arxiv.org/abs/1804.02767
YOLO v3 theory explained: Understand how YOLO v3 works�https://pylessons.com/YOLOv3-introduction/

26 of 31

YOLO Support in DeepStream

one of the DeepStream GStreamer plugins is called Get-nvinfer
as of DeepStream 4, this Gst-nvinfer plugin carries support for YOLOv3, YOLOv3-tiny, YOLOv2, and YOLOv2-tiny

the baseline-deepstream.ndk image currently available to WINLAB members comes with the DeepStream 5 Developer Preview, CUDA 10.2, CudNN version 7.6, and TensorRT version 7

27 of 31