Zirui Zang and the F1TENTH Team

Contact: Rahul Mangharam <rahulm@seas.upenn.edu>

Portions of these slides are from Penn Course MEAM620, CIS680

F1TENTH Autonomous Racing

​

​

�Vision I : Classical Methods

​

​

Vision Module Overview

Lecture 1 : Classical Methods

• Vision Hardware
• Accessing Camera on Linux
• Camera Model & Distance Estimation
• Useful OpenCV Functions
• Visual SLAM

Lecture 1I : DL Methods

• Deep Learning Basics
• Object Detection w/ Image
• Object Detection w/ Pointcloud
• Recent Trend in DL
• Network Deployment

Vision Hardware

Vision Sensors - Cameras

• RGB Camera
• Event-based Camera
• Depth Camera
• Stereo Camera
• Structural Light Camera
• Active Stereo (Combining both)

​

Vision Sensors - Lidar

• Dense 3D Lidar can be used as a vision sensor
• xyz coordinates + intensity
• Mechanical Lidar vs. Solid State Lidar
• Moving parts, Scanning speed, Beam density

​

“Image” from one dense 3D lidar scan

Intel Realsense D435i

• Active depth up to 3m.
• Output image, depth map and point cloud.
• Supports most libraries, including OpenCV and ROS2.

Let’s read its spec.

And see why do they provide these spec.

Camera Spec - Shutter Type

• Shutter Type
• Rolling: Cheaper
• Global: better for high speed objects

​

Rolling

Global

Camera Spec - Sensor Size & Pixel Size

• Sensor Size & Pixel Size
• Bigger is better, more light gathering, more information.

​

iPhone 12 Pro Max marketing, but it’s true

Camera Spec - FOV

• Field of View (FOV)
• FOV = arctan(sensor_size/focal_length * 2) * 2

Camera Spec - Connector

• Connector
• Normal webcam is USB2.0, but why does Realsense use USB3.1?

​

Image Signal Processor (ISP)

• A specialized chip for image processing. A more and more important part of a SoC chip.
• ISP can handle image denoise, resize, transform, filter, auto-exposure, auto-whitebalance, etc. faster than CPU or GPU.
• As for our Realsense, there is an onboard SoC.

new Snapdragon 8 Gen 1 chipset offers 18-bit triple-ISP design

Accessing Camera on Linux

ls /dev/video*

• In Linux, every device is mounted as a file. So are cameras.
• After you connect your camera, you can use ls /dev/video* to see if the system has recognized your camera.
• For example, if the Realsense D435i is recognized by Jetson NX:

16-bit Depth Map

IR Image

RGB Image

v4l2src

• Video4Linux (v4l2) is an API to access camera on Linux.
• v4l2src can be used to capture video from v4l2 devices.
• You can give the pipeline to an OpenCV VideoCapture object.

v4l2-ctrl -d /dev/video2 --list-ctrl-menus

• v4l2-ctrl can be used to control the camera on Linux.
• OpenCV also has an API that can control the camera. Most cameras will support either (if not both) v4l2 or OpenCV.

v4l2-ctrl -d /dev/video2 --list-formats-ext

• --list-formats-ext can show all operating modes for the camera.
• Note that if we want the Realsense camera to operate at 60Hz, it can only support 960x540.

Camera Model & Calibration

Image - Camera - World

• Goal: We need to find transformations between:

Image pixel coordinates Camera coordinates World coordinates

​

​

​

Camera Intrinsics

• From trigonometry we can find:

Camera Extrinsics

• Camera Extrinsics is a homogeneous transform from camera frame to world frame, both are 3D systems.
• It will be consist of a rotation and a translation.

​

Intrinsics & Extrinsics

• But how do we calculate them for an actual camera?

Here X and Y are flipped, but it’s equivalent.

Get Intrinsics - Camera Calibration

• We can use OpenCV functions findChessboardCorners and calibrateCamera to do the calibration.
• It will not only calculate the camera matrix K, but also a distortion matrix to correct the image of any distortion.
• Tutorial, OpenCV documentation.

Camera Matrix

Get Extrinsics - Computing a Transformation

• The extrinsics can be calculated with:
• Find enough corresponding points in camera and world frames;
• Solve the linear equation for the homography;
• Separate out rotation and translation using singular value decomposition.

Distance Estimation

Distance Estimation

• What if there are rotations in camera mounting?

We can add more corresponding points to solve for any missing variable.

Detection with OpenCV

Useful for doing the vision lab.

Edge & Blob Detection

cv2.Canny

detector = cv2.SimpleBlobDetector()

keypoints = detector.detect(img)

​

​

Parameters for the SimpleBlobDetector

HSV color space

• Convert (red, green, blue)
• To (hue, saturation, value)
• value is also called intensity or brightness.
• HSV is sometimes more useful than RGB when doing color-based filtering.
• We can use to cv2.cvtColor convert color space.

HSV color space

Hue value

Example: lawn detection

• Suppose you are an engineer in an autonomous lawn mower startup.
• The investor is coming to see the prototype tomorrow. There is no time to train a neural network.
• You are given the task to detect the boundary of a lawn, with classical methods.

• It seems difficult to do it in RGB space. We can try HSV color space.

Used function:

cv2.GaussianBlur

Used function:

cv2.cvtColor

cv2.threshold

Used function:

cv2.connectedComponentsWithStats

Used function:

cv2.dilate

cv2.erode

​

​

​

​

Feature Extraction

• To detect an object in an image, we always need to find features.
• What is a feature?

• AB: flat surfaces in some channels.

• CD: certain edges

• EF: certain corners or curves.

• We need special filters to respond to them.

2D Convolution

• The weights g is often called a kernel or a filter.
• Kernels with different weights will respond differently to features.

Some Classical Feature Extraction Methods

Shi-Tomasi Corner Detector:

cv2.goodFeaturesToTrack()

SIFT (Scale-Invariant Feature Transform):

sift = cv2.SIFT_create()

kp = sift.detect()

​

ORB (Oriented FAST and Rotated BRIEF):

orb = cv2.ORB_create()

kp = orb.detect()

kp, des = orb.compute()

​

By matching these features between frames, we can track object, classically.

Put it together

• We can find unique features in images and match them between frames.
• And we can find their coordinates in the world frame with camera model.
• Can we use them as landmarks in SLAM?

​

https://introlab.github.io/rtabmap/

Visual SLAM

Visual SLAM Framework

• Feature Matching: Extract and match visual feature between camera frames.
• Visual Odometry: Calculate the motion between consecutive camera frames.
• Bundle Adjustment: Correct the transformations with historical data.
• Map Building: Build the map based on the measurements and corrections.
• Loop Closure: Detect path returns to a previous point and correct the accumulated drift error.

​

ORB-SLAM2

Bundle Adjustment

• Optimizing the camera projections by considering all points and cameras.
• Starting from an initial guess.
• Treating camera from different time as different cameras.
• It is a non-linear optimization, including rotation, distortion correction etc.
• Statistically optimal by expensive.

​

​

Challenges in Visual SLAM

• Exponentially increasing number of feature points to match.
• Bundle adjustment is especially expensive.
• Can it work on monocular camera without IMU?
• Can the algorithm handle drifts?
• Can it close the loop?

​

ORB-SLAM

• 3 parallel threads: Tracking, Local Mapping and Loop Closing.
• Selection of keyframes and Culling policies of keyframes and points.
• Separating of local and global information with Covisibility Graph and Essential Graph.
• Use Bags of Words for place recognition in loop closure.

​

References

• https://medium.com/@dcasadoherraez/introduction-to-visual-slam-chapter-1-introduction-to-slam-a0211654bf0e
• https://arxiv.org/pdf/2107.07589.pdf