F1TENTH Autonomous Racing
�Automatic Emergency Braking
Safe Autonomy Lab
University of Pennsylvania
Rahul Mangharam
University of Pennsylvania
rahulm@seas.upenn.edu
Acknowledgements
Except where otherwise noted, this work is licensed under
This course is a collaborative development with significant contributions from:
Hongrui Zheng (lead), Matthew O’Kelly (lead), Johannes Betz (lead), Houssam Abbas, Joseph Auckley, Madhur Behl, Luca Carlone, Jack Harkins, Paril Jain, Kuk Jang, Paritosh Kelkar, Sertac Karaman, Dhruv Karthik, Nischal KN, Thejas Kesari, Matthew Lebermann, Kim Luong, Yash Pant, Varundev Shukla, Nitesh Singh, Siddharth Singh, Nandan Tumu, Zirui Zang, and many others.
We are grateful for learning from each other
Lesson Plan
Part I: Automatic Emergency Braking (AEB) on the road
Part II: Range sensors for autonomous vehicles
Part III: Working with laser scan data
Part IV: Measuring safety
Part V: Lab overview
3
AEB on the road
x
Automatic Emergency Braking (AEB)
Stop the vehicle before you collide with an obstacle...
5
Source: Nissan USA
6
Safety Lab: AEB on the F1Tenth race car
Problem:
Prevent the car from crashing while trying new algorithms.
Understand:
Real-life implementations, sensors, failure modes.
Implement: Time-to-collision based braking
AEB in Today’s Vehicles
Fuse radar and calibrated vision pipeline (object detection, tracking, visual odometry)...
8
1. Detect
Objects
3. Determine Critical Object
2. Find range, heading and velocity
4. Check trajectory for imminent collision
Source: Mobileye
Apply AEB?
T/F
Quiz: AEB in Today’s Vehicles
What type of model is this system? Regression? Deep Network? Or something simpler?
9
1. Detect
Objects
3. Determine Critical Object
2. Find range, heading and velocity
4. Check trajectory for imminent collision
Source: Mobileye
Apply AEB?
T/F
Answer: AEB in Today’s Vehicles
What type of model is this? Binary Classifier.
10
1. Detect
Objects
3. Determine Critical Object
2. Find range, heading and velocity
4. Check trajectory prediction for imminent collision
Source: Mobileye
Apply AEB?
T/F
Quiz: Types of Failures
Can you fill in this table?
11
Classifier Output
Ground Truth
True False
False True
Quiz: Types of Failures
Which one of these is a serious problem?
12
True Positive
False Positive
False Negative
True Negative
Classifier Output
Ground Truth
Answers: Types of Failures
Quiz: Which one of these is a serious problem?
It’s not this...
13
True Positive
False Positive
False Negative
True Negative
Engineering incentives aligned.
But… no one will buy a system with false positives.
False Positive
Classifier Output
Ground Truth
Answers: Types of Failures
14
Answers: Types of Failures
Quiz: Which one of these is a serious problem?
15
True Positive
False Positive
False Negative
True Negative
Classifier Output
Ground Truth
False negatives kill innocent people. Regulatory & insurance provide incentives.
Types of Failures: False Negative
Uber autonomous car kills a pedestrian
Viewer discretion is advised
16
Source: The Guardian
Is this problem solved?
AEB could have stopped the vehicle, but was turned off because the vehicle had frequent false positives.
17
Source: Unknown
False Positives & False Negatives
You will need to tune this for lab
18
HIGH
True Positive
NO�False Positive
LOW
False Negative
HIGH
True Negative
Classifier Output
Ground Truth
Addressing AEB false negatives in the Automotive Industry
19
Example car sales labels
20
Later in the course...
Not just about safety, can lead to better racing:
Lecture 21 & 22: Vehicle detection & tracking
Lecture 23 & 24: Reinforcement learning
21
Consider implementing AEB based on CV Pipeline. Details in Lecture 21 & 22.
Use concepts presented in this lecture as a signal to switch to a safer policy. Lecture 23 & 24.
Later in the course
22
Detect opponent and generate candidate goals
Predict Opponent Behavior
Choose the “best” trajectory
Later in the course...
23
Range sensors for AVs
x
Range Sensor Overview
25
Structured Light
Stereo Camera
Monocular Camera
Radar
Ultrasonic
Planar LIDAR
3D LIDAR
Solid State LIDAR
Source: Intel realsense, Zed, Pointgrey, Continental, Hukoyu, Velodyne and Vex
Range Sensor Suite on a Typical AV
26
Source: Vivota.com
RGB-D Camera
Demo Hardware: Intel Realsense D435i
Working Principle: Active Infrared Stereo
Advantages: 3D point cloud - great for visuals SLAM, IR Stereo works in any environment, Mature SDK, widely used, calibrated IMU
Disadvantages: Limited range, can be noisy, lighting conditions affect performance, requires building Linux kernel
27
Realsense RGB-D Camera
28
Source: Jim Benson, JetsonHacks
Stereo Camera
Demo Hardware: ZED Camera
Working Principle: Stereo/Multiview Geometry
Advantages: Range, outdoor performance, ‘scalability’, calibrated IMU
Disadvantages: Low texture surfaces, baseline width determines range, processing requirements.
29
Monocular Depth
Example Hardware: Logitech Webcam & Monodepth Net
Working Principles: Learn from stereo camera data (unsupervised) to map monocular to depth.
Advantages: No special hardware, orthogonal failure modes to other options
Disadvantages: Relative poor accuracy (good for near vs. far questions).
30
RADAR
Example Hardware: Continental (etc)
Working Principles: Emit RF energy and measure ‘echo’, can use doppler shift etc to get velocity.
Advantages: Cheap, long range, orthogonal failure modes.
Disadvantages: Poor spatial resolution and field of view, false positives on overhead signs etc.
31
Ultrasonic Proximity Sensor
Example Hardware: VEX IQ Ultrasonic Distance Sensor
Working Principle: Measures time of flight of high-frequency sound waves.
Advantages: Accuracy, cost, size
Disadvantages: Range, resolution, only good enough for parking applications (eg. 5m)
32
Planar 2D LIDAR
Demo Hardware: Hokuyo 30LX
Working Principle: Time-of-flight using laser.
Advantages: Relatively low cost, simple data structure, high update rate, low processing requirements
Disadvantages: Primarily working in flat environments, harder to detect objects etc.
33
Simulated 2D Lidar
Actual 2D Lidar
3D LIDAR
Demo Hardware: Ouster OS-2
Working Principle: Time-of-flight, laser, note different wavelength of this product
Advantages: Full 3d information, can get image like information
Disadvantages: Cost, reliability (mechanical), processing point cloud.
34
PointPillars: Fast encoders for object detection from point clouds
Solid-State LIDAR
Demo Hardware: Velodyne Velarray
Working Principle: Time-of-flight, steer laser with solid-state components.
Advantages: Compact, no moving parts, range etc
Disadvantages: Field of view, availability, technical feasibility.
35
Working with Laser Scan Data
x
LIDAR on F1Tenth
37
50 mm
50 mm
70mm
Planar LIDAR
38
Transmitter
Receiver
Reflector
Distance to obstacle = (speed of light * time traveled)/2
Planar LIDAR
39
270o
10m range
Planar LIDAR
Working with the Hokuyo LIDAR…
40
LIDAR: ROS message structure
41
# Single scan from a planar laser range-finder��Header header # timestamp in the header is the acquisition time of � # the first ray in the scan.� �float32 angle_min # start angle of the scan [rad]�float32 angle_max # end angle of the scan [rad]�float32 angle_increment # angular distance between measurements [rad]��float32 time_increment # time between measurements [seconds] �float32 scan_time # time between scans [seconds]��float32 range_min # minimum range value [m]�float32 range_max # maximum range value [m]��float32[] ranges # range data [m]
(Note: values < range_min or > range_max should be discarded)
float32[] intensities # intensity data [device-specific units].
A simple scan sequence
42
TOP VIEW
Moving person
Whiteboard
What LIDAR “sees”
LIDAR
LIDAR: range data
Array A[1080]: A[i] is distance measurement of ith step.
Measurements beyond the min and max range (like inf) should be discarded.
43
LIDAR scan in hallway
44
Measuring Safety
x
Safety as an Indicator Function
46
Safe
Unsafe
Safe AEB with moving vehicles + successful evasion maneuver
No AEB triggered for stationary object in the driving path
From Binary Safety to Continuous Safety…
47
Safe
Unsafe
Marginal
Why not Euclidean Distance?
Pros: Easy to compute, have sensors which can measure accurately with well understood uncertainty.
48
Why not Euclidean Distance?
Cons: Classifies safe things as ‘near’ unsafe. Slam on the brakes when you are already stopped at T-junction? Tailgater etc...
49
Time-to-collision
TTC is defined as the time it would take for the ego-vehicle and an object to intercept one another given that they each maintain their current heading and velocity.
50
Image courtesy: Mathworks
Time-to-collision
TTC is defined as the time it would take for the ego-vehicle and an object to intercept one another given that they each maintain their current heading and velocity.
51
Time derivative of range between vehicle and object “range-rate”
Range between vehicle and object
Note:
for
Operator defined as:
How to Compute Range-rate
Range-rate is the time derivative of this distance which is simply computed by projecting the relative velocity of vehicle onto the vector between the vehicles and object’s poses…
52
Forward speed in the vehicles reference frame, this is simplified because we are not dealing with moving obstacles
Multiply by the cosine of the beams angle.
Planar LIDAR
Working with the Hokuyo LIDAR…
53
How to Compute Range-rate
If T is less than your acceptable threshold, apply the brakes…
54
Quiz: Sample Interview Question...
Describe a feature which captures the relative pose of all vehicles in the vicinity of the car.
55
Quiz: Sample Interview Question...
What if you that feature as input to a neural network and the number of vehicles changes, will this be a problem? Why or why not?
56
What’s Next?
x
Safety Lab Overview
Implement TTC based safety node in simulator.
Tune implementation to reduce false positives and avoid false negatives
Keyboard control - try to crash the car into the wall. AEB should engage
Demonstrate a crash-free lap with AEB activations
58
Safety Lab Overview
Implement TTC based safety node in simulator.
Tune implementation to reduce false positives and avoid false negatives
Demonstrate it on the F1Tenth race car
59
Next Lectures
Pose transformations
Map-less autonomous navigation
Build your car!
60