1 of 17

Building Autonomous Vehicles:�On the Road and in the Sky

1

Sukrit Kalra, Le Xu, Ionel Gog, Joseph E. Gonzalez, Ion Stoica

Peter Schafhalter

@pschafhalter

2 of 17

2

What is the Current State of Autonomous Driving?

3 of 17

3

End-To-End Deadline

Localization

Prediction

Planning

Control

Perception

Autonomous vehicles execute complex computational pipelines that must finish execution within a deadline.

Cameras

LiDARs

GPS

1-2 GB / s

Image courtesy of Waymo Inc.

So, how does a self-driving car work?

Sensors such as cameras and LIDARS collectively generate ~1-2GB of data each second.

*CLICK*

The data from these sensors is passed to a complex computational pipeline consisting of hundreds of tasks with diverse resource requirements.

The pipeline processes this data to detect objects around the vehicle and predict their future trajectories. These trajectories are used to come up with a safe motion plan for the autonomous vehicle that is then converted to steering and acceleration commands.

In order to respond in a timely manner, this complex pipeline must finish execution by a specific *CLICK* end-to-end deadline.

<CLICK>

Although most existing research has focused on the models and algorithms that make the driving decisions, the focus of our work is the development of a system that orchestrates the execution of these complex pipelines.

4 of 17

4

Deadlines Change

The optimal deadline varies widely and depends on the driving environment.

1

2

3

4

5

6

7

8

9

10

Scenario

Driving Scenarios From Waymo Dataset

Time [s]

0

30

Detector 1

Detector 4

Detector 8

Accuracy

Runtime

10 ms

150 ms

No single optimum choice!

Images courtesy of CARLA.

5 of 17

5

Meeting Deadlines is Hard

The optimal deadline varies widely and depends on the driving environment.

Runtime of the components depends on the complexity of the environment.

Highway

Urban

Images courtesy of Waymo Inc.

0

200

400

600

Runtime [ms]

Number of obstacles around the vehicle

1

4

7

10

Increased Runtime

6 of 17

6

D3 Execution Model

Time

Deadline Policy

Environment

Deadline

Runtime

Accuracy

Runtime

Accuracy

Runtime

Accuracy

Execute reactive measures.

3

Adjust to available computation time.

2

1

Centralize deadline management.

End-To-End Deadline

Reactive

To solve these challenges, we came up with the D3 execution model which centralizes the management of deadlines by modeling missed deadlines as exceptions.

D3 abstracts the <CLICK> computation of the application as a directed operator graph, and <CLICK> centralizes the management of deadlines through a <CLICK> deadline policy that takes the state of the environment as the input and decides the end-to-end deadline.

This end-to-end deadline is then further split into <CLICK> individual deadlines and communicated to each of the operators.

The operators then <CLICK> adjust their computation and return the most accurate results possible within the given deadline. For example, given <CLICK> three implementations and a lax deadline, the components choose the <CLICK> implementation that takes the longest but provides the highest accuracy.

However, if due to runtime variability, the <CLICK> prediction component extends it runtime and misses its deadline, D3 notifies the planning component of the reduction in its available computation time allowing it to <CLICK> adjust itself to execute a lower accuracy but faster implementation such that the end-to-end deadline is still achieved.

In addition to adjusting downstream computation, D3 also models missed deadlines as exceptions and allows components to <CLICK> execute reactive measures that <CLICK> execute quickly in case of a missed deadline and release results to allow downstream components to minimize the loss in available computation time and execute <CLICK> higher accuracy implementations.

7 of 17

7

Contributions

D3

ERDOS

Pylot

Dynamic Deadline-Driven Execution Model

High Performance Implementation of D3

Modular Autonomous Driving Platform

D3: A Dynamic Deadline-Driven Approach for Building Autonomous Vehicles. EuroSys ’22.

Complete, open-source AV pipeline!

Pylot: A Modular Platform for Exploring Latency-Accuracy Tradeoffs in Autonomous Vehicles. ICRA ’21.

8 of 17

8

D3 Improves Safety

Number of

collisions

25

3.1x

0

80

Periodic

Data-Driven

D3

(Static Deadlines)

D3

40

2.3x

34

2.2x

36

78

Execution Model

Lower is better

70+km of challenging driving

9 of 17

9

D3 Improves Safety

10 of 17

10

D3 Improves Safety

11 of 17

11

D3 Improves Safety

12 of 17

12

What is the Future of Autonomous Driving?

13 of 17

13

Loose integration with cloud

14 of 17

14

A New Way of Driving

More information enables safer driving.

An obscured pedestrian is crossing the street.

Next-generation mapping: report traffic and road conditions in real time.

The road is blocked.

Snowy conditions.

Image courtesy of Google Maps.

Image courtesy of the San Miguel Sheriff’s Office.

Coordinate driving to benefit all vehicles on the road.

Wu et al. Flow: A Modular Learning Framework for Mixed Autonomy Traffic

Image courtesy of the US Department of Transportation.

Let’s imagine the benefits we can unlock if we use the information from a fleet of autonomous vehicles in real time.

*CLICK*

We can improve driving safety. Returning to the earlier scenario, *CLICK* the truck can notify the approaching vehicle about the obscured pedestrian crossing the street. Our preliminary experiments suggest that sharing information will result in further reductions in collisions.

*CLICK*

Moreover, AV sensors can report detailed traffic and road conditions in real time.

*CLICK*

By reporting the boulder on the road, navigation systems can rapidly re-route affected vehicles.

*CLICK*

Moreover, reporting the snowy conditions of the road can notify nearby vehicles to take proactive measures and reduce the chance of an accident.

*CLICK*

Finally, AVs can use the cloud to coordinate driving behaviors and make decisions that beneft all vehicles on the road. Behaviors such as platooning can reduce fuel consumption and improve traffic flow. Moreover, *CLICK* studies have suggested that even a small number of AVs sharing the road with human drivers can improve road utilization as a whole.

15 of 17

15

Tight integration with cloud

Image courtesy of CARLA.

16 of 17

16

Challenges

D3

ERDOS

Pylot

Low response times

Mitigate unreliable connections

State sharing

Data synchronization

Offload computation

Cooperation

Furthermore, we believe that we are well-positioned to solve the challenges of tightly integrating the cloud.

*CLICK* D3’s use of watermarks and timestamps provides tools for managing data synchronization.

*CLICK* Moreover, D3’s deadline mechanisms can be used to mitigate spikes in latency and lost connections, to ensure that the vehicle can seamlessly fall back to local computation.

*CLICK* Furthermore, we can use ERDOS’s state sharing to manage the sharing of geospatial information.

*CLICK* Likewise, ERDOS’s high performance implementation ensures that tight response time requirements can be met.

*CLICK* Using Pylot as a testbed, we can experiment with offloading resource-intensive algorithms to the cloud.

*CLICK* Pylot also allows us to experiment with cooperative algorithms, such as joint planning among several vehicles, which have the potential to improve driving safety and increase road utilization.

17 of 17

17

Questions

D3

ERDOS

Pylot

pschafhalter@berkeley.edu

erdos-project

Low response times

Mitigate unreliable connections

State sharing

Data synchronization

Offload computation

Cooperation