1 of 21

1

Artificial Intelligence Robots

Self-Driving Car and Comma Body

1

​

劉晉良�

Jinn-Liang Liu

清華大學動力機械工程學系

Department of Power Mechanical Engineering

National Tsing Hua University, Taiwan

�Nov 8, 2023

​

​

國立臺灣大學資訊網路與多媒體研究所

虛擬人與遙現課程

​

2 of 21

2

Motivation

3 of 21

3

Motivation

1989 CMU Vehicle Computer

2019 Tesla

2023 commaai

2023 Tesla (AI)

2023 commaai (AI)

2019 RbCup

2023 BD (non-AI)

BDAII (Raibert)

Musk

Hotz

4 of 21

E2E AI (Learning & Planning) System

跨域研究：電腦（機器學習、軟體工程、高效能計算、演算法、晶片韌體），數學（線性代數、統計、最佳化、數學建模），物理（相機、雷達、全球定位、衛星導航、慣性測量、車輛控制）

5 of 21

AI 聽說讀寫食衣住行育樂醫金研… a²bcd

AI is intelligence demonstrated by computers, as opposed to human or animal intelligence. "Intelligence" encompasses the ability to learn, to reason, to generalize, and to infer meaning. - - Wikipedia�人工智慧是電腦展現的智慧，而不是人類或動物的智慧。 “智慧”是擁有學習、推理、推廣、推測意圖的能力。

algorithm, big data, coding, deployment

6 of 21

Planning and Learning

Planning : y = f(x) = ax2 + bx + c, f : known

x : input, y : output, a, b, c : known

x, y : variables

Learning : y = f(x) = ax2 + bx + c, f : unknown

x : input, y : output, a, b, c : unknown

Learn a, b, c (regression parameters)

# of parameters : 1,000,000,000,000

F = ma

f(x, t)

y = wx + b

f(x, t)

6

⇒

⇒

⇒

⇒

Deep Learning (GTP4) #Wij ≈ 1.76 × 1012

Human Brain #Wij ≈ 100 × 1012

7 of 21

What and How to Learn?

7

Source: M. Gorner, Google

L = f1(x) = Wx + b

MNIST (train: 55000 imgs; validate: 10000; test: 5000) by Y. LeCun

8 of 21

Linear Regression: y = f1(x) = Wx + b

W: weights, b: biases

Learned W (Model)

8

9 of 21

9

openpilot by comma.ai

Source: comma con 2021

10 of 21

10

Car: Electronic Control Units (ECUs), Controller Area Network (CAN)

11 of 21

11

OP Architecture

12 of 21

12

​

E2E DNN by comma.ai�

12

​

PoseNet�

​

EffNet�

​

RNN�

​

Consumer

Reports 2020�

Total Params: 13M

Total MACs: 459M

GPT4/OP ≈ 105

13 of 21

13

Instance Segmentation DNN�

13

​

AlexN, GooLeN, MT-CNN, UNet, AttN

DeepLab, Yolo, Yolact�

14 of 21

14

Data

Taiwan & USA Data > 200 G

15 of 21

comma.ai Simulator

16 of 21

16

Control

Lateral and Longitudinal Controls: AI-Based Adaptive Cruise Control, PID, Kalman Filter, Model Predictive Control

Source: U Stuttgart

Source: A. Becker

Source: Wikipedia

17 of 21

17

E2E MT-CNN & Control Algorithms

D.-H. Lee, K.-L. Chen, K.-H. Liou, C.-L. Liu, J.-L. Liu, Deep learning and control algorithms of direct perception for autonomous driving, Applied Intelligence (2020)

D.-H. Lee, J.-L. Liu, End-to-end deep learning of lane detection and path prediction for real-time autonomous driving, Signal, Image and Video Processing (2022)

D.-H. Lee and J.-L. Liu, Multi-task UNet architecture for end-to-end autonomous driving, arXiv (2023)

18 of 21

18

Model B5

Step 1. Install Ubuntu and OP

Step 2. Prepare Data: fcamera.hevc => yuv.h5

Step 3. Create Model: modelB5.py

Step 4. Generate Data: yuv.h5 => serverB5.py, datagenB5.py

Step 5. Train Model: modelB5.py => trainB5.py => modelB5.h5

Step 6. Verify Model: modelB5.h5 => simulatorB5.py

Step 7. Install SNPE

Step 8. Deploy Model: modelB5.h5 =>  modelB5.pb => modelB5.dlc

19 of 21

19

Deployment

Qualcomm: Snapdragon Neural Processing Engine

20 of 21

20

Comma Body

Demos: Yolact, Car 1, Car 2, Car 3, Body

openpilot vs autopilot

127,855 US$ 81.5B

22 US$ 5.57M

21 of 21

21

Thank You

李德浩、陳冠霖、劉冠漢、劉昌倫

劉宜朋、李碧寒、黃奕棠、施威宇

唐旭蓮、陳思豪、李重岳、吳怡靜

OP-ai 開發研究群