CSE 5539: �Machine Learning and Computer Vision for Perception in Autonomous Driving

Face-to-face policies

Face-to-face policies

• University COVID policies: https://safeandhealthy.osu.edu
• University FAQ on teaching under COVID: https://safeandhealthy.osu.edu/teaching
• University policy on expectations and accountability measures: https://safeandhealthy.osu.edu/accountability

​

• If you need to have a medical accommodation for COVID, please go to SLDS (Office of Student Life Disability Services) and request an accommodation. https://safeandhealthy.osu.edu/accommodations

​

Course information

• Course website:

https://sites.google.com/view/osu-cse-5539-sp22-chao/

(mainly for weekly schedule and reading update)

​

• Instructor: Prof. Wei-Lun (Harry) Chao (chao.209@osu.edu)
• Assistant professor in CSE (PhD: USC; Postdoc: Cornell)
• Office: DL 587

My research

Machine learning and its applications to

• Autonomous driving
• Computer vision
• Natural language processing
• Health care

​

​

​

​

My research

Learning with imperfect data sources

• Limited data
• Imbalanced data
• Inaccessible data
• Domain shifts

​

​

Questions?

Course information

• Lecture time: Wednesday, 1:00 pm – 2:45 pm
• We can take a 5-min bio break at 1:50 pm

​

• Office hours: TBA
• No office hour the first week
• How many of you prefer online office hours using Zoom? We can do in-person as well.

​

​

Course information

• Carmen Zoom:
• For office hours

​

• Piazza:
• piazza.com/osu/spring2022/cse5539chao (Access code: CSE5539)
• For discussion. For questions that are not personal, please first post them on Piazza rather than directly e-mailing me.

​

• Carmen:
• For announcement, posting course materials (slides), and assignment submission

​

​

Course information

• This research course will cover various "advanced" machine learning (ML) and computer vision (CV) topics and their applications to perception for autonomous driving (AD).
• We will study
• ML topics such as domain adaptation, meta-learning, generative models, imbalanced learning, semi-supervised learning, self-supervised learning, etc.
• CV topics such as object detection, instance/semantic segmentation, depth estimation, etc.
• How these techniques can be applied to perception problems in autonomous driving, which involves data captured by cameras, LiDAR, etc.
• The format of the class will be a mix of lectures and research paper presentations. Students who participate in this class are expected to be self-motivated graduate or senior undergraduate students.

Course information: Pre-requisites

• Students are expected to have a decent degree of mathematical sophistication and to be familiar with linear algebra (Math 2568), multivariate calculus, probability, and statistics. Students are also expected to have knowledge of algorithm design and data structures. Students are expected to be able to code in Python.
• Students are expected to (self) learn deep learning software (e.g., Tensorflow and Pytorch).
• Students are expected to have a strong interest in machine learning, computer vision, and their applications in autonomous driving
• Students are expected to have taken courses in artificial intelligence/machine learning (3521/6521, 5523, or 5526) and/or computer vision (5524)

​

Course information

• This is not an introductory deep learning, machine learning or computer vision course.
• Instead, the course will be focusing on recent techniques, and you will need to read many papers (or tutorial slides). The goal is to prepare you essential knowledge and ability to conduct research in machine learning, computer vision, and autonomous driving.
• The course involves lots of paper reading and thinking!
• The course has a heavy-loading final project.

​

​

Grading and work

 Grading (tentative)

• Participation: 5%
• attendance, asking questions, discussion
• Paper presentation, survey (2 people): 45%
• 30%: presentation; 15%: survey
• based on efforts and clearness in presenting the ideas of the papers.
• based on efforts, clearness, and how well you organize the papers that you read.
• Final project (1-3 people): 50%
• 10%: first presentation
• 25%: final presentation (results)
• 15%: report

​

Paper presentation & survey

• Presentation: 2-3 papers for 50 mins each time
• Survey (must by Latex): Extend to 6-10 papers
• At least 8 pages, excluding reference
• Please include Introduction, an overview of the background, descriptions of some key algorithms and their concepts, and important experimental results and findings
• Due day: 2 weeks after your presentation. Late reports will lead to an immediate 30% deduction on your report scores. If your report is late by a week, the deduction will become 60%. If your report is late by two weeks, you will get 0 points.

Grading and work

Final project (tentative 3 directions)

• Must be related to ML, CV, and AD
• Comprehensive topic studies: A team selects a topic and performs a comprehensive survey of the techniques, datasets, and evaluation metrics. Then, the team is going to re-implement those techniques and re-evaluate on all the datasets with all the metrics. Students are encouraged to propose new techniques and experimental setups.
• Competition & challenges: A team selects a competition & challenge held in top conferences and participates in it.
• Self-picked research topic: A team selects a research topic. The expectation is to be ready to submit to a conference (e.g., NeurIPS 2022).

Computational resource

• You will be able to access the computational resource of OSC

Tentative schedule

Paper presentation sign-up

• In the 2^nd or 3^rd week
• Presentation will start in the 4^th or 5^th week
• Please prepare to form a team of two people

​

Final project

• First presentation: late February
• Final presentation: 4/20 and 4/28 (final exam time)
• Final report: TBA, must be two days before the grade posting date

Policy

Academic integrity

• Plagiarism and other unacceptable violations
• Zero tolerance
• Please study the related sections in the syllabus (pdf) on academic integrity.

Questions?

Suggested reference

​

• Not required, but for students who want to read more, we recommend

​

​

​

​

​

​

​

​

​

​

​

​

​

​

Pattern Recognition and Machine Learning

Deep Learning

Understanding Machine Learning

Suggested reference

Learning from Data

Foundations of Machine Learning

Machine Learning:

A Bayesian and Optimization Perspective

The Elements of Statistical Learning

Machine Learning: A Probabilistic Perspective

Neural Networks and Deep Learning

Suggested reference

Introduction to Machine Learning

Machine Learning Refined

Suggested reference

• Computer Vision for Autonomous Vehicles: Problems, Datasets and State of the Art. Foundations and Trends in Computer Graphics and Vision, 2020

​

​

​

​

​

​

• Self-driving cars: A survey. Expert Systems With Applications, 2021

​

• Tutorials & workshop talks in CVPR, ICCV, ECCV, ICML, ICLR, and NeurIPS

​

• The Matrix Cookbook

​

Suggested reference

• Exemplar tutorial: All About Self-Driving
• https://cvpr2021.waabi.ai/

​

• Exemplar tutorial: New frontiers in data-driven autonomous driving
• https://www.self-driving-cars.org/

​

• Exemplar tutorial: Visual Recognition and Beyond
• http://feichtenhofer.github.io/cvpr2019-recognition-tutorial/

​

​

Other excellent courses

• Stanford CS 229: http://cs229.stanford.edu/

​

• Stanford CS 231n (deep learning for computer vision): http://cs231n.stanford.edu/

​

• Cornell ML: https://www.cs.cornell.edu/courses/cs4780/2018fa/

​

• Cornell CV:
• https://www.cs.cornell.edu/courses/cs6670/2021fa/
• https://www.cs.cornell.edu/courses/cs4670/2021sp/

​

Important for this week

• Register. If you are on the waitlist, you might or might not get in depending on how many empty seats or how many students drop.
• Register for the class on piazza --- our main platform for discussion and communication
• ML review and CV reading: extremely important to check your readiness for the course
• Python: check suggested tutorials on the website
• Decision: stay or drop

​

• Office hours: start next week

​

How to do/learn Well?

• Lecture and lecture slides for basics

​

• Paper reading and discussion is the key part

​

• Presentation: learn to organize papers, ideas, and insights, and learn to criticize

​

• Survey: learn to write a formal paper

Questions?

Today

​

Introduction

• Who are you?
• What is ML, CV, AD?
• Big picture of the course

​

Machine learning review

​

Who are you?

How to read papers?

• How to Read a Paper: https://web.stanford.edu/class/ee384m/Handouts/HowtoReadPaper.pdf

​

​

What is ML?

What is machine learning?

This book is about learning from data.

​

Sergios Theodoridis. Machine learning: a Bayesian and optimization perspective.

We choose the title “learning from data” that faithfully describes what the subject is about.

​

Y. Abu-Mostafa, M. Magdon-Ismail, H-T Lin. Learning from data.

Machine Learning Overview

• What is machine learning?

​

Learning from Data

​

​

Machine Learning Overview

• What is machine learning?

​

Learning from Data

​

​

Algorithm

Data

​

Evaluation

Machine Learning Overview

• What is machine learning?

​

Learning from Data

​

​

Algorithm

Data

​

Evaluation

Goal

Example: coin classifier

Key ingredients

• Data: collected from past observations (we often call them training data)

​

• Modeling: devised to capture the patterns (or knowledge) in the data
• The model does not have to be true --- if it is close, it is useful
• We should tolerate randomness and mistakes --- many interesting things are stochastic by nature.

​

• Prediction: apply the model to forecast what is going to happen in future

What is deep learning (deep neural networks)?

image

label

Classifier

Example: image classification

Example: image classification

The progress of deep learning

The progress of deep learning

Visual transformers

[Liu et al., 2021]

[Battaglia et al., 2018]

Graph neural networks

[Qi et al., 2017]

PointNet

[Zoph et al., 2017]

Neural architecture search

The progress of deep learning

Meta-learning

[Finn et al., 2017]

Adversarial learning

[Ganin et al., 2016]

[He et al., 2020]

Contrastive learning

How can we handle all these variations?

Neural networks can sometimes be simplified as a learnable function.

​

​

​

​

​

​

​

​

Challenges in machine learning

• How to collect data? How many data do we need?
• How to design the neural network “architecture” for the problem at hand?

​

​

​

​

​

​

​

• How to learn the neural network “parameters” from the data?

​

​

​

​

​

​

Each block here is a bunch of digits!