1 of 108

1

EC 500 A1: �Foundations of Computer Vision

2 of 108

Course information

Course website:

https://sites.google.com/view/bu-ec-500-au26-chao

(for course information, weekly schedule, and reading assignment updates)

Instructor:

Dr. Wei-Lun (Harry) Chao (chao209@bu.edu), Office: PHO437

Associate professor in ECE (PhD: USC; Postdoc: Cornell)

TA:

Sanjana Sanjeev Kumar (sanjask@bu.edu), CS MS student

2

3 of 108

Machine learning, computer vision, and applications in

3

4 of 108

Machine learning, computer vision, and applications in

4

Application-inspired ML & CV

ML & CV for applications

5 of 108

Learning with “imperfect” data

Limited data and supervision
Imbalanced data
Inaccessible data
Domain shifts

5

[Zhu et al., 2014]

KITTI

(Germany)

Argoverse

(USA)

nuScenes

(USA, Singapore)

Lyft

(USA)

Waymo

(USA)

[Wang et al., 2020]

6 of 108

Course information

Lecture time: Monday and Wednesday, 12:20 PM - 2:05 PM

Office hours: PHO437

Wednesday 2:10 – 3:00 PM or by appointment
No office hours the first & second weeks

TA Office hours: TBA

TBA
No office hours the first week

7 of 108

Course information

Blackboard/Gradescope/GitHub:

For announcements, posting course materials (slides), and homework submission
Caution! We won’t read the messages on Blackboard

Piazza:

For discussion. Please register!
Link: https://piazza.com/bu/fall2026/ec500a1
Please use the BU email
Access code: ec500fall2026

Detailed syllabus (pdf):

Can be found on Blackboard and the course website

7

8 of 108

Communications

Schedule and readings will be updated on the website
Announcements will be made through Blackboard
Discussions and questions must be posted in Piazza
Please use email only to contact me or the TA for urgent or personal issues. Please include the tag "[BU-EC-500]” in the subject line.
More details: See the website, Blackboard, and the syllabus

8

9 of 108

9

Questions?

10 of 108

Grading and homework (tentative)

Grading (subject to slight change)

In-class quiz – 20% (including 4% linear algebra)
Homework – 30%
Midterm (October 28, in class) – 20%
Final project – 30%
The final project consists of multiple parts:

proposal sketch
proposal
milestone report
project presentation (December 16, scheduled final exam time)
project report

Guidelines

Expect around 6 homework assignments (including problem and programming sets)

Solutions may involve derivations. Grading is based on correctness and clarity. Be concise and show your reasoning in a clear and precise way.
Homework completion and submissions are individual, but feel free to discuss. You must strictly follow the submission instructions.
NOT ALLOWED: ask/search for solutions
Late day policy strictly applies.

11 of 108

Final project first glance (subject to change)

Team forming:

2 – 3 students: same expectations
MUST be a team project

Steps:

Team forming in late September
Project sketch: 1 page at most: what you plan to do, who your teammates are
Project proposal: 2 – 3 pages
Project milestone report: : 2 – 3 pages
Project presentation: 7 – 10 minutes
Project report & code release: academic paper format (e.g., NeurIPS); LaTeX is required

12 of 108

Final project first glance (subject to change)

Pre-defined tasks:

CVPR 2026 competition: https://cvpr.thecvf.com/Conferences/2026/Workshops

Keywords: competition & challenge
Example: https://sites.google.com/view/fgvc13/competitions

Improvement over an existing paper
Benchmarking existing algorithms
Instructor-provided tasks

Self-defined “research” tasks:

Need approval
Need justification (not your lab’s work)

13 of 108

Schedule

In-class quizzes (linear algebra)

Monday, 9/14
Policy:

20 minutes at the beginning of class
All multiple-choice questions
2 cheat sheets (4 pages) are allowed
4% of the semester
Exemplar questions will be provided by Sept. 9

In-class quizzes

Starting on Wednesday, 9/16
Content:

The last lecture
The last homework (after its submission)

Policy:

5 minutes at the beginning of class
No cheat sheets are allowed
There will be around 20 quizzes
The 16 highest scores will be counted

14 of 108

Schedule

Homework

Release dates: subject to change

9/14
9/28
10/12
10/26
11/9
11/23

You will have 2 weeks to complete each homework assignment
Due on Monday at 23:59 ET

Exams & final project presentation

Midterm date(s): 10/28/2026
Final project presentation: 12/16/2026
Other final project-relevant deadlines will be announced soon.

15 of 108

Policy

Academic integrity

Plagiarism and other unacceptable academic misconduct and violations (https://www.bu.edu/academics/policies/academic-conduct-code/)

Zero tolerance
I MUST report incidents

Please study the related sections at the end of the syllabus (pdf) on academic integrity.
More on AI policy to be announced

(Re-)grading

Only factual errors will be corrected.
Request:

one week within the release of your homework grade
during the after-exam discussion day

Format: TBA

16 of 108

AI Policy for the course

To be announced soon, before the first homework

AI and Academic Conduct. https://www.bu.edu/academics/policies/academic-conduct-code/

AI Development Accelerator (AIDA) Resources: https://www.bu.edu/aida/ai-education/

16

17 of 108

17

Questions?

18 of 108

Pre-requisites & what to expect?

Prerequisites (undergraduate)

ENG EK 130 (Python), ENG EK 103 (linear algebra), ENG EK 381 (probability & statistics), ENG EC 414 (machine learning), ENG EC 401 (signals and systems)
Consent of the instructor

Prerequisites (graduate)

Graduate prerequisites: No prerequisites, but prior attendance of the following courses is a plus.
ENG EC 503 (learning from data), ENG EC 520 (digital image processing), ENG EC 523 (deep learning)

General background requirements

Decent degree of mathematical sophistication
Knowledge of programming, algorithm design, and data structures

Math and programming-related homework

Multivariate calculus, linear algebra, and probability
Python 3
PyTorch & Hugging Face

CV algorithms are often difficult to debug

We strongly recommend that you start early, for both the homework and the final project.

18

19 of 108

Review materials

Please see the website: https://sites.google.com/view/bu-ec-500-au26-chao

Please see the reading list in the spreadsheet on the website

Linear algebra review slides: https://drive.google.com/drive/folders/10jgSkiCbFEm5XArjSCea6nH19Y53UBai

4-point in-class quizzes related to linear algebra

19

20 of 108

Caution!

20

This is a 500-level course!

21 of 108

Caution!

21

One of the aims is to provide students with a strong foundational background, enabling them to pursue computer-vision-centered or machine-learning-centered MS/PhD paths or explore future opportunities in the computer vision, machine learning, and artificial intelligence industries.

This is a 500-level course!

22 of 108

Caution!

22

The course is not simply knowledge feeding, and I will leave space for you to read, think, and explore!

This is a 500-level course!

23 of 108

23

Questions?

24 of 108

Course descriptions & goals

Course Description: Computer vision algorithms for use in human-computer interactive systems; image formation, image features, image processing, object recognition, image generation, 3D from images, and applications.

This course focuses on the foundations of computer vision, with particular emphasis on learning-based methods and 3D. To build background, the course covers the basics of image formation, camera modeling, machine learning, and neural networks. With this groundwork, the course introduces image-processing-based methods and probabilistic models of images. Then, the course explores modern neural network architectures for computer vision, including convolutional neural networks and transformers. The course then builds upon these models to develop algorithms for image feature extraction, visual recognition, image generation, and vision-and-language understanding. Moving beyond single images, the course further introduces stereo vision and multi-view vision, including structure from motion and neural radiance fields. Finally, the course introduces algorithms for motion estimation and tracking. Along with the course, representative applications of computer vision will be introduced and discussed.

24

25 of 108

Course descriptions & goals

Course Goals / Objectives:

Master fundamental and recent computer vision concepts and algorithms
Be competent with computer vision application design and evaluation
Gain a deep understanding of learning-based algorithms and 3D inference for computer vision
Be exposed to original research and applications in computer vision
Be familiar with the Python/PyTorch programming environment
More broadly, the aim is to provide students with a strong foundational background, enabling them to pursue computer-vision-centered or machine-learning-centered MS/PhD paths or explore future opportunities in the computer vision, machine learning, and artificial intelligence industries.

25

26 of 108

Textbook

Required

https://mitpress.mit.edu/9780262048972/foundations-of-computer-vision/
Purchasable at the MIT Press Bookstore or Amazon
eBook accessible through: https://visionbook.mit.edu/
This course is enrolled in the BU First Day program!

26

Foundations of Computer Vision

27 of 108

First Day^®

ACCESS

CONVENIENCE

AFFORDABILITY

27

Benefits of Inclusive Access:

• Lower price than traditional purchase

• Guaranteed to get the right materials for your course

• Seamless digital access

• Option to opt out before deadline

28 of 108

28

Exclusive Preferred Pricing Through First Day^®

Your First Day Price	$54.85

ENG EC 500

FOUNDATIONS OF COMPUTER VISION

*This course material charge will be applied to your student account in October.

29 of 108

Opt Out Deadline: September 22nd

29

If you do not wish to participate in the program, you can choose to opt out within Blackboard by September 22nd. Opting out is not recommended, and you will be responsible for purchasing your required materials without preferred pricing. To opt out, use the “Course Materials” link in Blackboard.

If you do not opt out by the deadline, you have agreed to purchase these materials, and the cost will be charged to your student account at the price listed.

30 of 108

Suggested References

30

Generative Deep Learning:

Teaching Machines To Paint, Write, Compose, and Play

(second edition)

Computer Vision: Algorithms and Applications

(second edition)

31 of 108

Other great textbooks

31

Deep Learning:

Foundations and Concepts

Understanding Deep Learning

Dive into Deep Learning

PDF accessible for the 1^st and 3^rd books – check their websites

32 of 108

Other excellent CV courses

MIT

Stanford

CS 131: http://vision.stanford.edu/teaching/cs131_fall2223/
CS 231n: http://cs231n.stanford.edu/

CMU: http://16385.courses.cs.cmu.edu/spring2024/

Brown CV: https://browncsci1430.github.io/

32

33 of 108

Other excellent CV courses

NYU CV: https://www.sainingxie.com/cv-fall2024/

Wisconsin-Madison CV: https://sites.google.com/view/cs639spring2023dlcv

Michigan CV

Computer vision courses are hard to be comprehensive and unified

3D vision, generative vision, deep learning for vision, robotic vision, etc.
Even the basic CV courses can be very different

33

34 of 108

How to do/learn well?

Lecture and lecture slides for basics

Describe basic concepts, tools
Describe algorithms and their development with intuition and rigor

Textbook reading for completeness and extension

Homework for practice, generalization, and implementation

Final project for thinking, exploration, implementation, and integration

Discussion (Piazza, office hours) for further understanding

34

35 of 108

Semi-flipped classroom approach

You don't need to watch a video of the lecture or read the entire chapters before coming to class.

I suggest you take a quick overview of the chapters/topics before coming to class, like spending 15~30 minutes.

Why this approach? Computer vision is quite broad, containing many topics, and some of them are very different. Taking a "semi" flipped classroom approach would help you warm up before coming to class.

35

36 of 108

How to read a textbook?

Check this one: How to Read a Paper https://web.stanford.edu/class/ee384m/Handouts/HowtoReadPaper.pdf

Key takeaway

Don’t do “linear” pass
Skip math equations for the first time if you cannot grasp it right away
Focus on its goal, not its detail
Understanding the topic, context, and scope is more important than understanding the equations, at least in the first pass

37 of 108

My vision for this course

You learn breadth and depth
You read the textbook to expand your understanding and scope
You are comfortable chatting with people about what computer vision is and what it can do

My suggestions …

If CV is what you are interested in … Take it!
If CV is what might be useful in the future … Take it!
If CV is what you simply need credits … Don’t take it!

38 of 108

Important for this week

Register. If you are on the waitlist, you might or might not get in, depending on how many empty seats or how many students drop.
Register for the class on Piazza --- our main platform for discussion and communication
Math review/self-diagnostic: do Homework #0 and check suggested materials on the website (e.g., linear algebra slides) --- extremely important to check your readiness for the course
Python: check suggested tutorials on the website
Decision: stay or drop

Office hours: start next week (TA) and two weeks later (instructor)

38

39 of 108

Important dates

Midterm: in class (date: 10/28/2026)

Final project presentation: in class

12/16/2026, 12 – 2 pm
We may potentially extend it to accommodate more presentations

Online/pre-recorded teaching or guest lectures

For some weeks, I may be traveling, such as 9/9, 12/7, and 12/9

39

40 of 108

40

Questions?

41 of 108

About using AI tools

41

https://www.nature.com/articles/s44222-025-00323-4.pdf

42 of 108

Writing is thinking

I “love” writing papers because it gives me a chance to rethink what I’m doing, identify logical holes, and uncover implications we miss.

Writing rebuttals is painful, but I keep doing it because it helps sharpen my skills to convince people and debate.

42

43 of 108

Writing is thinking

43

44 of 108

About math

Groundbreaking ideas in computer vision often come from math and physical concepts or insights

44

45 of 108

About math

Groundbreaking ideas in computer vision often come from math and physical concepts or insights

45

46 of 108

About linear algebra

46

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

47 of 108

47

Questions?

48 of 108

Today

Introduction

What is computer vision?

Course overview

48

49 of 108

49

What is computer vision?

50 of 108

What is computer vision?

Human vision is capable of extracting information about the world around us using only the light that reflects off surfaces in the direction of our eyes.

Our eyes are sensors. Our brains have to translate the information collected by millions of photoreceptors in our retinas into an interpretation of the world in front of us.

Computer vision studies how to reproduce in a computer the ability to see

Antonio Torralba, Phillip Isola, and William T. Freeman, Foundations of Computer Vision, 2024.

50

51 of 108

Input: the structure of ambient light

51

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

52 of 108

Output: measuring lights vs. scene properties

52

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

53 of 108

The study of vision is interdisciplinary

involving many disciplines (physics, phycology, biology, neuroscience, art, and computer science)

53

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

54 of 108

The study of vision is interdisciplinary

Gestalt phycology grouping rules for perceptual organization

54

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

55 of 108

Visual pathways

55

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

56 of 108

56

Questions?

57 of 108

What is computer vision?

Computer vision studies how to reproduce in a computer the ability to see

Antonio Torralba, Phillip Isola, and William T. Freeman, Foundations of Computer Vision, 2024.

57

Vision is the process of discovering from images what is presented in the world, and where it is.

David Marr, Vision A Computational Investigation into the Human Representation and Processing of Visual Information, 1982.

58 of 108

Computer vision

58

[Source: Detectron2]

A computer sees the world through sensors, which generate images, videos, point cloud, etc.

[Source: Graham Murdoch/Popular Science]

59 of 108

Computer vision: data

59

Image (s)

Video (s) = sequence of images

RGB image (s): Three matrices

60 of 108

Computer vision: data

RGB images:

What is inside each matrix?

{0,1,……,255}
Interval: [0, 1]

60

124	255	125
125	126	60
126	60	126
0	127	60
0	0	128

124	255	125
125	126	60
126	60	126
0	127	60
0	0	128

124	255	125
125	126	60
126	60	126
0	127	60
0	0	128

61 of 108

Computer vision: data

61

Image (s)

Video (s) = sequence of images

RGBD image (s): Four matrices

Entry value

= depth

62 of 108

Computer vision: data

62

Point cloud

A collection of 3D (or 4D) points

x coordinate
y coordinate
z coordinate
reflectance

N points = 3-by-N or 4-by-N matrix

63 of 108

Computer vision: data

63

Image aligned with point cloud

64 of 108

LiDAR-based vision

64

[Source: Graham Murdoch/Popular Science]

LiDAR:

Light Detection and Ranging sensor
accurate 3D point clouds of the environment, centered at the ego-car

65 of 108

LiDAR-based vision

A point cloud is formed by LiDAR responses within a short time period

65

[Credits: Lisa Wu’s presentation]

66 of 108

66

Questions?

67 of 108

What is computer vision?

Computer vision studies how to reproduce in a computer the ability to see

Antonio Torralba, Phillip Isola, and William T. Freeman, Foundations of Computer Vision, 2024.

67

Vision is the process of discovering from images what is presented in the world, and where it is.

David Marr, Vision A Computational Investigation into the Human Representation and Processing of Visual Information, 1982.

68 of 108

Three representation directions

68

S: scene

I: image

2: Reconstruction

1: Recognition

tree

3: Generation

tree

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

69 of 108

Computer vision: representative tasks

69

70 of 108

Computer vision: representative tasks

71 of 108

Computer vision: representative tasks

71

Retrieval, image-to-image search

72 of 108

Computer vision: representative tasks

72

Depth estimation and 3D reconstruction

73 of 108

Computer vision: representative tasks

73

74 of 108

Computer vision: representative tasks

74

Style transfer

[Figure credit: CycleGAN, ICCV 2017]

75 of 108

75

Questions?

76 of 108

What is computer vision?

Computer vision studies how to reproduce in a computer the ability to see

Antonio Torralba, Phillip Isola, and William T. Freeman, Foundations of Computer Vision, 2024.

76

Vision is the process of discovering from images what is presented in the world, and where it is.

David Marr, Vision A Computational Investigation into the Human Representation and Processing of Visual Information, 1982.

77 of 108

How to let computers recognize objects?

A cat?

A lion?

A car?

Percept:

See a picture

Action:

Tell the object class

78 of 108

Human design vs. machine-learning-based

cat

Design

cat

Data

collection

“Learn”

“Coding” the rules:

Can you list the rules of recognizing a cat?

Underlying idea:

Humans sometimes are good at “making decisions” BUT are not good at “explaining decisions”.

79 of 108

Learning-based computer vision

79

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

80 of 108

What is machine learning?

This book is about learning from data.

Sergios Theodoridis. Machine learning: a Bayesian and optimization perspective.

We choose the title “learning from data” that faithfully describes what the subject is about.

Y. Abu-Mostafa, M. Magdon-Ismail, H-T Lin. Learning from data.

81 of 108

Machine Learning Overview

What is machine learning?

Learning from Data

81

82 of 108

Machine Learning Overview

What is machine learning?

Learning from Data

Algorithm

Data

Evaluation

82

83 of 108

Machine Learning Overview

What is machine learning?

Learning from Data

Algorithm

Data

Evaluation

Goal

83

84 of 108

Example: coin classifier

84

Machine learning algorithms

Training data

Learned models

Test data

[Figure credit: Y. Abu-Mostafa, M. Magdon-Ismail, H-T Lin. Learning from data.]

85 of 108

What is deep learning (deep neural networks)?

Image

Label (e.g., dog or cat)

Classifier

See a picture

Tell the object class

A sequence of “learnable” computation!

86 of 108

Example: image classification

86

[Gif credits: Gradient descent 3Blue1Brown series S3 E2]

A sequence of “learnable” computation!

87 of 108

The progress of deep learning

[Simonyan et al., 2015]

[Szegedy et al., 2015]

[Huang et al., 2017]

[He et al., 2016]

[Krizhevsky et al., 2012]

88 of 108

The progress of deep learning

Visual transformers

[Liu et al., 2021]

[Battaglia et al., 2018]

Graph neural networks

[Qi et al., 2017]

PointNet

[Zoph et al., 2017]

Neural architecture search

89 of 108

89

Questions?

90 of 108

Today

Introduction

What is computer vision?

Course overview

90

91 of 108

Topics

1. Introduction to computer vision

a. Introduction to the course

b. A simple vision system

91

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

92 of 108

Topics

2. Image formation

a. Concepts of imaging and lenses

b. Images and 3D geometry

c. Camera modeling

d. Cameras as linear systems

92

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

93 of 108

Topics

3. Foundations of image processing

a. Linear filtering and convolution

b. Fourier analysis

c. Blur filters, image derivatives, and filter banks

d. (Up/down) sampling

e. Image pyramids

93

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

94 of 108

Topics

4. Foundations of learning

a. Introduction to learning

b. Gradient-based learning algorithms

c. Generalization

d. Neural networks as distribution transformers

94

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

95 of 108

Topics

5. Probabilistic models of images

a. Color

b. Statistical image models

c. Textures

95

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

96 of 108

Topics

6. Neural architectures for vision

a. Convolutional neural nets

b. Transformers

96

Visual transformers

[Liu et al., 2021]

[Simonyan et al., 2015]

ConvNet (VGG Net)

97 of 108

Topics

7. Generative image models and representation learning

a. Representation learning

b. Generative models

97

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

98 of 108

Topics

8. Understanding vision with semantics and language

a. Visual recognition

b. Vision and language

98

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

99 of 108

Topics

9. Challenges in learning-based vision

a. Data bias and shift

b. Robustness and generality

c. Transfer learning and adaptation

99

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

100 of 108

Topics

10. Understanding geometry

a. Stereo vision

b. Homographies

c. Depth estimation from single images

d. Feature detection and matching

e. Multi-view geometry and structure from motion

f. Radiance fields

100

101 of 108

Topics

11. Understanding motion

a. Motion estimation

b. Optical flow estimation

c. Object tracking

101

102 of 108

Topics

12. Advanced topics

a. Exciting 3D & 4D models

b. Computer vision for biodiversity

102

[Figure credit: Jianyuan Wang, VGGT, CVPR 2025.]

103 of 108

TODO

See the beginning slides and the course website for suggested reading
Background review: math and programming

103

104 of 108

Today

Preview of Wednesday, Sept. 9

104

105 of 108

Neural networks for image classification

105

[Gif credits: Gradient descent 3Blue1Brown series S3 E2]

A sequence of “learnable” computation!

106 of 108

Learning neural networks for image classification

107 of 108

Pre-trained network (from GitHub, Huggingface)

[Very Deep Convolutional Networks for Large-Scale Image Recognition, ICLR 2015]