CSE 5524: �Foundation of learning - 4

Homework assignment & midterm

• Homework – 1
• Released on 2/4 morning
• Due on 2/18 midnight

​

• Homework – 2
• Release around 2/14
• Due on 2/28 midnight

​

• Midterm
• 3/4 in class

Problem overview

Given

Goal: Generate

Y: 3D height

Z: 3D depth

Convention

• In this homework, given a map (or a matrix), say I
• I[i, j] means the i-th horizontal index (left-right) and j-th vertical index (bottom-up)
• i >= 0, j >= 0

i

j

For implementation, locations are indexed by [I, j]

Y: 3D height

Z: 3D depth

(i, j)

y: 2D vertical

Cue 1: edges (white pixels mean edges)

All edges

Contact edges

Vertical edges

Horizontal

You need to find edge locations!

Put all the constraints together

Least square solution!

For example,

A is like 2300-by-1681

Why linear system? Try this toy example

1

4

7

2

?

8

3

6

9

i

j

How to read a textbook?

• Check this one: How to Read a Paper https://web.stanford.edu/class/ee384m/Handouts/HowtoReadPaper.pdf

​

• Key takeaway
• Don’t do “linear” pass
• Skip math equations for the first time if you cannot grasp it right away
• Focus on its goal, not its detail
• Understanding the topic, context, and scope is more important than understanding the equations, at least in the first pass

My vision of this course

• You learn breadth and depth
• You read the textbook to expand your understanding and scope
• You are comfortable chatting with people about what computer vision is and what it can do

​

My experiences …

• If CV is what you are interested in …
• If CV is what you simply need credits …

​

​

Today

• Recap
• Neural networks overview (continued)
• Neural networks

​

​

11

Three tools in the search for Truth

• Data: what we observe

​

• Prior: what we prefer & believe

​

• Hypotheses: what the true function may be

​

​

Three tools in the search for Truth

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

More data, less over-fitting

[Figure: Bishop, PRML]

Green: true data distribution

Blue: training data

Red: learned model

Today

• Recap
• Neural networks overview (continued)
• Neural networks

​

​

15

General formulation for all these variants

​

​

​

​

​

​

​

​

16

Convolution

A special computation between layers

• A current node is not directly affected by “all nodes in the previous layer”
• The network “weights” on the edges can be “re-used”

17

Convolution

18

“Filter” weights

(3-by-3)

“Filter” weights

(3-by-3-by-“2”)

Inner product

Feature map (nodes) at layer t

Feature map at layer t+1

Convolution

19

“Filter” weights

(3-by-3-by-“2”)

Inner product

Feature map (nodes) at layer t

Feature map at layer t+1

One filter for one output “channel” to capture a different “pattern” (e.g., edges, circles, eyes, etc.)

Convolution: properties

• Process nearby pixels together
• Translation invariant: “local patterns” can show up at different pixel locations
• Can process arbitrary-size images

20

Top-left, Top right: has ears

Middle: has eyes

Convolutional neural networks (CNN)

21

Representative CNN networks

• AlexNet

[Krizhevsky et al., 2012]

​

​

• VGGnet

[Simonyan et al., 2015]

22

• A block: parameters/compute
• Edge: activation/tensors

Representative CNN networks

• GoogleNet [Szegedy et al., 2014]
• Inception

Representative CNN networks

• ResNet

[He et al, 2016]

​

​

​

• DenseNet

[Huang et al, 2017]

24

Advantages:

• Optimization
• Collect more information

• A block: parameters/compute
• Edge: activation/tensors

Representative CNN networks

A general architecture involves

• Multiple layers of convolutions + ReLU (nonlinearity) + pooling + striding
• These result in a (final) feature map
• Positions on the map correspond to the image
• The map goes through FC layers (MLP)
• Usually, we keep the network till the feature map
• For feature extraction
• For down-stream tasks
• For image-to-image search

25

Training a DNN for classification

26

Four factors behind deep learning developments

• Data

​

• Neural network architecture

​

• Powerful “learning” algorithms and loss

​

• Computational resource

Accessibility to large amount of data

28

Flexible neural networks for modeling

Visual transformers

[Liu et al., 2021]

[Battaglia et al., 2018]

Graph neural networks

[Qi et al., 2017]

PointNet

ConvNet

[Huang et al., 2017]

[Gu et al., 2024]

Recurrent neural networks

Powerful algorithms + losses to learn from data

Bi-level optimization

[Finn et al., 2017]

Adversarial learning

[Ganin et al., 2016]

[He et al., 2020]

Contrastive learning

Diffusion (denoising)

[Ho et al., 2020]

Autoregressive

[El-Nouby et al., 2024]

Preference learning

[Rafailov et al., 2023]

Computational resource

Today

• Recap
• Neural networks overview (continued)
• Neural networks

​

​

32

Deep neural networks (DNN)

Re-Introduction

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Perceptron

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Perceptron as classifiers

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Learning a classifier

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Multi-layer perceptron (MLP)

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Activations vs. Parameters

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Fast activation and slow parameters

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Why do we need activation?

Deep Nets

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Deep Nets

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Deep nets are universal approximator

• Deep or wider enough

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Deep learning: learning with neural nets

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Data structure

• Classification problem

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Data structure

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Layers

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Reading

• Read 12.7.3
• Read 12.7.4
• Read 12.8