CSE 5524: �A Simple Vision System (Cont.)�& Image Formation

Course information, grading, reading, policy, etc.

• Please check slide decks 1 & 2, course website, Carmen, and syllabus.

​

• Course website:

https://sites.google.com/view/osu-cse-5524-sp25-chao/home

​

• Office hours start this week!
• Dr. Chao (DL587): Tuesday 3 – 4 pm & Friday 9 – 10 am
• Zheda Mai (BE 406): Monday 11 am - 12 pm & Wednesday 2 pm - 3 pm

​

• Linear algebra quizzes: released last week, due 9/16

2

Today

​

• Recap and continuation: a simple vision system
• Image formation

3

Three representation computer vision sub-fields

4

S: scene

I: image

2: Reconstruction

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

A simple world: the blocks world

What are inside?

• Simple but varied set of objects
• Flat horizontal or vertical surfaces
• White horizontal ground plane

​

Image formation assumptions

• Parallel (orthographic) projection

​

​

​

​

​

​

​

5

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

[Figure credit: https://www.geeksforgeeks.org/parallel-othographic-oblique-projection-in-computer-graphics]

Our goal: recover the world coordinate of all pixels

We want to know X(x, y), Y(x, y), and Z(x, y) from the given image!

What we know:

​

​

We need some cues from images and the 3D world!

​

​

​

​

6

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Our goal: recover the world coordinate of all pixels

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

(x, y)

Reconstructed 3D worlds from other views

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Reconstructed 3D worlds from other views

Depth estimation and 3D reconstruction

Questions?

Before we dive into details, let’s take a step back

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Can you infer the 3D information from 2D?

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Can you write down what you just said by math/code?

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Cue 1: edges

• Edges: image regions with strong color/intensity changes w.r.t. location

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Cue 2: Surfaces & Cue 3: properties from 3D to 2D

• Separate into foregrounds (figures)/backgrounds

​

​

​

​

​

​

• Not always true, but let’s assume it is true
• Vertical in 3D will project to vertical in 2D; thus, vertical in 2D mean vertical in 3D
• Non-vertical in 2D means horizontal in 3D

​

​

​

​

​

​

​

​

Our goal: recover the world coordinate of all pixels

We want to know X(x, y), Y(x, y), and Z(x, y) from the given image!

​

​

​

​

16

Our goal: recover the world coordinate of all pixels

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Questions?

How to represent the “3D height map” Y?

How to represent the “3D height map” Y?

• Vectorization: matrix to vector

​

​

​

​

​

​

​

How to estimate the “3D height map” Y?

Reconstruction

• Known equations
• Cues from edges, surfaces, and 2D/3D relationships

Cues encoded by linear equations

Estimating Y(x, y) from the input image

Y

Estimating Y(x, y) from the input image

Y

Estimating Y(x, y) from the input image

Y

Estimating Y(x, y) from the input image

Y

Horizontal edges: Y won’t change along the edge

= 0

Estimating Y(x, y) from the input image – horizontal edges

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Estimating Y(x, y) from the input image

Y

Horizontal edges: Y won’t change along the edge

= 0

Estimating Y(x, y) from the input image

Y

Surfaces: flat, not curved

Questions?

Information propagation via “optimization”

• All the information:
• 3D vertical edge:
• 3D horizontal edge:
• Flat surfaces:

​

​

​

​

​

• Contact edges & background: Y = 0

​

​

= 0

Information propagation

Information propagation

Information propagation via “optimization”

• All the information:
• 3D vertical edge:
• 3D horizontal edge:
• Flat surfaces:

​

​

​

​

​

• Contact edges & background: Y = 0
• They can be rewritten as an overdetermined system of linear equations

​

​

​

= 0

Information propagation via “optimization”

Least square solution!

Results

Reconstructed 3D worlds from other views

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Caution

• The approach we developed so far does not always work.
• We have made lots of assumptions.

​

​

​

​

​

​

• Still, it gives some sense of the scope and challenge of computer vision.

​

​

​

​

​

Reading & keywords

• Chapter 2

​

• Image intensity
• Edge, shadow edges
• Surface
• Linear system, Overdetermined system of linear equations
• Least square solutions
• 3D reconstruction
• Parallel and perspective projection

​

HW1

• Release tonight or Thursday night!

Questions?

What is 3D reconstruction nowadays?

What is 3D reconstruction nowadays?

https://depth-anything-v2.github.io/

What is 3D reconstruction nowadays?

https://vgg-t.github.io/

How to let computers recognize objects?

A cat?

A lion?

A car?

Human design vs. machine-learning-based

“Coding” the rules:

Can you list the rules of recognizing a cat?

Underlying idea:

Humans sometimes are good at “making decisions” BUT are not good at “explaining decisions”.

Today

​

• Recap and continuation: a simple vision system
• Image formation

47

Goal

• How are images formed?
• How can light illuminating the space be captured by a device to form a picture?

​

“Visible” light interacting with surfaces

• Light:
• Wave (with wavelength, frequency)
• Light ray – specified by position, direction, and intensity, as a function of wavelength and polarization

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Lambertian surfaces

• Bidirectional reflection distribution functions (BRDFs) can be complex

​

• Assumptions: Lambertian model

​

• Lambertian model: the outgoing ray intensity is a function of
• Incoming light power
• Wavelength
• Surface orientation relative to the incoming ray directions
• A scalar surface reflectance, aka, albedo

​

• No dependency on the outgoing direction of the ray

​

​

​

Specular surfaces

• Phong reflection model: widely used, with specular components of reflection
• Ambient: Constant
• Diffuse: Lambertian model
• Specular reflection

Why are these models important?

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Two light sources

From lights to world interpretation

• To understand our world from the lights
• We need to “associate” the reflected light with the surface in the world.
• We need to know which light rays come from which direction in space.

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Questions?

Images & cameras

• Forming an image = identifying which rays coming from which directions

​

• Camera: organizing rays

​

• Pinhole camera:
• One location on the wall
• Light from one direction

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Examples of pinhole cameras

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Does the distance between the projection surface and the pinhole matter?

The world is full of accidental cameras

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Image formation by perspective projection

• A (pinhole) camera projects 3D coordinates in the world to 2D positions on the projection plane, through the straight-line paths of each light ray through the pinhole

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Image coordinates vs.

virtual camera coordinates?

Perspective projection equations

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Orthographic (parallel) projection equations

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Good for the telephoto lenses

Can we really have orthographic projection?

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Questions?

What’s wrong with pinhole cameras

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Images are dime …

Limited lights ...

From pinholes to lenses

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Light needs to be concentrated/ bent!

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Lensmaker’s formula

• From one material to the other, light changes its wavelength and speed

​

​

​

​

​

• The changes at the surface will cause light to bend, i.e., refraction
• Depend on the change of speed and orientation

Snell’s law

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

A lens

• A specifically “shaped” piece of transparent material, positioned to focus light from a surface point onto a sensor

​

• Ideally …

​

​

​

• Need: numerical optimization!

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Simplified optical system

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

• Assumptions:
• Paraxial: the angle is small
• Thin lens: negligible thickness

​

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

• Assumptions:
• Paraxial: the angle is small
• Thin lens: negligible thickness

​

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

• Assumptions:
• Paraxial: the angle is small
• Thin lens: negligible thickness

​

• Lensmaker’s formula:

​

General cases

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]