Lecture 9: Motion estimation
1
Most slides from S. Lazebnik, which are based on other slides from S. Seitz, R. Szeliski, M. Pollefeys
Announcements
2
Corners
https://web.eecs.umich.edu/~justincj/slides/eecs442/winter2020/442_WI2020_lecture09.pdf
Slides 43-61 for eigenvalue intuition, through 71 for Harris Detection
Motion
Motion is a powerful perceptual cue
5
Yuwen Xiong, Mengye Ren, Wenyuan Zeng, Raquel Urtasun. Self-Supervised Representation Learning from Flow Equivariance. arXiv 2021.
[Yang et al., “Self-supervised Video Object Segmentation by Motion Grouping”, 2021]
Motion is a powerful perceptual cue
6
G. Johansson, “Visual Perception of Biological Motion and a Model For Its Analysis", Perception and Psychophysics 14, 201-211, 1973.
Source: S. Lazebnik
Optical flow
7
Source: S. Lazebnik
Estimating optical flow
8
I(x,y,t–1)
I(x,y,t)
Source: S. Lazebnik
The brightness constancy constraint
Simple loss function [Lucas & Kanade 1981]. Find flow that minimizes:
I(x,y,t–1)
I(x,y,t)
Source: S. Lazebnik
The brightness constancy constraint
Brightness Constancy Equation:
10
Linearizing the right side using Taylor expansion:
I(x,y,t–1)
I(x,y,t)
Therefore:
Source: S. Lazebnik
Derivative in y direction
Derivative in time:
The brightness constancy constraint
11
edge
(u,v)
(u’,v’)
gradient
(u+u’,v+v’)
If (u, v) satisfies the equation, �so does (u+u’, v+v’) if
Source: S. Lazebnik
The aperture problem
12
Perceived motion
Source: S. Lazebnik
The aperture problem
13
Actual motion
Source: S. Lazebnik
The barber pole illusion
14
Source: S. Lazebnik
The barber pole illusion
15
Source: S. Lazebnik
Solving the aperture problem
16
…
…
…
[Lucas & Kanade 1981]
Source: S. Lazebnik
Lucas-Kanade flow
17
When is this system solvable?
[Lucas & Kanade 1981]
Least squares problem:
…
…
…
Source: S. Lazebnik
Lucas-Kanade optical flow
18
(summations are over all pixels in the window)
M = ATA is the
“second moment” matrix
(also Gauss-Newton approximation to Hessian)
[Lucas & Kanade 1981]
…
…
…
Source: S. Lazebnik
Analyzing the second moment matrix
19
λ1
λ2
“Corner”�λ1 and λ2 are large,� λ1 ~ λ2
λ1 and λ2 are small
“Edge” �λ1 >> λ2
“Edge” �λ2 >> λ1
“Flat” region
Eigenvalues
of M
Source: S. Lazebnik
Conditions for solvability
20
Source: S. Lazebnik
Bad case: single, straight edge
Conditions for solvability
21
Source: S. Lazebnik
Good case
Lucas-Kanade flow example
22
Input frames
Output
Source: MATLAB Central File Exchange
Source: S. Lazebnik
Fixing the errors in Lucas-Kanade
23
Source: S. Lazebnik
Large motions
24
Source: Khurram Hassan-Shafique CAP5415 Computer Vision 2003
Idea #1: iterative estimation
Initialize flow: u0(x, y) = v0(x, y) = 0
For each iteration i:
Goal: minimize matching error
Iterative algorithm:
where the window width/height is 2N+1
Idea #2: Multi-scale estimation
26
Source: Khurram Hassan-Shafique CAP5415 Computer Vision 2003
Idea #2: Multi-scale estimation
For each iteration i:
Solve:
Initialize flow: u0(x, y) = v0(x, y) = 0
For each scale s of Gaussian pyramid:
Initialize flow from previous (coarser) scale
2x scale
2x flow magnitude
Example
28
Flow visualization
Coarse-to-fine LK with median filtering
Coarse-to-fine LK
Input two frames
Source: Ce Liu
Smoothness assumption
Solve:
1
Goal: minimize matching error + smoothness [Horn and Schunck 1981]
where p and p’ are neighboring pixels
match cost
smoothness
Smoothness assumption
30
Flow visualization
Coarse-to-fine LK
Input two frames
Horn-Schunck
Source: Ce Liu
Warping
31
Warp image 1 to image 2
Flow with warping
Solve:
Initialize flow: u0(x, y) = v0(x, y) = 0
For each scale s:
Initialize flow from previous (coarser) scale
Simple application: slow motion
Solve:
1
(x, y) at time t-1
(x + u, y + v) at time t
Structure from motion with flow
[Teed and Deng, 2021]
Use optical flow to find correspondences between frames.
Also called “passive structure from motion”�
Passive Structure From Motion
The camera stay still but the scene moves
The same equations can estimate camera movement and depth!
Camera 1
Camera 2
Camera 3
R1,t1
R2,t2
R3,t3
X1
X4
X3
X2
X5
X6
X7
p1,1
p1,2
p1,3
Application: motion magnification.
enlarge tiny motions, making them easier to perceive [Liu et al., 2005].
Daniel Geng
Zhaoying Pan
[Pan*, Geng*, Owens, “Self-Supervised Motion Magnification by Backpropagating Through Optical Flow”, NeurIPS 2023]
Motion magnification model
Input frames
[Geng, Pan, Owens, “Self-supervised Lagrangian Motion Magnification”, In Submission]
Magnified 2nd frame
U-Net
Magnification loss
Optical flow
Flow field
Magnification loss
Optical flow
Flow field
1
Motion magnification loss
magnification factor
1
Motion magnification loss
Backprop through the optical flow during training!
1
Motion magnification loss
1
Motion magnification loss
1
Motion magnification loss
1
Next lecture: machine learning
46
Coarse-to-fine refinement
47
Smooth & down-sampling
Feature tracking
48
C. Tomasi and T. Kanade. Shape and motion from image streams under orthography: �A factorization method. IJCV, 9(2):137-154, November 1992.
Feature tracking
49
Tracking example
50
J. Shi and C. Tomasi. Good Features to Track. CVPR 1994.