Image Morphing

Tong-Yee Lee

Blending

2D/3D Morphing/Metamorphics

a: Correspondence

Problem

b: interpolation

More challenging in 3D, i.e,

polygons

are different

​

Feature Line 2D Image Morphing

https://www.youtube.com/watch?v=3Iw-vUBo7v4

Beier&Neely (SIGGRAPH 1992)

Image Morphing Example

An Ideal Example

color blending (r,g,b)

src

dst

Cross-Dissolving

• Blend images with “over” operator

(r,g,b)

Dog Averaging

What to do?

• Cross-dissolve doesn’t work
• Any ideas?

​

​

Ghost effect

Blending is still useful and has some applications when shapes are similar by just replacement

Image morphing

• The goal is to synthesize a smooth transformation from one image to another.

Image Morphing

• Animate transitions between two images
• Specify Correspondence
• Warping to change shapes
• Blending

​

​

Image Morphing

• Combine warping and cross-dissolving

Ideally change shapes during morphing

Image morphing

image #1

image #2

Warping is key and hard part !!!

Morphing procedure:

for every t,

1. Find the average shape (the “mean dog”☺)

- local warping

2. Find the average color

- Cross-dissolve the warped images

Image morphing

• Why ghosting?
• Morphing = warping + cross-dissolving

shape

(geometric)

color

(photometric)

Image Morphing

• The warping step is the hard one
• Aim is to align features in images

​

Warp Specification

How can we specify the warp?

• Specify corresponding vectors
• interpolate to a complete warping function
• The Beier & Neely Algorithm�

• Thaddeus Beier, Shawn Neely, Feature-Based Image Metamorphosis, SIGGRAPH 1992, pp35-42.

t = 0

t=1

P_t = (1-t) *P_L + t* P_R

control each line point (i.e., x1,x2, y1,y2): P_t

Each line: starting point (x1,y1) and ending point (x2,y2)

A control line vector (x2-x1, y2-y1)

Control Line Vector

(0,0)

image #1

image #2

t= 0

t= 1

t

Color t = (1-t)Color_L+tColor_R

(0,0)

t= 0

t= 0.5

t= 1

Image Morphing

�How do we compute colors at dest pixels? (resampling�

Two Options

Forward Mapping - Problems

holes

Holes

Forward mapping

by rotation

Inverse mapping

Backward Mapping

t

1-t

Feature-based Warping

• Berier and Neeley use feature pairs of lines to control warp
• Given a pixel P in dest image, where is P’ in source image?

But, u is fraction but not length, it will make correspondence between two control lines with different lengths

Beier&Neely (SIGGRAPH 1992)

(0,0)

The dot product is useful for several things. One of the important uses is in a formula for finding the angle between two vectors that have the same initial point.

u

v

θ

Technically there are two angles between these vectors, one going the "shortest" way and one going around the other way. We are talking about the smaller of the two.

||v||cosθ >0

if θ <90

https://www.mathsisfun.com/sine-cosine-tangent.html

Assume (Q-P)

= (a,b)

​

Perpendicular

(Q-P) = (b, -a)

x

y

(a,b)

(b,-a)

u>0

u<0

(Q-P)

x >0 such as (1,0)

Warping with One line pair

• What happen to the “F”?

Warping with One line pair

• What happen to the “F”?

Warping with One line pair

• What happen to the “F”?

​

Warping with One line pair

• What happen to the “F”?

​

Warping with Multiple Line Pairs

(0,0)

(h-1,0)

(w-1,0)

(h-1,w-1)

Warping with Multiple Line Pairs

Weighting Effect of Each Line Pair

• To weight contribution of each line pair
• T. Berier and Neeley use:

A constant a is used to avoid dividing by a zero distance[i]

Values of b in the range [0.5, 2] are the most useful. The value of p is typically in the range [0, 1]

Ex: p’=w1/(w1+w2)p1+w2/(w1+w2)p2

p1

p2

Color at p = Color at p’

Weight is computed for each line on destination image

Note: we need to compute color in a bilinear way

Warping Psuedocode

Berier and Neeley’s Examples

Warping Psuedocode

Morphing is not only for faces

http://www.fantamorph.com/index.html

Summary

Image warping

​

​

​

Image morphing

Multiple Image Morphing

Barycentric coordinates

Barycentric coordinates

t1 = Area(p,A2,A3)/Area(A1,A2,A3)

t2 = Area(p,A1,A3)/Area(A1,A2,A3)

t3 = Area (p,A1,A2)/Area(A1,A2,A3)

Stages of eating red apple stock photo�

Can you try to create it from left to right directly?

Convert to mesh warping

Define a triangular mesh over the points

• Same mesh in both images!
• Now we have triangle-to-triangle correspondences
• Need to find a good corresponding mesh mapping

Defining Keypoints

Matching keypoints are manually annotated on both the start and final image. Below are two examples of such matched annota

https://inst.eecs.berkeley.edu/~cs194-26/fa17/upload/files/proj4/cs194-26-aae/

Some application:�Medical slice interpolation

​

Tong-Yee Lee, Chao-Hung Lin

Feature-guided Shape-based Image Interpolation

IEEE Transactions on Medical Imaging, Vol. 21, No. 12, pp. 1479-1489.2002 [Web]

Some application in 3D Morphing

Tong-Yee Lee, P.H Huang.�Fast and Institutive Polyhedra Morphing Using SMCC Mesh Merging Scheme.�IEEE Transactions on Visualization and Computer Graphics, Vol. 9, No. 1, pp. 85-98, 2003 [Web]

Warping from 3D to 2D

Matching feature points by warping

Basic Idea

3D Metamorphis

Image Morphing Example

Some tricks

Viewing

continuousity

In neighboring

images

View Morphing

http://homes.cs.washington.edu/~seitz/vmorph/vmorph.htm

Need to consider view transformation when morphing

1. Do a morphing on a specific view
2. Rotation the morphing results to a selected view

​

Viewing difference is large

inbetween neighboring images

Face-Morphing using Generative Adversarial Network(GAN)�

• https://medium.com/swlh/face-morphing-using-generative-adversarial-network-gan-c751bba45095

​

• Image Morphing with Perceptual Constraints and STN Alignment

http://efaidnbmnnnibpcajpcglclefindmkaj/https://arxiv.org/pdf/2004.14071.pdf