Panoramas

CSC606: Intro Computer Vision

Space Shuttle Discovery Flight Deck, Gigapan

These slides are taken from Cornell university

From last time: Projection matrix

(in homogeneous image coordinates)

Back to panoramas

Can we use homographies to create a 360 degree panorama?

Idea: project images onto a common plane

We’ll see what this homography means next

Can’t create a 360 panorama this way… we’ll fix this shortly

“Mosaic”

Creating a panorama

• Basic Procedure
• Take a sequence of images from the same position
• Rotate the camera about its optical center
• Compute transformation between second image and first
• Transform the second image to overlap with the first
• Blend the two together to create a mosaic
• If there are more images, repeat

​

Geometric interpretation of mosaics

• If we capture all 360º of rays, we can create a 360º panorama
• The basic operation is projecting an image from one plane to another
• The projective transformation is scene-INDEPENDENT
• This depends on all the images having the same optical center

Image 1

Image 2

Optical Center

Image reprojection

• Basic question
• How to relate two images from the same camera center?
• how to map a pixel from PP1 to PP2
• Answer
• Cast a ray through each pixel in PP1
• Draw the pixel where that ray intersects PP2

​

PP2

PP1

What is the transformation?

Image 1

Image 2

Optical Center

How do we map points in image 2 into image 1?

image 1

image 2

3x3 homography

Step 1: Convert pixels in image 2 to rays in camera 2’s coordinate system.

Step 2: Convert rays in camera 2’s coordinates to rays in camera 1’s coordinates.

Step 3: Convert rays in camera 1’s coordinates to pixels in image 1’s coordinates.

intrinsics

extrinsics

(rotation only)

Can we use homography to create a 360 panorama?

Answer: No

Panoramas

• What if you want a 360° field of view?

Spherical projection

• Map 3D point (X,Y,Z) onto sphere

unit sphere

• Convert to spherical coordinates

• Convert to spherical image coordinates

• s defines size of the final image
• often convenient to set s = camera focal length

Unwrapping a sphere

Credit: JHT’s Planetary Pixel Emporium

Spherical reprojection

• Map image to spherical coordinates
• need to know the focal length

f = 200 (pixels)

input

f = 800

f = 400

Aligning spherical images

• Suppose we rotate the camera by θ about the vertical axis
• How does this change the spherical image?

Aligning spherical images

• Suppose we rotate the camera by θ about the vertical axis
• How does this change the spherical image?
• Translation by θ
• This means that we can align spherical images by translation

​

Assembling the panorama

• Stitch pairs together, blend, then crop

Problem: Drift

• Error accumulation
• small errors accumulate over time

Problem: Drift

• Solution
• add another copy of first image at the end
• this gives a constraint: y_n = y₁
• there are a bunch of ways to solve this problem
• add displacement of (y₁ – y_n)/(n -1) to each image after the first
• apply an affine warp: y’ = y + ax [you will implement this for P3]
• run a big optimization problem, incorporating this constraint
• best solution, but more complicated
• known as “bundle adjustment”

(x₁,y₁)

Spherical panoramas

+

+

+

+

Microsoft Lobby: http://www.acm.org/pubs/citations/proceedings/graph/258734/p251-szeliski

Different projections are possible

Cube-map

Blending

• We’ve aligned the images – now what?

Blending

• Want to seamlessly blend them together

Image Blending

Feathering

Effect of window size

left

right

Effect of window size

Good window size

“Optimal” window: smooth but not ghosted

• Doesn’t always work...

Alpha Blending

Encoding blend weights: I(x,y) = (αR, αG, αB, α)

color at p =

Implement this in two steps:

1. accumulate: add up the (α premultiplied) RGBα values at each pixel

2. normalize: divide each pixel’s accumulated RGB by its α value

Q: what if α = 0?

see Blinn (CGA, 1994) for details:

Compositing, Part 1: Theory

I₁

I₂

I₃

p

Poisson Image Editing

For more info: Perez et al, SIGGRAPH 2003

Some panorama examples

“Before SIGGRAPH Deadline” Photo credit: Doug Zongker

Some panorama examples

• Every image on Google Streetview

Magic: ghost removal

M. Uyttendaele, A. Eden, and R. Szeliski. �Eliminating ghosting and exposure artifacts in image mosaics. �ICCV 2001

Magic: ghost removal

M. Uyttendaele, A. Eden, and R. Szeliski. �Eliminating ghosting and exposure artifacts in image mosaics. �ICCV 2001

Other types of mosaics

• Can mosaic onto any surface if you know the geometry
• See NASA’s Visible Earth project for some stunning earth mosaics

https://www.nasa.gov/centers/wallops/news/frozen_sos.html

Science on a Sphere

https://news.vcu.edu/article/Science_On_a_Sphere_now_at_VCU_offers_a_world_of_possibilities

Questions?