Remaining course logistics

1

• Optional homework assignment 7 is due on 12/18 at 23:59 ET.

- Gradescope submission site available.

​

• Final project reports are due on Sunday 12/20 at 23:59 ET.

- Gradescope submission site available.

​

• One upcoming computational photography talk:

- Tuesday 12/22, 1 – 2 pm, Grace Kuo (lensless cameras, diffuserCam, AR/VR displays).

​

• Returning borrowed equipment:

https://docs.google.com/spreadsheets/d/1CVg7nUbI701pvZFPX3BR0uzKB76Y6PEl3tXmq1UF4AU/edit#gid=0

Class evaluation*s* – please take them!

2

• CMU’s Faculty Course Evaluations (FCE): https://cmu.smartevals.com/

​

• TA evaluation: https://www.ugrad.cs.cmu.edu/ta/F20/feedback/

​

• 15-463/663/862 end-of-semester survey: https://docs.google.com/forms/d/e/1FAIpQLSew6XHzagr0KqWPFCmdoVmnHXQR5vfi4A6nEG0jMg6O1lBixA/viewform

​

• Please take all three of them, super helpful for developing future offerings of the class.

​

• Thanks in advance!

Using incident lightfield for simulating GelSight

Arpit Agarwal

Objective

Capture spatially and directionally varying illumination and use this lighting to generate images with synthetic objects

Key Principle

1. Measured scene radiance and global illumination by placing light probe
2. Use raycasting to find normals of surface at each pixel and interpolate colors

​

Debevec, Paul. "Rendering synthetic objects into real scenes: Bridging traditional and image-based graphics with global illumination and high dynamic range photography." ACM SIGGRAPH 2008 classes. 2008. 1-10.

Capture

Render

Spatially varying incident lightfield

1. Previous method only works if the object is very small as compared to the captured image
2. Capture images with multiple probe locations

Unger, Jonas, et al. Capturing and rendering with incident light fields. UNIVERSITY OF SOUTHERN CALIFORNIA MARINA DEL REY CA INST FOR CREATIVE TECHNOLOGIES, 2003.

Motivation

1. GelSight has small form factor
2. Camera can not be masked

2cm

2cm

Capture Method

GelSight

Capture Method

GelSight

Capture Method

GelSight

Rendering Method

Diffuse material

Precomputed

Rendering Method

Diffuse material

Scaled to [0,1]

Simulation setup

1. Setup scene in Mitsuba 2
2. Probe location along grid 8x9 with 0.5mm spacing

Simulation Results

GelSight data collection

1. Collected low dynamic range raw images at 5 exposure settings
2. Merged LDR images into HDR using MergeDebevec function

GelSight Results

Summary

1. Increasing the number of probe locations might result to better image
2. Implement the method for general material
3. The method could be used to place fake light sources for global illumination

18

Deep High Dynamic Ranging of Dynamic Scenes

​

Presenter: Uma Arunachalam

Andrew ID: uarunach

Brief Overview

• To merge LDR images that are not necessarily aligned or based on scenes that are not necessarily static, using deep learning (DL) and produce a HDR image as output
• Motivation-
• Reduce the setup-requirements on capture process
• Can be applied to any random scene captured with a handheld camera instead of using tripods, gphoto for capture
• Reduce the processing time required for merging
• Time is spent only on aligning images and inference time of the DL network.
• Mimic and extend on Baseline reference [1]

19

Approach

3 misaligned images

Optical Flow

3 misaligned images

CeLiu’s Classical Method

• Explores brightness constancy assumption, and uses iterative re-weighted least squares to minimize pixel intensity values between reference image and the ‘estimated’ flow guided pixel of another image

​

​

​

Approach

3 misaligned images

Learning based LDR merging

3 misaligned images

Network Architecture

Direct

Weight estimator

Fully Differentiable Architecture

Experimental Setup

• Default optical flow parameters, as suggested in [2]
• For training Direct and WE architectures:
• Used online available datasets plus few custom captures
• Xavier uniform initialization
• Adam optimizer, Learning rate = 0.0001
• Trained images over ~100 epochs, logging every 20 iterations
• Trained for about 3 hours before achieving convergence per method on NVIDIA Tesla v100
• Torch MSE Loss, with reduction to ‘sum’

Results: Error metric

Direct

WE

Results: Optical Flow

Results: Deep merging vs Tent

Results: WE vs Direct

Results

* not profiled, averaged over 2 runs

Main References

[1] https://cseweb.ucsd.edu/~viscomp/projects/SIG17HDR/PaperData/SIGGRAPH17_HDR.pdf

[2] https://people.csail.mit.edu/celiu/OpticalFlow/

​

​

​

Thank you!

Questions?

Real-time Cartoonization

Ricky Bao

What is NPR and cartoonization?

• Non-photorealistic rendering favors artistic styles instead of faithful recreation of a scene
• Paintings and drawings
• Cartoonization is a form of NPR
• Enhanced lines
• Textures and colors smoothed out
• Use cases of cartoonization
• Entertainment
• Simplification of scenes

Example of cartoonization

How is cartoonization performed?

• Line extraction
• Canny
• Gradient
• Difference of Gaussian
• Color reduction
• Bilateral filter
• Anisotropic diffusion filter
• Image enhancement
• Combine line extraction and color reduction

Real-time cartoonization

Resources

• R. Raskar, K.-H. Tan, R. Feris, J. Yu, and M. Turk, “Non-photorealistic camera: Depth edge detection and stylized rendering using multi-flash imaging,” eng, ACM Transactions on Graphics (TOG), vol. 23, no. 3, pp. 679-688, 2004, issn: 0730-0301.
• A. B. Patankar, P. A. Kubde, and A. Karia, “Image cartoonization methods,” eng, IEEE, 2016, pp. 1-7, isbn: 9781509032914.

Multi-Flash Imaging for Depth Edge Detection and Photometric Stereo

Matthew Baron (mcbaron)

Setup

Depth Edge Detection

Photometric Stereo

References

[1] T. Papadhimitri and P. Favaro, “A New Perspective on Uncalibrated Photometric Stereo,” in 2013 IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA, Jun. 2013, pp. 1474–1481, doi: 10.1109/CVPR.2013.194.

[2] R. Raskar, K.-H. Tan, R. Feris, J. Yu, and M. Turk, “Non-Photorealistic Camera: Depth Edge Detection and Stylized Rendering Using Multi-Flash Imaging,” p. 12.

[3] Y. Taguchi, “Rainbow Flash Camera: Depth Edge Extraction Using Complementary Colors,” p. 16.

[4] L. Wu, A. Ganesh, B. Shi, Y. Matsushita, Y. Wang, and Y. Ma, “Robust Photometric Stereo via Low-Rank Matrix Completion and Recovery,” in Computer Vision – ACCV 2010, vol. 6494, R. Kimmel, R. Klette, and A. Sugimoto, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2011, pp. 703–717.

[5] Y. Quéau, F. Lauze, and J.-D. Durou, “Solving Uncalibrated Photometric Stereo Using Total Variation,” J Math Imaging Vis, vol. 52, no. 1, pp. 87–107, May 2015, doi: 10.1007/s10851-014-0512-5.

Tracking Microscopic Motion Using Speckle Imaging

Yash Belhe

Setup

Optically Rough Surface

Laser

Bare Camera Sensor

Approx Co-located with Laser

Example with horizontal motion (with stutters)

Tracking Results

Learning to predict Depth and Autofocus using Focal and Aperture stack

Akankshya Kar and Anand Bhoraskar

{akankshk,abhorask}@andrew.cmu.edu

Contents

• Problem Statement
• Related Work
• Dataset
• Proposed Method
• Ground Truth generation for Autofocus
• Depth Map Generation
• Autofocus
• Future Work

Problem Statement

Low Light Image

Bright Autofocus Image

Depth Map

AFI Image

Related Work

Learning to Autofocus¹

Focal Stack

CNN

F

Auto-Focused Image

• Uses Soft ordinal regression⁴ for predicting F from 49 indices in focal stacks
• Captures its own images using 5 Pixel phones in High resolution
• Use MobileNetV2, to make the prediction very fast, to enable on-device use case

F

Datasets

DDFF⁷

​

• 720 images (12 scenes)
• Resolution 383 × 552
• Missing area in depth
• Very low baseline
• Lenselet size = 9
• Available online

Flowers Dataset⁸

​

• 3343 images
• Resolution 376 x 541
• No depth data
• Limited Domain(1)
• Synthetic data
• Lenslet size = 14

Learning to Autofocus dataset¹

• 387,000 patches of 128x128
• Resolution 1512 × 2016
• 50 Varied Scenes (indoor and outdoor)
• Not released yet

Proposed Method

CNN

(Dark to light)

AFI

CNN

All in focus

Depth

Refocusing

CNN

F, A

Selected Patch

GT generation

Project scope

Ground Truth generation for Autofocus

f_min

f_max

Subset Focal stack

F (median depth)

Merging to get partial focused image

SSIM

(Grid search)

A, F(median)

Selected Patch

​

Selected subset using focus to disparity

​

AFI

Partial Focused

Results

GT generation

# of Patches: 19,311

Patches Size : 64 x 64

Stride : 100

Selected Patch in Full image

Depth Patch and cropped out patch

Depth Map Generation

• AFI is generated and given to Encoder-Decoder style architecture, which has 80 channels (10 focal and 8 aperture)
• The Encoder is MobileNetV2 and Decoder is FCN with Upsampling, the loss is Virtual Normal Loss

AFI

Depth Map

CNN

Virtual Normal Loss²

• Enforces a high order geometric constraint in the 3D space for the depth prediction task.
• Makes N groups of 3 points in Point cloud and fits a plane
• DDFF is an indoor dataset, hence VNL was chosen

Results

Depth from confocal stack

• Current encoder-decoder network with MobileNet-V2 backbone and VNL loss
• So far we trained ~4000 iterations
• Reasons for slow training
• Small dataset
• Limited compute resources
• Large size of confocal stack (8 times the light field image)
• Pre-trained model not used to initialise

Ground Truth

Predicted

Auto-Focus

• AFI is generated and selected patch is given to given to CNN
• The selected patch is given as masked and as extra 2 channels in CoordConv⁵ method, loss is using Soft Ordinal Loss⁴

Refocusing

CNN

F, A

Selected Patch

Ordinal Regression Loss^3,4

• Ordinal Regression solves tasks with relative ordering by penalising more for further away predictions
• Soft Ordinal Regression makes the ground truth vector have a Softmax like weighting scheme rather than hard one hot vector

Future Work

• DDFF dataset has many limitations, we would shift to Flowers dataset as it is bigger and more blurring is there due to bigger aperture size
• Flowers dataset does not have depth, but we can generate the same Aperture and Focus dataset using Focus measure to get the same mapping between focus and disparity
• Work on removing the bug, which gives maximum aperture size (could be a DDFF related problem)
• Complete the training and ablation experiments
• Work on Low light aspect of it by using RetinaNet to generate low light images from light field and use that for training

​

References

1. Herrmann, Charles, et al. "Learning to Autofocus." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.
2. Yin, Wei, et al. "Enforcing geometric constraints of virtual normal for depth prediction." Proceedings of the IEEE International Conference on Computer Vision. 2019.
3. Diaz, Raul, and Amit Marathe. "Soft labels for ordinal regression." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019.
4. Fu, Huan, et al. "Deep ordinal regression network for monocular depth estimation." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.
5. Liu, Rosanne, et al. "An intriguing failing of convolutional neural networks and the coordconv solution." Advances in neural information processing systems 31 (2018): 9605-9616.
6. Tsai, Yu-Ju, et al. "Attention-based view selection networks for light-field disparity estimation." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 07. 2020.
7. Hazirbas, Caner, et al. "Deep depth from focus." Asian Conference on Computer Vision. Springer, Cham, 2018.
8. Srinivasan, Pratul P., et al. "Learning to synthesize a 4D RGBD light field from a single image." Proceedings of the IEEE International Conference on Computer Vision. 2017.

Fast Separation of Direct/Global Images In A Pocket

Presenter: Zili Chai (zilic)

Hardware Setup

Sony IMX219

RPi 4B

DLPDLCR2000EVM

Camera Calibration

10 bit bayer data, BGGR ordering

AAAAAAAA BBBBBBBB CCCCCCCC DDDDDDDD AABBCCDD

Linearity ∝ exposure

ISO = 100

​

​

​

Dark Frame

Mean = 63.90 (on 100 dark images)

1ms

100ms

Projector also needs time!

Global/Direct Images

Global Illumination

scattering, interreflection, shadows…

​

Separation In Real World

Global/Direct Images

Separation in Real World

Results

Checkerboard shift

max

min

direct

global

direct

global

Results

More Patterns

random pattern with phase shift

0.5+0.5·sinφ

+2pi/7, 4pi/7, 6pi/7, 8pi/7, 10pi/7,12pi/7

direct

global

Results

More Patterns

gray code

References

Sony, IMX219 product brief version 1.0, (Jan 2017).

​

Texas Instruments, TI DLP® LightCrafter™ Display 2000 EVM User's Guide, (Oct 2017).

​

Pagnutti, Mary A., et al. "Laying the foundation to use Raspberry Pi 3 V2 camera module imagery for scientific and engineering purposes." Journal of Electronic Imaging 26.1 (2017): 013014.

​

Nayar, Shree K., et al. "Fast separation of direct and global components of a scene using high frequency illumination." ACM SIGGRAPH 2006 Papers. 2006. 935-944.

Q&A

Kirchhoff Migration for NLOS Imaging

Dorian Chan

NLOS Imaging

Figure from:

David B. Lindell, Gordon Wetzstein, and Matthew O’Toole. 2019. Wave-based non-line-of-sight Imaging using fast f−k migration. ACM Trans. Graph. 38, 4, 116.

Traditional Approach: Backprojection

From http://www.xsgeo.com/course/mig.htm

F-K Migration: a Wave-Based Theory

Fourier Transform

Resample and Filter

Inverse Fourier Transform

Supposedly more BRDF robust!

Kirchhoff Migration: another idea from seismology

• Can be implemented as a backprojection
• Theoretically equivalent to F-K migration - derived from scalar theory

Second Order Kirchhoff Migration

• Attempt to model propagation from light source to hidden scene
• Looks a lot like the LCT

Relating F-K migration to volumetric albedo models

• Derive F-K from a volumetric albedo model (inspired by radar literature)
• Derive a volumetric albedo model using a slightly modified F-K derivation
• Main difference is a ramp filter

Questions?

Colorization through Optimization

Shruti Chidambaram | 15-463, Fall 2020

Colorization: Background

• Colorization typically requires:
• Image segmentation + tracking segments over image sequences
• Lots of user input/intervention
• Generally an ill-formed problem
• Contemporary techniques: deep learning to predict likely colors of pixels

​

​

Acknowledgement

My project reimplements the algorithm presented in Colorization using Optimization

(Anat Levin, Dani Lischinski, and Yair Weiss, 2004)

​

​

Algorithm

• Neighboring pixels should have similar colors if their intensities are similar
• Working in YUV color space
• Y → intensity
• U, V → chrominance (encode color)
• Minimize difference between color at a pixel and the weighted average of colors at neighboring pixels, subject to constraints of the user-specified colors
• Optimization problem → solve large, sparse system of linear equations

Results

MARKED-UP GRAYSCALE

COLORIZED

GROUND TRUTH

Results: Experimenting with Varied User Input

MARKED-UP GRAYSCALE

COLORIZED

GROUND TRUTH

Experimenting with a Deep Learning Approach

​

Autocolorization results from demos.algorithmia.com/colorize-photos

paper: Colorful Image Colorization (Zhang, Isola, Efros; 2016)

OUTPUT

INPUT

GROUND

TRUTH

Non-Realistic Rendering App

Han Deng (handeng)

Heshan Liu (heshanl)

Motivation

Nowadays there are a lot of different types of filters you can choose when you are using camera app. We would like to make something different and slightly more complicated rather than just adjust the color of the image.

Original

​

Blending

​

Sketch

​

Watercolor

​

​

Goal

• Implement multiple image stylization effects on mobile device
• Effects
• Pencil Sketch
• Watercolor Painting
• Image Blending
• Android App
• Take photos or select photo from the album.
• Then do the target filter on the taken/selected photo.

Environment Setup

• Mobile Device: Android

​

• Package
• Original Bitmap Factory provided by Android
• RenderScripts: Framework for running computationally intensive tasks

Algorithms: Pencil Sketch

• Two types of implementation
• Grayscale Image divided by Gaussian Blurred result
• Bilateral Filtering with a shade factor -> into Grayscale Output.

Input Image

Output 1

Output 2

Algorithms: WaterColor

Input Image

Output Image

Algorithms: Image Blending - Poisson Blending

• Goal: Seamless clone the source image into the target image

​

• Mix gradients of source and target image in target region:

​

Pérez, P., Gangnet, M., & Blake, A. (2003). Poisson image editing. In ACM SIGGRAPH 2003 Papers (pp. 313-318).

Algorithm and app Demo: Image blending

Possible Improvement

1. More complicated filters could be added like oil painting style

​

• The camera UI could be more elegant

Thank you!

Uncertainty in Radiometric Calibration

15663 Course Project

Advait Gadhikar

Goal

• Quantifying the uncertainty in Inverse Tone Mapping function
• Estimate a distribution to quantify the uncertainty in linearizing tonemapped images
• Using this uncertainty model to combine a HDR stack

Motivation

• Linearizing images is an important preprocessing step in many Computer Vision Applications
• Tone mapping for narrow gamut displays is a lossy compression
• Standard methods to linearize JPEG images use a deterministic mapping
• Such a mapping doesn’t consider information lost while converting to JPEG
• A Probabilistic Model for inversion (JPEG to RAW) is more informative
• Gives better performance on downstream tasks like HDR imaging

Standard Tone Map vs Learnt Tone Map

f is a polynomial and v is a linear transform, with g being a correction factor

If tone mapping is deterministic, Debevic and Malik’s method assumes this mapping

RMSE for Rendering Function (Raw to JPEG)

• RMSE on a stack of images collected under multiple exposure
• 16 images, with 1 stop exposure step
• RMSE: 16.995 (For 8-bit JPEG images)

​

​

Sample Images in stack

Inverse Distribution for Linearization

• The inverse distribution is assumed to be a Multi Variate Gaussian

​

​

​

• Here J is the forward rendering model
• The integral is approximated over a grid of pixel values
• A grid of 10 x 10 x 10 points was used for rgb values in [0,1]³

HDR with Uncertainty

• Combining exposure stack to get HDR Image with Uncertainty

​

Estimated HDR image with uncertainty

Thank You

Questions!?

References

• Chakrabarti, Ayan, et al. "Modeling radiometric uncertainty for vision with tone-mapped color images." IEEE Transactions on Pattern Analysis and Machine Intelligence 36.11 (2014): 2185-2198.
• Paul E. Debevec and Jitendra Malik. 1997. Recovering high dynamic range radiance maps from photographs. In Proceedings of the 24th annual conference on Computer graphics and interactive techniques (SIGGRAPH '97). ACM Press/Addison-Wesley Publishing Co., USA, 369–378. DOI:https://doi.org/10.1145/258734.258884

​

One-Shot to Vertigo: Novel View Synthesis using Light Field Cameras

Rohan Rao

(rgrao@andrew.cmu.edu)

​

Introduction

• Extension of The “Vertigo Effect” on Your Smartphone: Dolly Zoom via Single Shot View Synthesis [1], a CVPR2020 paper on single-shot view synthesis. Also implements US Patent by Lytro/Google, “Generating dolly zoom effect using light field image data”. [2]
• Creating a “dolly-zoom” or “vertigo” effect, popularized by Alfred Hitchcock’s 1958 movie, Vertigo.
• Keep the subject in focus while moving the camera and zooming simultaneously, creating a tunnelling/falling effect. Also widely used for aerial cinematography shots.

Single-camera Pipeline: Pre-processing

• Process RAW lightfield images (calibration, demosaicking, etc) using the PlenPy library [3].
• Obtain depth image from lightfield using Structure Tensors [5] to estimate disparity [3].
• Used the INRIA Lytro dataset [4] for testing calibration and depthmap results.

(from [1])

Single-camera Pipeline: Digital Zoom

• Can obtain closed-form expression for digital zoom using focal length and camera intrinsics.
• Implemented clipped zooming using OpenCV and efficient resizing.

(from [1])

Single-camera Pipeline: View Synthesis

• Can obtain closed-form expression for single-shot view synthesis and warp image and depth map using z-buffering and forward warping.
• Implemented with a z-buffer and looping over all pixels (more efficient in C++ though)

(from [1])

Single-camera Pipeline: Image/Depth fusion

• Can fuse the images by creating image/depth occlusion masks from the previous step (regions where warping did not update the value)

(from [1])

Single-camera Pipeline: Depth Hole Filling

• Simplistic method - just takes max of neighboring values to replace missing

(from [1])

Single-camera Pipeline: Image Inpainting

• Extension: convert continuous depth values to discrete ones using K-Means clustering
• Inpainting using masks and nearest valid pixel in the row

(from [1])

Single-camera Pipeline: Shallow Depth of Field (SDoF)

• Extension: smoother blending of depth segments using finite bounding values, then scaling
• Blur depth segments with different sigma values to keep subject focused

(from [1])

Intermediate steps towards final results

Post digital zoom (I1)

Without digital zoom (I2)

I₁^DZ

I₂^DZ

I₁^DZ

Final result

Extensions of prior work

Ideas for future work

• Improved depth map estimate obtained from a light field instead of monocular depth estimation from a single RGB image.
• Algorithm for using continuous-valued depth maps through K-Means based discretization.
• Algorithm for Shallow Depth of Field (SDoF) and smoother merging of multiple image segments to prevent artifacts at the edges.

• Improve the inpainting approaches using more sophisticated search for valid pixels (currently searches same row) or using deep-learning for inpainting (experimented but didn’t show results here)
• Obtain more realistic depth values with calibration, since this will give better results
• Improve the view synthesis from light fields using Multiplane Images (learning view synthesis using MPIs), this can be used to create more novel views from a single image/video sequence

Thanks for listening! Questions?

References

1. Liang et al., The “Vertigo Effect” on Your Smartphone: Dolly Zoom via Single Shot View Synthesis, CVPRW 2020.
2. US Patent US8971625B2, Generating dolly zoom effect using light field image data, Lytro Inc.
3. Schambach, Maximilian and Puente León, Fernando, Microlens array grid estimation, light field decoding, and calibration, IEEE Transactions on Computational Imaging, 2020.
4. M. Le Pendu, X. jiang, C. Guillemot, Light Field inpainting propagation via low rank matrix completion, IEEE Trans. on Image Processing, vol. 27, No. 4, pp. 1981-1993, Jan. 2018.
5. Sven Wanner and Bastian Goldlueck, Globally Consistent Depth Labeling of 4D Light Fields

Epipolar scanning for shape-from-silhouette in scattering medium

Shirsendu S Halder

shirsenh

Shape from silhouette

Shape from silhouette

Collimated illumination

Orthographic camera

Object

Imaging through scattering medium

We want the transmissive paths (ballistic photons) as they sharpen the object image and also enhances contrast.

Imaging through scattering medium

Rows of an orthographic projector lit.

Camera captures ballistic paths from each row.

Combination of position cues, angle cues, and probing

Orthographic projector

Telecentric camera

Imaging through scattering medium

Rows of an orthographic projector lit sequentially.

Camera captures ballistic paths from each row.

Orthographic projector

Telecentric camera

Why telecentricity?

• Low perspective distortion���
• Zero angular field-of-view: minimum parallax error�����
• Larger usable depth of field than conventional lenses due to symmetrical blurring

Image credit: https://www.edmundoptics.com/knowledge-center/application-notes/imaging/advantages-of-telecentricity/

Setup

Setup

Syncing electronics

Setup

Laser line

Object used for scanning

Scanning (in air)

Global

Scanning (in air)

Epipolar

Scanning (in water)

Global

Scanning (in water)

Epipolar

Scanning (milk + water - low concentration)

Global

Scanning (milk + water - low concentration)

Epipolar

Scanning (milk + water - medium concentration)

Global

Scanning (milk + water - medium concentration)

Epipolar

Scanning (milk + water - high concentration)

Global

Scanning (milk + water - high concentration)

Epipolar

Thank you

Questions?

HDRnet and Artistic Style Transfer for HDR replication

Brandon Hung

HDRNet

Artistic Style Transfer

HDRnet for Tonemapping

Input Output Learned

​

Style Transfer to Preserve Content

• Idea: map bright exposure image on to low-exposure image
• Bright exposure = “style”, low-exposure = “content”

​

Combining the two?

• Style transfer maps colors to image
• HDRnet tonemaps color
• Idea: tonemap image using HDRnet transfer tonemap on original image
• Hopefully preserves tonemap color + original image content

​

Combining the two?

Unstructured Lightfields

Akshath Jain and Deepayan Patra

Inspiration

Assignment 4 - Planar Unstructured Lightfields

155

Technical Background

Goal: Generating arbitrary viewpoint rendering given unstructured, non-planar, input

​

Proposed Solution: Interpolate on a triangular mesh of input viewpoints with appropriate depth information to generate new image

156

Steps

Generate Input Poses

1

2

3

Triangulate Viewpoints

Rendering

157

Generating Poses

COLMAP estimates camera poses and a 3D reconstruction of the scene using structure from motion. This is broken down into:

• Feature detection and extraction
• Feature matching and geometric verification
• Structure and motion reconstruction

158

Triangulating Viewpoints

From the new projected camera origins, we develop a Delaunay Triangulation of the viewpoints.

​

Each point within the input space has exactly three neighboring viewpoints to interpolate.

159

Rendering

Generate each novel viewpoint’s image as:

1. Backproject novel pixel onto mesh to determine intersecting face
2. Project novel pixel onto scene
3. Backproject scene pixel onto mesh face vertices
4. Assign novel pixel color using Barycentric interpolation

160

Results

Unstructured Images

Output

Credits

Our implementation was inspired by the Unstructured Light Fields paper by Davis et. al. We also relied on work done by Mildenhall et. al in LLFF to get camera poses from COLMAP and to identify ideal new sample viewpoints. Our full list of sources is as follows:

Davis et al. “Unstructured Light Fields.” Eurographics 2012.

Mildenhall et. al. “Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines.” SIGGRAPH 2019.

Levoy, Marc. “Light Fields and Computational Imaging.” IEEE 2006.

Buehler et. al. “Unstructured Lumigraph Rendering.” SIGGRAPH 2001.

Isaksen et. al. “Dynamically Reparameterized Light Fields.” SIGGRAPH 2000.

Schönberger, Johannes Lutz and Frahm, Jan-Michael. “Structure-from-Motion Revisited.” CVPR 2016.

Pollefeys, Marc. “Visual 3D Modeling from Images” 2002.

164

15663: Computational Photography

Fall 2020

​

Using Spatio-Temporal Radiance Variations to Look Around Corners

​

Varun Jain

varunjai@andrew.cmu.edu

​

​

Idea^[1]

Figure 1a: A and B represent two people hidden from the camera’s view by a wall.

Idea^[1]

167

Figure 2: (top) the hidden scene with 2 actors, (bottom) penumbra as seen by the naked eye, as seen by the camera and the groundtruth trajectories over time.

Contributions

1. Re-implementation of the existing code-base in Python

​

• Benchmark the implementation by comparing results on the given video sequences

​

• Explore the efficacy of features extracted by deep methods

Results ^{[1 person]}

Fig: Observed video

Fig: 1-D angular projections of the hidden scene

Results ^{[2 people]}

Fig: Observed video

Fig: 1-D angular projections of the hidden scene

Comparative Results

Limitations

• 1D reconstruction

​

• Temporal motion required

​

• Assumes uniform floor albedo

Contributions

• Re-implementation of the existing code-base in Python

​

• Benchmark the implementation by comparing results on the given video sequences

​

• Explore the efficacy of features extracted by deep methods

Dataset and Methodology

Figure 3: (left) generated dataset, and, (right) model architecture.

References

1. Bouman, Katherine L., Vickie Ye, Adam B. Yedidia, Frédo Durand, Gregory W. Wornell, Antonio Torralba, and William T. Freeman. "Turning corners into cameras: Principles and methods." In Proceedings of the IEEE International Conference on Computer Vision, pp. 2270-2278. 2017.

​

• Seidel, Sheila W., Yanting Ma, John Murray-Bruce, Charles Saunders, William T. Freeman, C. Yu Christopher, and Vivek K. Goyal. "Corner occluder computational periscopy: Estimating a hidden scene from a single photograph." In 2019 IEEE International Conference on Computational Photography (ICCP), pp. 1-9. IEEE, 2019.

​

• Tancik, Matthew, Guy Satat, and Ramesh Raskar. "Flash photography for data-driven hidden scene recovery." arXiv preprint arXiv:1810.11710 (2018).

​

• Saunders, Charles, John Murray-Bruce, and Vivek K. Goyal. "Computational periscopy with an ordinary digital camera." Nature 565, no. 7740 (2019): 472-475.

175

Evaluating Confocal Stereo

Kyle Jannak-Huang

Relative Exitance Estimation

Aperture f3.5

Focus dist ~ 2cm

Aperture f7.1

Focus dist ~ 2cm

Relative Exitance Estimation

Aperture f3.5

Focus dist ~ 2cm

Aperture f3.5

Focus dist ~ 90cm

Image Alignment

Depthmap from Aperture-Focal Images

• Diagonal line artifacts related to small differences in corner detection during image alignment
• White/black artifacts on leaves that moved due to air flow during capture
• Poor depth estimation on rest of plant due to incorrect depth estimates

Depthmap from Aperture-Focal Images

• Diagonal line artifacts related to small differences in corner detection during image alignment
• White/black artifacts on leaves that moved due to air flow during capture
• Poor depth estimation on rest of plant due to incorrect depth estimates

Depthmap from Aperture-Focal Images

Fuzz dataset images

Denoised result

Discussion

• Benefits from using a custom feature detector that will find features in the center regardless of blur
• Improvements in capture - optical table, diffuse plane, better lighting
• Optimizer for image alignment needs tuning
• Could benefit from segmenting out the background

Gradient-Domain Path Tracing

Zhi Jing (Zoltan), Ran Zhang (Ryan)

Gradient-Domain Path Tracing

• Perform standard Monte Carlo rendering to obtain primal image
• Sample gradients
• Reconstruct image from primal and gradients

Gradient-Domain Path Tracing

• Perform standard Monte Carlo rendering to obtain primal image
• Sample gradients
• Reconstruct image from primal and gradients

Standard Monte Carlo rendering

Gradient-Domain Path Tracing

• Perform standard Monte Carlo rendering to obtain primal image
• Sample gradients
• Reconstruct image from primal and gradients

Render result - primal image

Sample gradients

The most naive approach

Gradient image - naive approach

dx

dy

Reconstruction result - naive approach

primal

final

Reconnection

Gradient image - with reconnection

dx

dy

Gradient image - naive approach

dx

dy

Reconstruction result - with reconnection

primal

final

Paper’s approach - add MIS

dx

dy

Paper’s approach - add MIS

​

primal

final

Gradient-Domain Path Tracing

• Perform standard Monte Carlo rendering to obtain primal image
• Sample gradients
• Reconstruct image from primal and gradients

Image Reconstruction

Find an image that best fits the estimated primal and gradients

Primal

Gx

Gy

Poisson Reconstruction

L2-norm reconstruction

​

​

L1-norm reconstruction

​

Gradient term

Primal term

Poisson Reconstruction

A x = b

Solve the least squares minimization problem using Conjugate Gradient Descent

Naïve Implementation using CGD

Naïve Implementation using CGD

L2 norm

L1 norm

Re-implement and Experiment (L1 norm vs. L2 norm)

L2 Norm

L1 Norm

Experiment (L1 norm vs. L2 norm)

L2 Norm

L1 Norm

Unbiased but has some artifacts

Biased but has less artifacts

Experiment (different α values)

α = 0.2

α = 1.0

α = 10

Thank you!

References

Kettunen, Markus, et al. "Gradient-domain path tracing." ACM Transactions on Graphics (TOG) 34.4 (2015): 1-13.

Lehtinen, Jaakko, et al. "Gradient-domain metropolis light transport." ACM Transactions on Graphics (TOG) 32.4 (2013): 1-12.

Manzi, Marco, et al. "Gradient-Domain Bidirectional Path Tracing." EGSR (EI&I). 2015.

Pérez, Patrick, Michel Gangnet, and Andrew Blake. "Poisson image editing." ACM SIGGRAPH 2003 Papers. 2003. 313-318.

Manzi, Marco, Delio Vicini, and Matthias Zwicker. "Regularizing Image Reconstruction for Gradient‐Domain Rendering with Feature Patches." Computer graphics forum. Vol. 35. No. 2. 2016.

Ji, Hao, and Yaohang Li. "Block conjugate gradient algorithms for least squares problems." Journal of Computational and Applied Mathematics 317 (2017): 203-217.

View Synthesis using Neural Radiance Fields

Rohan Joshi

Problem Definition:

Given: Images of a scene from known camera poses,

Task: Render new views of the scene

Method: Scene Representation as Neural Radiance Fields

• Use a fully connected network instead of a voxel grid �(Advantages: Simplicity and efficient storage)������
• Calculate by querying points along rays

Algorithm for Volume Rendering

Forward Pass

Step 1: Generate Camera Rays

Step 2: Sample points along each ray (affects the resolution of output image )

Step 3: Get RGB and σ (Volume Density at the point) value

​

​

Algorithm for Volume Rendering

Error Calculation and Backpropagation

​

Step 1: Composite the pixel color along the ray to get the rgb image

Step 2: Minimise the mse for known images

Step 3: Backprop ...

​

W(σi)

Results: Performed on Synthetic Data

�

• Training Images:
• 100 images taken from different views
• Camera Poses provided in the dataset

​

Training Set

Result: Rendered Views

References

• DeepStereo: Google quartet has method for new-view synthesis- Credit: John Flynn et al. arXiv:1506.06825
• Nerf: https://arxiv.org/pdf/2003.08934.pdf
• https://www.matthewtancik.com/nerf

Thank you

Color-Filtered Aperture For Image Depth & Segmentation

Leron Julian

Depth Estimation Techniques

Stereo

​

​

​

Depth From Disparity Using Color Filter (This Project)

​

Depth From Defocus

Constructing Depth Using Color Misalignment

• Red, Green, Blue Color Aperture placed in a certain arrangement
• A scene point farther than focused depth =
• Right-Shift in R
• Up-Shift in G
• Left-Shift in B
• Color shifts come from geometric shifts.

​

How? (Red & Green Example)

Background

Camera Sensor

Object In Focus

Color Aperture

Sample Images

Depth Estimation

• Let d be the hypothesized disparity between the RGB Channels:
• Need to measure the quality of between:

​

• Colors in natural images form elongated clusters in RGB space

Depth Estimation

• Consider a set of pixels with hypothesized disparity d. Set can be represented as:

​

• Goal is to find a disparity d that minimizes the color alignment measure

​

​

• As L gets small when cluster is elongated meaning the RGB components are correlated

RGB Color Model

d = 1

d = 3

d = 5

• As the disparity d increases, the cluster becomes more isotropic.

Depth Map

• Can now solve for the disparity d to by minimizing L for a predetermined range of disparities.
• Can construct a depth map using local estimates (prone to error) or energy minimization graph-cuts:

​

Captured Images

Local Estimates

Estimate with Graph-Cuts

Trimap

Foreground

Unknown

Background

Matting

• Matting Equation:

​

​

Matte Optimization Flow

• Optimize the matte with various consistency methods based on foreground and background color
• Obtain the optimal matte

Applications of Matting

• Changing the background of images
• Refocusing
• Artificial Background Blur
• Video Matting

References

• Bando et. al, “Extracting Depth and Matte using a Color-filtered Aperture”. SIGGRAPH 2008.

​

• Levin et. al, “A closed form solution to natural image matting”. CVPR 2006

Depth from Defocus in the Wild

Alice Lai, Adriana Martinez

Goal: Two-frame depth from defocus using tiny blur condition

Original Image

Image w/Blur

Idea: Combine local depth/flow estimation w/ �spline-based scene understanding

Idea: Combine local depth/flow estimation w/ �spline-based scene understanding

Local Depth/Flow Estimation (Bottom-up Likelihood)

Defocus Equalization Filters

Minimize sum-of-squared difference between two equalized images!

Ensures brightness constancy and equalizes the variances between the two images

Local Likelihood Q

Local Depth/Flow Estimation (Bottom-up Likelihood)

Local Prior L_qp

Local smoothness prior due to overlapping patches

Local Likelihood Q

Fit quadratic function to make it analytical for global optimization

Local Depth/Flow Estimation (Bottom-up Likelihood)

Q_p evaluated at every pixel q for some (d,v) patch pixel p estimate (p denotes center pixel)

M x M array of values that encodes smoothness between every pixel w.r.t. every pixel

Minimize over LDFD loss to optimize (d,v) patch estimates using Markov Random Field (MRF)

Idea: Combine local depth/flow estimation w/ �spline-based scene understanding

Global DFD (Top-Down Likelihood)

Inputs:

Control Points, �Feature Map, �Patch Likelihoods

Update weight vectors

Update occlusion map

Update depth planes, 2D affine transformations, segment labels of control points

Update control points’ feature vectors

​

Output:

Pixel-based depth and flow

Patch-based depth and flow

Scene segmentation

Spline parameters

​

Control Point C_n: �- depth plane D�- 2D affine transformation U, V

​

Pixel q (including control points):

- weight vector

- feature vector

- segment label

Global DFD (Top-Down Likelihood)

Control Points

Scene Segmentation

Results

Arbitrary L_qp smoothness depth map

Ground truth depth

Original input

Patch-based L_qp smoothness depth map

Results

Arbitrary L_qp smoothness depth map

Ground truth depth

Original input

Patch-based L_qp smoothness depth map

Acknowledgements

Dr. Ioannis Gkioulekas for the tireless support of our project through many OH sessions and Slack messages

Dr. Kiriakos Kutulakos for sharing his research group’s dataset and contact information of first authors

​

​

​

References:

Huixuan Tang, Scott Cohen, Brian Price, Stephen Schiller and Kiriakos N. Kutulakos, Depth from defocus in the wild. Proc. IEEE Computer Vision and Pattern Recognition Conference, 2017. https://www.dgp.toronto.edu/WildDFD/

​

Questions?

A Reconstruction Framework for Time Series Thermal Traces using Mixed Stereo Photography

Arjun Lakshmipathy

Computational Photography (15-862)

Fall 2020

​

Context

Larger Thrust: Automated Design of Custom, Low-Cost Dextrous Hands from Demonstration

​

Principal Reference

The Approach

• Difficult to simulate, not so difficult to capture (with the right tools)
• Thermography can detect residual heat traces imparted on objects through human grasping
• With depth + different viewpoints, we can estimate contact texture on arbitrary geometries
• Use multiple cameras and fuse
• Approach: Manipulate physical object, rotate object and capture both thermal and depth images from multiple viewpoints, combine into contact map

The Problems

3 problems to solve:

• Image alignment
• Pose and location estimation of target object within the capture scene
• Texture mapping

Materials and Setup

•1 Intel RealSense D415

•1 Flir C5

•2 tripods + 1 clamp mount

•?? Hand warmers

•1 dressed up turntable

•1 calibration target

•1 house lab space

Turntable and second tripod taken borrowed from CMU Motion Capture Lab

​

Process Step 1: Calibration

•Thermal intrinsics

•Stereo Calibration

•~ 10 images apiece

Process Step 2: Capture

Process Step 3: Object Pose Estimation

•Entirely in Depth CCS

•Load 3D model, convert to point cloud

•Segment scene to extract object

•Solve R and T using Iterative Closest Point Method (ICP) [2]

​

Process Step 4*: Alignment

Process Step 5*: Texture Mapping via Color Map Optimization [3]

Concluding Remarks

•Process: Calibration -> Capture -> Pose Estimation -> Alignment -> Texture Mapping

•Works reasonably…in principle. Still some items to resolve for full operation

•Next steps: work through bugs and nuances with each step, collect initial and final grasps for multiple objects from multiple subjects. Use as basis for next research thrust.

​

*Synthesized from dataset captures only / not my own yet

​

​

Thanks for listening! Questions?

References

[1] S. Brahmbhatt, C. Ham, C. C. Kemp, and J. Hays, “Contactdb: Analyzing and predicting grasp contact via thermal imaging,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8709–8719, 2019. �

[2] PJ Besl and Neil D McKay. A method for registration of 3- d shapes. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 14(2):239–256, 1992

​

[3] Qian-Yi Zhou and Vladlen Koltun. Color map optimization for 3d reconstruction with consumer depth cameras. ACM Transactions on Graphics (TOG), 33(4):155, 2014.

HDR Image Reconstruction using Hallucinated exposure stack

Shamit Lal(shamitl), Sajal Maheshwari(sajalm)

Problem

Traditional approach

• Collect a stack of images at different exposures
• If the images are non-linear, linearize them.
• Merge the linear exposure stack.
• If we want to display the image, we can also do a tonemapping operation

Deep learning based methods

What if ?

• We can merge these ideas and :
• Create a linearized and de-quantized exposure stack using a single input image.
• Use this linearized stack and weights for each of the pixels and generate an output HDR image.
• Instead of traditional merging method, let a deep network decide the weights for each of the pixels in the stack.

​

Our approach

• Step 1 : Exposure stack generation
• Images should be sharp.
• The changes should be in brightness, not in color.
• No multiple iterations
• Possible approaches
• Naive encoder-decoder
• Mapping based model
• Modified encoder-decoder

Naive encoder-decoder

• A simple single encoder- multi decoder model with different decoder heads having supervision corresponding to different images of the exposure stack.
• Result :
• Blurry images
• Images with artefacts like color inconsistency
• Not great results as the output images looked very similar to the original input.

​

Mapping based approach

Mapping based approach

• The approach described in the previous slide uses multiple iterations. Therefore, although being a good starting point, it can not be used in our experiments.
• We now explain our approach, which can directly give us a mapping of higher degree.
• We output a 256 dimension vector (for each channel). Now, we make this vector monotonic and bound it from 0 to 1, and then for intensity value at each pixel, pick the output intensity value to be the value at the index of the input intensity.

​

Mapping based approach - Problem!

• This operation is non-differentiable. How to solve this ?
• During training, use an extra input of 256 x H x W dimensions, with each of the entries containing a one-hot vector. Multiply each of these values with the output vector (at each pixel value) earlier element-wise and sum the values along the channel dimension.
• Additionally, all the outputs should be identical, which we incorporate by adding loss to make all the pixel-wise embeddings closer to the mean embedding.
• Better and sharper results!

​

Mapping based approach - Results

• The results are much better now, however, we can see some color cast. This might be because we are creating the function for all channels. We are currently trying to run this method only on the luminance channel.

Modified encoder-decoder

• Instead of using a naive encoder-decoder trying to learn the output, try to learn the difference between the output and input instead (has been done before for many many problems!)

​

Modified encoder-decoder

​

• The modified encoder-decoder ensures that the results are close to the input but also have the brightness of the output where supervision is provided.
• The 1x1 convolution ensures that the output we get is sharp and not blurry outputs.

Merging HDR

• Once we get the stacks, we can either merge the images to get the HDR image using traditional approaches or a learning based method, which can again consist of an encoder-decoder with shared weights for each of the outputs.
• We have not yet run any tone-mapping model, so the HDR images generated look a bit dark. However, the images are decent when viewed in HDR viewers like openhdr(https://viewer.openhdr.org/). However, this is not working as expected and we believe there is still some training/tuning required.

​

Unsupervised methods ?

• We have currently been providing supervision at every stage - what happens if we try to generate the exposure in an unsupervised fashion ?
• Currently running experiments for unsupervised methods

​

References

1.Single-Image HDR Reconstruction by Learning to Reverse the Camera Pipeline, Yu-Lun Liu, Wei-Sheng Lai, Yu-Sheng Chen, Yi-Lung Kao, Ming-Hsuan Yang, Yung-Yu Chuang, Jia-Bin Huang (https://arxiv.org/abs/2004.01179)

2.Deep Reverse Tone Mapping, Yuki Endo, Yoshihiro Kanamori, and Jun Mitani(http://www.cgg.cs.tsukuba.ac.jp/~endo/projects/DrTMO/)

3.HDR image reconstruction from a single exposure using deep CNNs,Gabriel Eilertsen, Joel Kronander, Gyorgy Denes, Rafał K. Mantiuk, Jonas Unger(https://arxiv.org/abs/1710.07480)

4. Zero-Reference Deep Curve Estimation for Low-Light Image Enhancement, Chunle Guo, Chongyi Lim Jichang Guo, Chen Change Loy, Junhui Hou, Sam Kwong, Runmin Cong(https://openaccess.thecvf.com/content_CVPR_2020/papers/Guo_Zero-Reference_Deep_Curve_Estimation_for_Low-Light_Image_Enhancement_CVPR_2020_paper.pdf )

5. Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks , Jun-Yan Zhu, Taesung Park, Phillip Isola ,Alexei A. Efros(https://junyanz.github.io/CycleGAN/)

6. ExpandNet: A Deep Convolutional Neural Network for High Dynamic Range Expansion from Low Dynamic Range Content, Demetris Marnerides, Thomas Bashford-Rogers, Jonathan Hatchett, Kurt Debattista(https://arxiv.org/abs/1803.02266)

​

​

​

​

​

Shape Estimation

Qiqin Le, Qiao Zhang

Overview

• Limitations:
• Depth from defocus:
• Affected by occlusions
• Ambiguity
• Depth from correspondence:
• Correspondence errors
• Matching ambiguity at repeating patterns and noisy regions
• Affected by occlusions
• Combining depth from defocus and correspondence
• Require complicated algorithms / Camera modifications / Multiple image exposures

​

• Improvement:
• Use angular coherence to improve robustness.

​

• Pipeline:
• Combine defocus and correspondence metrics for dense depth estimation and then uses the additional cue of shading to refine details in the shape.

​

Pipeline

Tao, Michael W., et al. "Shape estimation from shading, defocus, and correspondence using light-field angular coherence."

​

Image Source

http://lightfield.stanford.edu/lfs.html

Our images taken by the Lytro Illum camera

Defocus Depth

​

Defocus Depth

​

Depth Map & Confidence Map & Shape Estimation

​

Correspondence Depth

​

​

Correspondence Depth

​

Depth Map & Confidence Map

​

Combined Depth from Defocus and Correspondence

​

​

​

Combined Depth from Defocus and Correspondence

​

​

Depth Map & Confidence Map

Shading Depth

​

Albedo

Shading Depth

​

Spherical Harmonic

Defocus Depth

​

Depth Map & Confidence Map

Correspondence Depth

​

Depth Map & Confidence Map

Combined Depth from Defocus and Correspondence

​

​

Depth Map & Confidence Map

​

Defocus Depth

​

Depth Map & Confidence Map

Correspondence Depth

​

Depth Map & Confidence Map

Combined Depth from Defocus and Correspondence

​

​

Depth Map & Confidence Map

​

References

​

[1] Tao, Michael W., et al. "Shape estimation from shading, defocus, and correspondence using light-field angular coherence." IEEE transactions on pattern analysis and machine intelligence 39.3 (2016): 546-560.

[2] Tao, Michael W., et al. "Depth from combining defocus and correspondence using light-field cameras." Proceedings of the IEEE International Conference on Computer Vision. 2013.

Q & A

Normal Estimation for Transparent Objects

Amy Lee

302

Overview + Motivation

• Naive photometric stereo methods do not work well for specular or transparent objects.
• Improvement idea:
• Use a reference chrome sphere to estimate the light direction for each image.
• Estimate the normals at highlighted points on the object using the reference light direction.

303

Pipeline

304

Image Set

305

• Video of static scene, moving spot light (dense set of images under variable illumination)
• Object set next to chrome sphere
• Assume orthographic camera

Image Segmentation

306

Image

Mask

Final

Initial Silhouette-based Surface Normals

Key Ideas:

• Surface normals are parallel to the image plane at the silhouette of the object.
• Interpolating the silhouette normals gives a decent approximation of the surface.
• Caveat: Initial normals heavily influence quality of final result.

307

Initial Silhouette-based Surface Normals

308

Orientation Consistency w/ Reference Sphere

309

Normal Clustering

• Can infer that highlights occur when intensity exceeds some threshold (e.g. >0.9). Highlight occurrences form clusters in time.

310

Normal Clustering

• May have multiple clusters due to interreflections (not true direct highlights).
• Find top 2 “salient” clusters (i.e. the 2 clusters with the most observations) corresponding to specular highlights for every pixel.
• Record the mean normal corresponding to each cluster (computed using reference sphere).

311

Finding Optimal Normal Clusters

Graph-cut cost minimization using max flow algorithm.

• At most 2 nodes per pixel (1 node per salient normal cluster).
• Want to label nodes as being either true or false highlights. If true, use reference normal.
• Edges between adjacent pixels’ nodes.
• Cost (energy): difference with initial normal (E₁) + neighboring normals (E₂)

312

Final Calibrated Normals

313

Quality of results are heavily influenced by the initial normals

References

1. Yeung, Sai-Kit & Wu, Tai-Pang & Tang, Chi-Keung & Chan, Tony & Osher, Stanley. (2015). Normal Estimation of a Transparent Object Using a Video. Pattern Analysis and Machine Intelligence, IEEE Transactions on. 37. 890-897. 10.1109/TPAMI.2014.2346195.
2. H.-S. Ng, T.-P. Wu, and C.-K. Tang, “Surface-from-gradients without dis-crete integrability enforcement: A Gaussian kernel approach,”IEEE Trans.Pattern Anal. Mach. Intell., vol. 32, no. 11, pp. 2085–2099, Nov. 2010.

314

Thank you for your time!

315

Event-based Object Tracking-by-Detection

Jessica Lee

Why Event-Based Vision?

Event-based vision measures log-based changes in brightness for each pixel in the sensor independently

In comparison to a typical camera, which captures via rolling shutter synchronously [1].

• Little motion blur
• High-dynamic range

Data point: (ts, x, y, p)

Prior Work on Event-Based Object Detection

Image Reconstruction [2]

​

Event Tensors:

• Event Volumes [5]
• Histograms
• Time Surfaces

​

RED [3] (SoTA)

​

​

backbone

heads

Our Approach - Tracking by Detection

CenterTrack [4]

Event Volumes [5]

Volume t

Volume t-1

Tracks t-1

Tracks t

Sizes t

Offsets t

Inputs

Outputs

Current Results & Future Work

Results:

• Optimized Event Volume Data Loading - From 4 to 0.001 seconds
• Currently, training is not converging
• Labels are incredibly sparse for the event data - taken every 5 seconds
• Doesn’t allow to learn tracking since we need adjacent labels

Future Work:

• Balance positive and negative samples to 50/50
• Trim videos to only use the event data that has labels
• Sample from random videos to make a diverse batch

Citations

[1] E. Mueggler, B. Huber, and D. Scaramuzza, Event-based, 6-DOF Pose Tracking for High-Speed Maneuvers. IROS 2014

[2] H. Rebecq, R. Ranftl, V. Koltun, and D. Scaramuzza. High speed and high dynamic range video with an event camera. IEEE Transactions on Pattern Analysis and Machine Intelligence 2019.

[3] E. Perot, P. Tournemire, D. Nitti, J.Masci, and A. Sironi. Learning to Detect Objects with a 1 Megapixel Event Camera. NeurIPS 2020.

[4] X. Zhou, V. Koltun, and P. Krähenbühl. Tracking Objects as Points. ECCV 2020.

[5] A. Zhu, L. Yuan, K. Chaney, and K. Daniilidis, Unsupervised event-based learning of optical flow, depth, and egomotion. CVPR 2019.

​

Distortion-Free Wide-Angle Portraits on Camera Phones

Ji Liu, Zhuoqian Yang

​

12/17/2020

Introduction and

Method Overview

Introduction

1. Modern mobile phones are equipped with wide angle cameras
2. Wide angle cameras introduce artifacts caused by perspective projection
3. An algorithm is needed to do correction on-site immediately after photo taking

Introduction

Before correction After correction

Method Overview

Results on Images of Our Own

Naive Stereographic Result

input

naive stereographic

flow

Subject Mask Segmentation Result

input

face mask

Correction Result

Before correction

After correction

Method - Subject Mask Segmentation

1. Use Detectron 2 to extract person mask
2. Use Dlib to extract face bounding box
3. Intersect the two to get face mask

Method - Stereographic Projection

Stereographic projection can picture the 3D world onto a 2D plane with minimal conformal distortion, at the expense of changing the curvature of long lines.

We enforce the stereographic projection locally on face regions to correct perspective distortion.

Given the camera focal length f , we compute the stereographic projection from the input using a radial mapping:

​

​

Method - Mesh Placement

Before and after naive stereographic, we can observe strong artifacts around the face regions.

Method - Local Face Undistortion

We minimize the following energy function to determine an optimal mesh.

​

​

where Et is the weighted sum of several energy terms including Face Objective Term, Line-Bending Term and Regularization Term.

​

Method - Regularization Term

we regularize the mesh by encouraging smoothness between 4-way adjacent vertices using a regularization term

​

Method - Line-Bending Term

On the boundary between the face and background, straight lines may be distorted because the two regions follow different projections. We preserve straight lines by encouraging the output mesh to scale rather than twist by adding a line-bending term. The line-bending term penalizes the shearing of the grid, and therefore preserves the edge structure on the background

​

​

Method - Mesh Boundary Extension

Similar to gradient-domain image processing, a simple boundary condition by forcing

vi = pi on the mesh boundary would do the job. However,

it creates strong distortions when faces are close to image boundary.

This distributes the distortion to padded vertices, and reduces artifacts near the boundary of the output image.

Method - Similarity Constraint

Constrain the transformation around each facial area to be a similarity transformation that preserves scale

Extension of the Method to Other Object Categories

Thank you!

Mirror, Mirror On the Wall: Detecting Mirrors and Correcting Point Clouds

Sachit Mahajan

sachitma

345

345

345

345

Motivation

• Mirrors are common in indoor scenes
• Mirrors appear as doors or entranceways to visual sensors.
• For scene reconstruction- point clouds have to be be manually corrected by giving the mirror boundaries
• Robots performing exploration cannot differentiate between doors and mirrors, and end up planning paths through mirrors or forming incorrect maps

Matterport3D: Learning from RGB-D Data in Indoor Environments

346

346

346

346

Motivation

Taken from Phone (camera+Lidar) and ‘3d Scanner App’

347

347

347

347

Setup

​

Logitech c920

Velodyne

VLP-16

AprilTAG

IMU

348

348

348

348

Methodology

349

349

349

349

Calibrate Extrinsics and Locate Mirror/Entrances

• Preprocess pointcloud: Median Filtering + Bilateral Filtering
• Use depth discontinuities to detect possible mirrors

350

350

350

350

Classify mirrors and Remove Points

Two different methods

1. Using self-reflection (mirrored fiducial marker) to identify mirror plane
• Detect the 6dof pose of AprilTag, this gives the location of one point on the mirror plane and its orientation
• Novel method: Divide the image into grids, use a region growing to identify mirror planes. Heuristics used to classify grid as mirror-
1. Part of AprilTag lies in Grid
2. Depth greater than mirror plane and high depth discontinuities
3. Grid normals differ from that of mirror plane/ High normal variance (sum of dot products)
4. Image intensities have high variance (mirror boundaries)
• Requires very accurate calibration
2. Segmentation mirrors in images by modeling semantical and low-level color/texture discontinuities -
• MirrorNet from ‘Yang, Xin and Mei, Haiyang and Xu, Ke and Wei, Xiaopeng and Yin, Baocai and Lau, Rynson W.H.Where Is My Mirror?’
• Works for only one mirror in the scene, fails very often when there is no mirror in the scene

351

351

351

351

Classify mirrors and Remove Points

Mask Obtained From Region Growing

Mask Obtained From MirrorNet

352

352

352

352

Results

353

353

353

353

Future Work

• Correct the errors in region growing,
• Region growing should be done using a coarse-to-fine approach such that mirror boundaries can be better estimated
• Instead of removing point clouds they can be corrected, since we know our pose and mirror plane we can find which 3D point they actually correspond to and reproject them.
• Once a mirror is found and estimated, it should be tracked through the scene even after the AprilTag is not visible

354

354

354

354

Image Deblurring Using Inertial Measurement Sensors

Kevin O’Brien

kobrien

Problem Motivation/Description

Blurry images stink! She gets it --------------------->

​

Deconvolution is difficult

-Optimization with no closed-form solution

-Different optimization strategies perform well for some types of scenes, poorly for others

-Generally want to enforce sparse gradients in the image to preserve sharp edges

​

​

​

​

​

​

​

Blind Deconvolution

Simultaneously solve for the sharp image I and the blur kernel K

-This is an even harder optimization problem, solving for two things at once

-Advances in blind deconvolution come from introducing additional constraints

-sharp edge prediction, color statistics

​

​

​

​

​

​

​

​

​

MAIN IDEA

When we’re dealing with camera shake, use the camera’s motion to help inform the blind deconvolution problem

​

​

​

​

​

​

Blur as a Homography

For small camera movements during exposure, blurring can be parametrized as small planar warps

Sensor Data Processing

Accelerometer readings give linear acceleration (m/s^2)

Gyroscope readings give rotational velocity (rad/s)

​

Procedure:

1. Integrate rotational velocity to get angular position, convert to rotation matrix
2. Rotate accelerometer readings into starting reference frame
3. Integrate accelerometer readings to get relative translations
4. Use recovered rotations/translations for homography calculation

​

Implementation Status

ADMM deblurring optimization with known blur kernel

Implementation Status

Deblurring using ground truth camera positions

Implementation Status

Use raw sensor values next

Thank you!

Light field 3D reconstruction

Wenxuan Ou (Owen)

​

Abstract

• Objective: Obtain high resolution depth map from structured light field
• Primary reference:
• Kim, Zimmer. “Scene Reconstruction from High Spatio-Angular Resolution Light Fields.” ACM transactions on graphics 32.4 (2013): 1–12. Web.
• Other materials:
• Lin, Yeh. “Depth Estimation for Lytro Images by Adaptive Window Matching on EPI.” Journal of imaging 3.2 (2017): 17–. Web.
• Motivation: Extend the algorithm to structured lightfield (Lytro image), achieve 3D reconstruction in a single-shot.

Algorithm

• Depth estimation from epipolar image (EPI)
• Depth relationship to disparity:

​

• Calculate the depth score of each possible disparity values, select the best estimate:

• Refine depth estimation based on confident masking and radiance/color

​

• Compute depth only at confident locations, thresholding

​

​

Reference: Kim et al

Algorithm

• Obtain EPI from Lytro image
• Does not need rectification

​

​

Reference: Lin et al

Results

Original image

EPI

Disparity

Confidence

Results

Original image

My depth map

AFI depth map

Discussion and conclusion

• Lytro image does not have large enough disparity to generate reliable depth estimation. Need a larger camera transition to capture fine details.
• Depth from EPI is strongly influenced by texture of the objects. Homogeneous region leads to ambiguous depth score.
• Future extension:
• The homogeneous problem can be potentially solved by doing a further step fine-to-coarse refinement.

Thank you !

3D Scanning with Structured Light

George Ralph

Hello to everyone passing through!

Good luck on your presentations ♥

Background

• Structured light reconstruction is dependent on accurate pixel correspondences

​

• Global Illumination
• Defocus

Reducing Global Illumination Effects

Pattern Binarization

Two-Image Binary Codes

One-Image XOR Gray Codes

Other Filtering Techniques

Median Filtering of Correspondences

Point Cloud Filtering

Examples

Examples

Examples

Thank You!

Feel free to ask questions.

Adaptive SPAD Imaging with Depth Priors

Po Ryan

Background

• High temporal resolution
• Only records first arriving photon
• Performs fine under no ambient light
• Pile-up effect under ambient light
• Coates’ Estimate used for pile-up compensation
• Earlier bins carry more information
• Earlier bins have better estimates

SPAD Sensor

[Gupta et al. “Photon-Flooded Single-Photon 3D Cameras” CVPR 2019]

Depth Estimate

Establish Depth Prior

Adjust Gate and Active Time of SPAD

Method

Stereo Pairs

Depth Estimate

Adaptive SPAD

Conventional SPAD

• Poor estimates for occluded pixels
• Certainty of estimate unknown

Stereo Depth Prior

[Xia et al. “Generating and Exploiting Probabilistic Monocular Depth Estimates” CVPR 2020]

CNN

Single RGB Image

Depth Estimate

Uncertainty

Adaptive SPAD

Conventional SPAD

NN-Based Probabilistic Single RGB Depth

• Estimates are probabilistic
• Shifting of gate and active time can be determined through uncertainty

Application to Async. Acquisition

[Gupta et al. “Asynchronous Single-Photon 3D Imaging”]

• Asynchronous Acquisition
• Uniform shifting of gate
• Every bin is near beginning of transient for at least one shift
• Depth prior can be utilized depth prior to determine shifting pattern instead of relying on uniform shifting schemes
• Depth prior based coding schemes yield at least 20% increase in accuracy

Integration with NN-Based Pipelines

[Siddiqui et al. “An Extensible Multi-Sensor Fusion Framework for 3D Imaging” CVPR 2020]

• Stand-alone adaptive SPAD may not give the best final measurements, but improves raw SPAD measurements
• Better raw measurements means that other multi-sensor fusion methods would also perform better

SPAD Only Adaptation?

• Per bin probability of error can be estimated
• CR-bound for variance of Coates’ estimate
• Chernoff Bound on probability of error for each bin
• Per bin probability is quite uniformly distributed making adaptive gating very ill-posed

[Pediredla et al. “Signal Processing Based Pile-up Compensation for Gated Single-Photon Avalanche Diodes”]

Thanks!

Computational Photography Project

CMU-15663

Tejas Zodage , Harsh Sharma

Handheld Photometric Stereo

Motivation

To develop a simple device for dense 3D reconstruction of static scene.

​

​

Problem definition

A set of images taken by camera with a fixed Point light source attached to it.

Add photometric constraints to multi-view stereo to get a dense depth map.

​

​

​

​

391

Problem definition

Camera

Point light source rigidly attached to the camera

Input: m images

.

.

.

​

Output: depth image

algorithm

Procedure

1. Use images from multiple views to get a per-pixel depth map estimate, using plane sweep stereo
2. Get estimates for normals, albedos and ambient lighting using RANSAC near-light photometric stereo.
3. Refine the depth, normal and albedo estimates by formulating an optimization problem with the photometric and depth smoothness costs.

​

​

393

Initial estimates

1. Plane sweep stereo for per pixel depth estimate-
1. Choose a reference frame and warp images.
2. Pixel locations are estimated to be on that plane where their NCC score is maximum.
2. RANSAC based near-light photometric stereo-
• Normals and albedos

Optimization

​

​

• Photoconsistency cost - account for varying lighting, since the light source is attached to the moving camera.
• Surface normal cost - Preferred depth estimates are those which are consistent with the surface normal estimates.
• Smoothness cost - penalize large discontinuities in the depth map

​

​

Model the problem into a graph-cut optimization problem and solve for the constraints jointly

​

​

​

​

395

Can’t we solve it like HW5?

No!

​

Data generation

Blender + Mitsuba

397

1. Generate scenes using blender (easy for visualization)
2. Render scenes using mitsuba (photorealistic rendering)

Results

Plane sweep stereo - depth map

398

Challenges

1. Data generation -
1. No readily available dataset
2. Graph-cut implementation (np hard problem) -
• Suggested algorithm for graph cut by authors - alpha beta swapping algorithm.
• Takes hours for optimization on images of size > 100 px X 100 px

​

Future work

• Data generation -
• Real world data.
• Graph-cut alternatives -
• Look for better solutions to solve energy minimization on graphs.
• Deep learning based methods?

​

References

• Higo, Tomoaki, et al. "A hand-held photometric stereo camera for 3-d modeling." 2009 IEEE 12th International Conference on Computer Vision. IEEE, 2009.
• Plane sweeping stereo https://github.com/ReynoldZhao/Stereo/blob/master/src/plane_sweep_stereo.py
• Graph cuts�https://github.com/roflmaostc/Fast-Approximate-Energy-Minimization-via-Graph-Cuts

​

Image Morphing

Jiazheng Sun

Sample effect

Step 1 - set up

• Load texture (image)
• Coordinates system conversion: world to canvas (cartesian)
• Move (0, 0) from top left to middle; flip y’s direction
• x / world_size * canvas_size - (canvas_size/2)
• (canvas_size/2) - y / world_size * canvas_size

Step 2 - build control points

• Define basic anchor points

• Define control points and map them to anchor points using Barycentric coordinates

Step 3 - perturb, interpolate & blend

• Perturb the control points (random or preset trajectory)

• Interpolate with curves; blending guarantees C2 continuty at intersection

Curves & Blending

• Neville interpolation:

​

• Blending function:

​

Citation: Cem, Yuksel. 2020. A class of C2 interpolating splies. ACM Transactions on Graphics, 39, 5, 2020

​

​

Step 4 - texture mapping

Additional functionalities

• Register key frames, and interpolate between frames to get customized animation.
• Customize anchor points.
• Directly select and drag control points.
• Precompute a trajectory for each control points, so they move automatically.

Precompute trajectory demo

3D Scene Flow Using Lightfield Images

Kevin Wang

Lightfield Image Structure

• 4-dimensional L(x, y, u, v):
• X and y are the camera coordinates
• U and v are the pixel coordinates
• Given two lightfield images, we want to find 3D scene flow (Motion in X, Y, and Z)

Relating Ray Flow to Optical Flow

• Optical Flow Equation: I_XV_X + I_YV_Y + I_t = 0
• Raw Flow Equation: L_XV_X + L_YV_Y + L_zV_z + L_t = 0
• L_z is found with relation to the pinhole camera position, partial derivatives of the X and Y positions and a fixed depth constant

Lucas-Kanade

• Because of the form of the ray flow equation, and it’s similarity to the traditional optical flow equation, we can use existing methods to estimate depth motion.
• For Lucas-Kanade, finding local motion (assumes that pixels in the same area will have the same motion)

Horn-Schunck

• Assumes that the flow between pixels is “smooth” (global)

​

​

​

• Minimum can be found using the Euler-Lagrange Equations

​

​

Example Results

A Computational Approach for Obstruction- Free Photography

Wesley Wang

(wesleyw)

Overview:

• Take photos through occluding elements like windows and fences
• Remove those occlusive elements from the photos
• Capture video (or image sequence) while moving the camera
• Background scene must be far enough that it remains static

​

Overview:

• Occlusive Foreground Objects

Overview:

• Window Reflections

Implementation:

• Treat image as composition of a background layer and obstruction layer

​

​

​

​

• I_O is occlusion (foreground) layer
• I_B is background layer
• A is alpha mask where occlusion exists in image

​

​

Implementation:

• Solve for those image components using info from other frames
• Also solve for the warping W(V) of layers between frames

​

​

​

• Solve optimization problem

​

​

Results:

• Motion estimation between frames after processing + edge detection

​

​

Questions?

​

Confocal Stereo

Hiroshi Wu

Confocal Stereo

• A method to extract depth information using multiple images of the same scene with varying focus and aperture
• https://people.csail.mit.edu/hasinoff/pubs/hasinoff-confocal-2006.pdf

​

​

In assignment 4...

1. Simulated a focal-aperture stack based on image from plenoptic camera
2. Constructed AFI (aperture-focus-image) for each pixel
3. Used direct variance evaluation to determine depth d at each pixel

The whole process (as detailed in paper)

1. Aperture-focal stack image acquisition using a DSLR
2. Relative exitance estimation (radiometric calibration)
3. Image alignment (lens warp calibration)
4. AFI construction
5. Depth estimate by AFI model fitting

​

Although the core idea is the same, the whole process is more involved

1. Aperture-focal stack acquisition

• Depth from blur: bigger bokeh = better resolution
• To maximize bokeh:
• Used a lens with f/2.8 aperture
• Zoomed to the maximum focal length on that lens: 70mm
• Took photos of the same scene for 80 focus settings (0.38m ~ 3.4m) and 16 apertures (2.8 ~ 16)
• Used gphoto2 to automate the process; one set of data took 1 hour to capture

Example: f=50, a=2.8

2. Relative exitance estimation

• Since we compare pixel values across the AF stack, we need to ensure that same values mean the same thing
• E.g. vignetting
• We take photos of a uniform white paper using similar focus settings and aperture values and check the exitance at each pixel and each setting so we can adjust for brightness differences

3. Image alignment

• Changing focus distance actually changes the scene due to lens mechanics:

​

​

​

​

• Modelled as a magnification + quadratic distortion

Smaller FOV

3. Image alignment

• Use focal stack of a scene with distinct features to obtain the warp relationships between different focus, and use nonlinear optimization to solve for the parameters

4. AFI

• Example from paper:

​

​

• Example from my data:

5. AFI model fitting

• There is a distinct pattern to AFIs
• Due to different amounts of blurring: similar blurs should have similar colors
• We derive blur = where the real-life distance of configuration x is due to the thin lens model.
• Quantizing theoretical blur gives us these bands; we seek to minimize the variance within the bands

6. Result from piano dataset

Image smoothing via L0 Gradient Minimization

Ye Wu

Objective

• Globally maintain and possibly enhance the most prominent set of edges by increasing steepness of transition while not affecting the overall acutance.
• A strategy to confine the discrete number of intensity changes among neighboring pixels, which links mathematically to the L0 norm for information sparsity pursuit.

Algorithm

Pipeline

Start

Gradient y

Gradient x

Mix 1

Pipeline

Mix 1 with threshold

Gradient x new

Gradient y new

Mix 2

end

Results

k=2.0

k=3.0

l=0.1

l=0.01

l=0.001

l=0.0001

l=0.00001

Results

k=2.0

k=4.0

k=6.0

k=8.0

k=10.0

l=0.1

l=0.03

l=0.00001

Comparison

L0 minimization

3.05s

Bilateral filtering

64.13s

Application

Image abstraction

​

Application

Clip-art compression artifact removal

​

Application

Combination of BLF and L0 method

​

Origin

L0

BLF

BLF+L0

Main References

http://www.cse.cuhk.edu.hk/~leojia/projects/L0smoothing/

https://www.caam.rice.edu/~yzhang/reports/tr0710_rev.pdf

Thank you!

Q&A

Fast Reflection Removal using Hierarchical Bilateral Grids

Zhichao Yin

(zhichaoy@)

Reflection removal

451

Photo taken through glass

Ground-truth background/transmissive layer

High-res reflection removal

• Q: can existing approaches work well for high-res input images?

​

• Challenges:
• slow runtime
• memory consumption
• degraded quality (CNNs are not scale equivariant)

452

High-res challenges – runtime & memory

453

CPU runtime & memory vs. input resolution on real-world test set (208 images)

1. Zhang et al. "Single Image Reflection Separation with Perceptual Losses." CVPR, 2018
2. Niklaus et al. ”Learned Dual-view Reflection Removal.” In Submission, 2020

High-res challenges – degraded quality

454

Niklaus etal.¹

Result upsampled from prediction with 512-res input

1. Niklaus et al. ”Learned Dual-view Reflection Removal.” In Submission, 2020

High-res challenges – degraded quality

455

Niklaus etal.¹

Result upsampled from prediction with 1024-res input

1. Niklaus et al. ”Learned Dual-view Reflection Removal.” In Submission, 2020

High-res challenges – degraded quality

456

Niklaus etal.¹

Result upsampled from prediction with 2048-res input

1. Niklaus et al. ”Learned Dual-view Reflection Removal.” In Submission, 2020

HDRNet¹ works well for tone mapping

457

1. Gharbi et al. ”Deep Bilateral Learning for Real-Time Image Enhancement.” SIGGRAPH, 2017

Input

Output

But HDRNet¹ cannot work for dereflection

458

1. Gharbi et al. ”Deep Bilateral Learning for Real-Time Image Enhancement.” SIGGRAPH, 2017

Addition of reflection parts makes the transformation highly non-linear in color space.

Input

Output

Our method – overview

459

High-res�Comparison

Input, Resolution: 2048 x 1536

GT, Resolution: 2048 x 1536

High-res�Comparison

Input, Resolution: 2048 x 1536

Niklaus etal.

1. Niklaus et al. ”Learned Dual-view Reflection Removal.” In Submission, 2020

High-res�Comparison

Input, Resolution: 2048 x 1536

Ours

High-res�Comparison

Input, Resolution: 2048 x 1536

Zhang etal.

1. Zhang et al. "Single Image Reflection Separation with Perceptual Losses." CVPR, 2018

High-res�Comparison

Input, Resolution: 2048 x 1536

Ours

High-res�Comparison

Input, Resolution: 2048 x 1536

High-res�Comparison

GT, Resolution: 2048 x 1536

High-res�Comparison

Niklaus etal.

1. Niklaus et al. ”Learned Dual-view Reflection Removal.” In Submission, 2020

High-res�Comparison

Ours

High-res�Comparison

Zhang etal.

1. Zhang et al. "Single Image Reflection Separation with Perceptual Losses." CVPR, 2018

High-res�Comparison

Ours

Quantitative Evaluation

471

LPIPS ↓

SSIM ↑

PSNR ↑

1. Zhang et al. "Single Image Reflection Separation with Perceptual Losses." CVPR, 2018
2. Niklaus et al. ”Learned Dual-view Reflection Removal.” In Submission, 2020

Runtime Comparison

472

CPU

GPU

Relative Improvement

1. Zhang et al. "Single Image Reflection Separation with Perceptual Losses." CVPR, 2018
2. Niklaus et al. ”Learned Dual-view Reflection Removal.” In Submission, 2020

Lensless Imaging with DiffuserCam

Emily Zheng

​

Exploring Lensless Imaging

• Cameras with lenses are most commonly used to capture images
• Lensless imaging systems
• Can be more lightweight
• Can be constructed easily without the precision of a normal lens using everyday materials
• DiffuserCam: replace lens with a diffuser (Scotch tape)
• From DiffuserCam Paper

DiffuserCam Setup + Calibration

Setup

• RaspberryPi PiCamera with lens removed (only sensor left)
• Place a diffuser at focal length of PiCamera (double sided Scotch tape)
• Create aperture using black paper / tape

Calibration

• Take a picture of a point source through pinhole to get psf of the diffuser

Example PSF and Image Capture

Reconstruction with Deconvolution

• Alternating direction method of multipliers (ADMM)
• Optimization problem
• Assume image of scene captured is result of some convolution of original scene with blur of diffuser
• Blur from diffuser is known from the psf
• Solve for original scene image using ADMM
• Can reconstruct original scene with a single diffused image

Results

Results

Remaining course logistics

480

• Optional homework assignment 7 is due on 12/18 at 23:59 ET.

- Gradescope submission site available.

​

• Final project reports are due on Sunday 12/20 at 23:59 ET.

- Gradescope submission site available.

​

• One upcoming computational photography talk:

- Tuesday 12/22, 1 – 2 pm, Grace Kuo (lensless cameras, diffuserCam, AR/VR displays).

​

• Returning borrowed equipment:

https://docs.google.com/spreadsheets/d/1CVg7nUbI701pvZFPX3BR0uzKB76Y6PEl3tXmq1UF4AU/edit#gid=0

Class evaluation*s* – please take them!

481

• CMU’s Faculty Course Evaluations (FCE): https://cmu.smartevals.com/

​

• TA evaluation: https://www.ugrad.cs.cmu.edu/ta/F20/feedback/

​

• 15-463/663/862 end-of-semester survey: https://docs.google.com/forms/d/e/1FAIpQLSew6XHzagr0KqWPFCmdoVmnHXQR5vfi4A6nEG0jMg6O1lBixA/viewform

​

• Please take all three of them, super helpful for developing future offerings of the class.

​

• Thanks in advance!