1 of 46

Large Scale Camera Array Calibration via SfM

Capstone Paper Survey Presentation

2 of 46

Large Scale Camera Array Calibration via SfM

Capstone Paper Survey Presentation

3 of 46

Team

Advisors

3

Sanjana Gunna

​

Gaini Kussainova

4 of 46

Project Summary

4

5 of 46

Introduction

​

Dome-like structure called “Mugsy”

​

​

5

Mugsy v1

Mugsy v2

6 of 46

Introduction

​

Dome-like structure called “Mugsy”

​

Use case - Virtual Human (Photo realistic avatars) Generation

​

​

6

7 of 46

What is the problem?

Calibration of the cameras in the dome is time-consuming!

​

​

7

Mugsy v1

Mugsy v2

8 of 46

What is the problem?

Calibration of the cameras in the dome is time-consuming!

​

Extrinsics dont change (1 day - 1 week)

8

Mugsy v1

Mugsy v2

9 of 46

What is the problem?

Calibration of the cameras in the dome is time-consuming!

​

Extrinsics dont change (1 day - 1 week)

​

Intrinsics change!

9

Mugsy v1

Mugsy v2

Images from the Multiface Dataset

10 of 46

Why?

Help save time!

10

11 of 46

How?

Current Approach

  • Despite high precision calibration, Calibration Time ~ 30 mins

​

  • Corner Detection is computationally expensive

11

Source of Images - https://openaccess.thecvf.com/content_ICCV_2017/papers/Ha_Deltille_Grids_for_ICCV_2017_paper.pdf

Deltille Grid

12 of 46

How?

Structure from Motion!

12

Face data

SfM

Source of Image - https://colmap.github.io/

13 of 46

How?

Structure from Motion!

13

Face data

SfM

Sparse Reconstruction

Motion

Parameters

(Intrinsics / Extrinsics)

Source of Image - https://colmap.github.io/

14 of 46

Related Works

14

15 of 46

Paper I

15

Wuu et al. Multiface: A Dataset for Neural Face Rendering. 10.48550/arXiv.2207.11243.

16 of 46

Multiface dataset

16

Mugsy v1 Mugsy v2

Source of images: https://arxiv.org/pdf/2207.11243.pdf

17 of 46

17

Source of images: https://arxiv.org/pdf/2207.11243.pdf

18 of 46

18

Source of images: https://arxiv.org/pdf/2207.11243.pdf

19 of 46

Capturing system

19

Mugsy v1 Mugsy v2

Source of images: https://arxiv.org/pdf/2207.11243.pdf

20 of 46

Dataset

20

  • Raw images
  • Unwrapped Textures
  • Tracked Meshes
  • Headposes
  • Audio
  • Metadata

21 of 46

Dataset

21

  • Raw images
  • Unwrapped Textures
  • Tracked Meshes
  • Headposes
  • Audio
  • Metadata

22 of 46

Paper II

22

Schönberger, Johannes & Frahm, Jan-Michael. (2016). Structure-from-Motion Revisited. 10.1109/CVPR.2016.445.

23 of 46

Incremental SfM Pipeline

23

Input Image Data

Triangulation

Bundle Adjustment

Sparse Reconstruction + Camera Calibration Data

Feature Extraction and Matching

Incremental

SfM pipeline

Source of images: https://demuc.de/tutorials/cvpr2017/sparse-modeling.pdf

24 of 46

Incremental SfM Pipeline

24

Input Image Data

Triangulation

Bundle Adjustment

Sparse Reconstruction + Shape Parameters

Feature Extraction and Matching

Incremental

SfM pipeline

Source of images: https://demuc.de/tutorials/cvpr2017/sparse-modeling.pdf

Morphable Face Model Shape Weights

3D Points

Metric Rectification

P = HP

X = H-1X

​

Loss = Reprojection Error + Geometry Error

​

Refine Camera Parameters and Shape Weights

Triangulation

25 of 46

25

25

Sf pipeline

Morphable Face Model Shape Weights

3D Points

Metric Rectification

P = HP

X = H-1X

​

Loss = Reprojection Error + Geometry Error

​

Refine Camera Parameters and Shape Weights

Triangulation

26 of 46

Correspondence Search

26

Input Image Data

Triangulation

Bundle Adjustment

Sparse Reconstruction + Camera Calibration Data

Feature Extraction and Matching

Incremental

SfM pipeline

Source of images: https://demuc.de/tutorials/cvpr2017/sparse-modeling.pdf

27 of 46

Correspondence Search

​

27

Input Image Data

Triangulation

Bundle Adjustment

Sparse Reconstruction + Camera Calibration Data

Feature Extraction and Matching

Incremental

SfM pipeline

Source of images: https://demuc.de/tutorials/cvpr2017/sparse-modeling.pdf

28 of 46

Correspondence Search

​

28

Input Image Data

Triangulation

Bundle Adjustment

Sparse Reconstruction + Camera Calibration Data

Feature Extraction and Matching

Incremental

SfM pipeline

Source of images: https://demuc.de/tutorials/cvpr2017/sparse-modeling.pdf

29 of 46

Correspondence Search

​

29

Input Image Data

Triangulation

Bundle Adjustment

Sparse Reconstruction + Camera Calibration Data

Feature Extraction and Matching

Incremental

SfM pipeline

Source of images: https://demuc.de/tutorials/cvpr2017/sparse-modeling.pdf

30 of 46

Incremental Reconstruction

30

Input Image Data

Initialization

Image Registration

Sparse Reconstruction + Camera Calibration Data

Feature Extraction and Matching

Incremental

SfM pipeline

Source of images: https://demuc.de/tutorials/cvpr2017/sparse-modeling.pdf

31 of 46

Incremental Reconstruction

31

Input Image Data

Triangulation

Bundle Adjustment

Sparse Reconstruction + Camera Calibration Data

Feature Extraction and Matching

Incremental

SfM pipeline

Source of images: https://demuc.de/tutorials/cvpr2017/sparse-modeling.pdf

32 of 46

Incremental Reconstruction

32

Input Image Data

Triangulation

Bundle Adjustment

Sparse Reconstruction + Camera Calibration Data

Feature Extraction and Matching

Incremental

SfM pipeline

Source of images: https://demuc.de/tutorials/cvpr2017/sparse-modeling.pdf

33 of 46

Incremental SfM Pipeline

33

Input Image Data

Triangulation

Bundle Adjustment

Sparse Reconstruction + Camera Calibration Data

Feature Extraction and Matching

Incremental

SfM pipeline

Source of images: https://demuc.de/tutorials/cvpr2017/sparse-modeling.pdf

34 of 46

Next Best View Selection

34

Lepetit et al. experimentally show that the accuracy of the camera pose depends on the number of observations and their distribution in the image.

​

  • Discretize image into a fixed-size grid
  • Increase the score of the image by weight if the cell is full
  • Choose the next image with the highest score

Source of images: https://openaccess.thecvf.com/content_cvpr_2016/papers/Schonberger_Structure-From-Motion_Revisited_CVPR_2016_paper.pdf

35 of 46

Bundle Adjustment

35

Input Image Data

Triangulation

Bundle Adjustment

Sparse Reconstruction + Camera Calibration Data

Feature Extraction and Matching

Incremental

SfM pipeline

Source of images: https://demuc.de/tutorials/cvpr2017/sparse-modeling.pdf

36 of 46

36

Source of ivideo:

37 of 46

Paper III

37

Wang et al, Structure from motion for ordered and unordered image sets based on random k-d forests and global pose estimation. ISPRS Journal of Photogrammetry and Remote Sensing. 147. 19-41.

38 of 46

Previously…

38

Input Image Data

Triangulation

Bundle Adjustment

Sparse Reconstruction + Camera Calibration Data

Image Matching

Incremental

Conventional SfM pipeline

39 of 46

In this paper

39

Input Image Data

Sparse Reconstruction + Camera Calibration Data

Image Matching

with k-d forest

Global Pose Estimation

  • ​

Triangulation

Bundle Adjustment

Proposed SfM pipeline

Incremental

40 of 46

Image Matching with random k-d forest

40

Example of a k-d tree in a 2D feature space

  • Built from the features extracted from all the images

​

  • Nearest Neighborhood (NN) search helps find images with highest overlap

​

  • Much faster than the conventional pairwise matching technique

Source of image - https://salzi.blog/2014/06/28/kd-tree-and-nearest-neighbor-nn-search-2d-case/

41 of 46

Global Pose Estimation

41

Relative Orientation Parameters

Rij, tij

Global Rotations

Ri

Global Translations

ti

Epipolar Graph

Source of image - https://www.researchgate.net/figure/Epipolar-graph-showing-valid-relative-orientations-between-view-pairs_fig3_224253054

42 of 46

Global Pose Estimation

42

Relative Orientation Parameters

Rij, tij

Global Rotations

Ri

Global Translations

ti

Epipolar Graph

C1

C2

C3

R21, t21

R32, t32

R13, t13

Source of image - https://www.researchgate.net/figure/Epipolar-graph-showing-valid-relative-orientations-between-view-pairs_fig3_224253054

Triplet Closed Loop Constraint

Outlier Removal

43 of 46

How does it compare?

43

*Run time is measured in seconds

  • Run time Comparison
  • Colmap_E - Colmap + Exhaustive Image Matching
  • Colmap_V - Colmap + VocMatch
  • Colmap_R - Colmap + Random k-d forest Matching

Source of image - https://www.sciencedirect.com/science/article/pii/S0924271618303058

44 of 46

Conclusion

44

45 of 46

Next steps…

  • Implement the baseline with Colmap
  • Compare and analyse the accuracy of calibration results and runtime
  • Improve the baseline results using Paper III
  • Look into other bottlenecks

45

46 of 46

Thank you!

46