1

DDA2082 Independent Study II

PatchMatch Multi-View Stereo

Reporter: Xiang Fei

School of Data Science

The Chinese University of Hong Kong, Shenzhen

Outline

2

Part 1 Introduction to PatchMatch Stereo

Part 2 Broad Adaptive Checkerboard Sampling

Part 3 Dynamic Multi-Hypothesis Joint View Selection

Part 4 Experiment Results

Slanted Support Windows

Fronto-Parallel Windows

A classic window model, which refers to windows directly in front of the camera that are parallel to the image plane after epipolar correction.

​

The characteristics of the window:

• The projection lengths of any line segments in the window on the left and right images (epipolar line image pairs) are equal.
• All spatial points in the window have the same depth. From D=bf/d, it can be seen that the disparity of the projection point of the spatial point on the image is also the same.

Slanted Support Windows

Slanted Support Windows

Fig. 1: The illustration of Fronto-parallel windows and slanted support windows, which shows the support regions (in 1D). The points of green surface shall be reconstructed. Support regions are show by red bars. (a) Fronto-parallel windows at integer disparities as used in standard methods. (b) Slanted support windows. The 3D plane is estimated at each point.

Slanted Support Windows

Disparity Plane

• Disparity estimation problem -> plane estimation problem.
• Stereo matching is to find the parameters of the optimal plane for each pixel, that is, to find the plane with the smallest aggregation cost for each pixel

Disparity Estimation Based on PatchMatch

The core idea of PatchMatch is: in images, the disparity planes of all pixels in a pixel block of a certain size can be approximated as the same. The goal of the algorithm is to find all the disparity planes of the image.

​

The procedures are as follows:

​

1. Random Initialization

Disparity Estimation Based on PatchMatch

2. Disparity Propagation

Disparity Estimation Based on PatchMatch

3. Plane Refinement

Outline

9

Part 1 Introduction to PatchMatch Stereo

Part 2 Broad Adaptive Checkerboard Sampling

Part 3 Dynamic Multi-Hypothesis Joint View Selection

Part 4 Experiment Results

Propagation Schemes in Previous Methods

Fig. 2: (a) Sequential propagation. (b) Symmetric checkerboard propagation [1]. (c) Adaptive checkerboard propagation. The light red areas in (c) show sampling regions [2]. The solid yellow circles in (b) and (c) show the sampled points.

[1] S. Galliani, K. Lasinger, and K. Schindler. Massively parallel multiview stereopsis by surface normal diffusion. In Proceedings of the IEEE International Conference on Computer Vision, pages 873–881, 2015.

[2] Xu, Qingshan and Wenbing Tao (2019). Multi-Scale Geometric Consistency Guided Multi-View Stereo. In: Computer Vision and Pattern Recognition (CVPR).

Problem: Time-consuming and inefficient.

Problem: Samples from eight fixed positions, leading to a decrease in accuracy.

Problem: Narrow extensions in four directions, resulting in many pixels not being considered when updating.

Broad Adaptive Checkerboard Sampling

Fig. 3: Propagation scheme of Broad Checkerboard Sampling. Blue line shows the window and the four areas. The solid yellow circles show the sampled points.

• A neighborhood window centered at the updating pixel, which is divided into four areas.
• Choose the best two hypotheses in each area for propagation based on the aggregation costs.
• This method broadly considers all the pixels in the neighborhood window, instead of extending in a specific direction, which facilitates capturing correct hypotheses in large low-texture areas.

Outline

12

Part 1 Introduction to PatchMatch Stereo

Part 2 Broad Adaptive Checkerboard Sampling

Part 3 Dynamic Multi-Hypothesis Joint View Selection

Part 4 Experiment Results

View Selection in Previous Methods

[1] S. Galliani, K. Lasinger, and K. Schindler. Massively parallel multiview stereopsis by surface normal diffusion. In Proceedings of the IEEE International Conference on Computer Vision, pages 873–881, 2015.

[2] Xu, Qingshan and Wenbing Tao (2019). Multi-Scale Geometric Consistency Guided Multi-View Stereo. In: Computer Vision and Pattern Recognition (CVPR).

Dynamic Multi-Hypothesis Joint View Selection

To obtain a robust multi-view matching cost for each pixel, we can further leverages these the obtained eight structured hypotheses to infer the weight of every neighboring views. Firstly, build a matching cost matrix for each of the eight pixels:

Good matching cost:

Bad matching cost:

Dynamic Multi-Hypothesis Joint View Selection

This makes good views more discriminative. The weight of each selected view can be defined as:

The multi-view aggregated cost:

Outline

16

Part 1 Introduction to PatchMatch Stereo

Part 2 Broad Adaptive Checkerboard Sampling

Part 3 Dynamic Multi-Hypothesis Joint View Selection

Part 4 Experiment Results

Experiment Results

Depth map comparisons

Much better estimation in low-texture area, even if the area is very large!

Results

Point cloud comparisons

Much better estimation in low-texture area, even if the area is very large!

Thanks for your attention!