Ohad Fried, Shai Avidan and Daniel Cohen-Or

In Computer Graphics Forum, volume 36,629 pages 183{194. Wiley Online Library, 2017.

Patch2Vec: Globally Consistent Image Patch Representation

Outline�

• Introduction
• Related work
• Method
• Evaluation
• Applications

​

Introduction

• Analyze patches by embedding them to a vector space, in which the Euclidean distance reflects patch similarity

Related Work

• MIKOLOV T., YIH W., ZWEIG G.: Linguistic regularities in continuous space word representations. In Human Language Technologies
• We are inspired by the work on Word2Vec that maps words with similar meaning to vectors with small distances between them.

Related Work

• SCHROFF F., KALENICHENKO D., PHILBIN J.: Facenet: A unified embedding for face recognition and clustering. In CVPR
• The network, termed FaceNet, learns a mapping from face images to a compact Euclidean space where distances directly correspond to a measure of face similarity.

Method

Method - Patch Embedding �

• Our goal is to embed image patches into a low-dimensional representation, such that l2 distances in the representation space correspond to some notion of patch similarity, with a focus on textured patches.

​

• we would like to learn a universal embedding operator f(p) for two given patches p1 and p2, the distance ║ f(p1)− f(p2) ║is small if p1 and p2 are similar textures, and large otherwise.

​

• We use Convolutional Neural Networks (CNN) to learn Patch2Vec.
• We use labeled images to learn the embedding in a supervised manner.

​

Method - Patch Embedding �

• SCHROFF F., KALENICHENKO D., PHILBIN J.: Facenet: A unified embedding for face recognition and clustering. In CVPR (June 2015), pp. 815–823

GoogleLeNet

Method - Patch Embedding �

• SCHROFF F., KALENICHENKO D., PHILBIN J.: Facenet: A unified embedding for face recognition and clustering. In CVPR (June 2015), pp. 815–823

Method - Patch Embedding �

• f(p) : For two given patches p1 and p2, the distance ║ f(p1)− f(p2) ║is small if p1 and p2 are similar textures, and large otherwise.
• triplet loss function:

We use a triplet loss for training: given an anchor patch Pa which makes a positive pair with Pp and a negative pair with Pn , the loss for a single triplet is defined as:

​

​

where m is a margin value (set empirically to 0.2),and [x]+ is defined as max{0, x}.

The triplet loss is a sum over all anchor-positive-negative triplets in the dataset D:

Method - Patch Embedding �

• Data set:
• Berkeley Segmentation Dataset (BSDS500)
• Randomly sample 50,000 32× 32 patches from each image in the BSDS500 training set

Method - Patch Embedding �

Patch size: 32x32 16x16

● A simple way to visualize the embedding is to project the 128D codes on the three leading principal components, producing a pseudo-RGB image.

Method - Patch Embedding �

Patch size: 32x32 16x16

Evaluation

Evaluation

• We measure the distance d between the two patch embeddings

​

​

​

• We can define a binary classifier C(p1, p2) that determines whether the patches belong to the same segment or not:

​

​

​

• We calculate the area under the curve (AUC) of the receiver operating characteristic curve for all t values.

Evaluation

Evaluation

Applications

Single-Click Segment Selection�

• Given a photo I, we calculate patch embeddings f(p) for each 32 × 32 image patch in a preprocessing step.
• At runtime, the user clicks a single pixel location that corresponds to patch Pc. For all other patches, we calculate the embedding distance

​

​

​

• Next, we threshold the distances using Otsu’s method , OTSU N.: A threshold selection method from gray-level histograms. IEEE Tran. on Sys. ,to produce a binary selection mask.

Single-Click Segment Selection�

Single-Click Segment Selection�

• Another important property is that training on pixel-accurate masks allows a click near the segment boundary to produce a successful mask.

Single-Click Segment Selection�

super-pixels

• Our method does not follow intra-texture edges, as can be seen when plotting the average super-pixel color in (b2) and (c2) — we get much smoother colors in the fur region. See text for more details.

We compare (a) SLIC [ASS∗ 12], (b1) a SLIC variant guided by edge detection [DZ13] and (c1) our method. Both (b1) and (c1) follow meaningful edges better than vanilla SLIC (see, e.g., zoomed regions).

super-pixels

END�