1 of 24

Selecting Suitable Image Retargeting Methods with Multi-instance Multi-label Learning

Muyang Song, Tongwei Ren, Yan Liu, Jia Bei, and Zhihong Zhao

2 of 24

Introduction

3 of 24

Introduction

  • Image retargeting methods:
    • Seam carving
    • Non-homogeneous warping
    • Scale-and-Stretch method
    • Multi-operator method
    • Shift map method
    • Streaming video method
  • Each image retargeting method succeeds on some images but fails on others.

​

4 of 24

Introduction

  • An institutive strategy is generating target images with different image retargeting methods, and selecting the good results from them.
  • A better strategy is selecting the suitable methods from all candidate methods, and generating the target images by the selected methods.

5 of 24

Introduction

6 of 24

Image Retargeting Methods Selection

7 of 24

Image Characteristic Analysis

  • Designate some easy-to-find features and ask the users to manually annotate these features to represent original image characteristic accurately.

8 of 24

Selection Using Multi-instance Multi-label Learning

  • Treat an image feature as an instance and a suitable retargeting method as a label.
  • Each image may have multiple features and multiple suitable retargeting methods.
  • The selection of suitable image retargeting methods can be represented as a multi-instance multi-label learning problem.

9 of 24

Selection Using Multi-instance Multi-label Learning

  •  

10 of 24

Selection Using Multi-instance Multi-label Learning

  •  

11 of 24

K-Medoids Clustering

  • 給定k的值，隨機挑選k個點作為每一個群(cluster)的中心點(medoids)
  • 計算剩下的每個點和每個群的中心點的距離，距離哪個群的中心點最近，那個點就會被分到那個群
  • 重新計算每個群的中心點：將群中的點兩兩算出距離，新的中心點就是與其它點的距離和最小的那一個
  • 重複第二和第三個步驟，直到整個群不再變化

​

12 of 24

Hausdorff Distance

  •  

13 of 24

Selection Using Multi-instance Multi-label Learning

  •  

14 of 24

Experiments

15 of 24

Dataset

  • RetargetMe
  • Randomly divide the dataset into 10 groups, use 9 groups as training data and the other group as testing data

16 of 24

Dataset

  • Features:
    • Lines/edges
    • Faces/people
    • Texture
    • Foreground objects
    • Geometric structures
    • Symmetry
    • Outdoors
    • Indoors

17 of 24

Dataset

  • Image retargeting methods:
    • Seam carving (SC)
    • Non-homogeneous warping (WARP)
    • Scale-and-stretch (SNS)
    • Multi-operator (MULTIOP)
    • Shift-maps (SM)
    • Streaming video (SV)
    • Uniform scaling (SCL)
    • Manual cropping (CR)

18 of 24

Results

  • RetargetMe provides manual evaluation results of target image quality.
  • For each original image, if the number of votes of a target image is not less than 80% of the highest vote of all the target images generated from it, this paper will treat the corresponding image retargeting method as a suitable method for this original image.

19 of 24

Results

(a) Original image

(b) Seam carving (SC)

(c) Non-homogeneous warping (WARP)

(d) Scale-and-stretch (SNS)

(e) Multi-operator (MULTIOP)

(f) Shift-maps (SM)

(g) Streaming video (SV)

(h) Uniform scaling (SCL)

(i) Cropping (CR)

20 of 24

Comparison

  • Compare the proposed approach with automatic quality assessment based selection strategy
    • Bidirectional similarity (BDS)
    • Bidirectional warping (BDW)
    • Edge histogram (EH)
    • Color layout (CL)
    • SIFT-flow (SIFTflow)
    • Earth-mover’s distance (EMD)
  • Calculate precision, recall F1 measure and hit-rate of each method

​

21 of 24

Comparison

22 of 24

Discussion

23 of 24

Conclusion

24 of 24

Conclusion

  • In this paper, we propose an image retargeting method selection approach based on the characteristics of original image.
  • Select the suitable image retargeting methods for a given image based on several simple features of the original image.
  • The future work will focus on enlarging the dataset and re-label the ground truth of suitable retargeting method manually.