Hard Negative Mining for

Multi-view Human Part Segmentation

16-622 Class Presentation

​

Presenters:�Ashwin Vaswani (apvaswan)

Minh Tran (mtanh)

Team

Students:

• Ashwin Vaswani (apvaswan)
• Minh Tran (minht)�

Advisors:

• Meta:
• He Wen
• Yuan Dong�
• CMU:
• Professor Deepak Pathak

2

Motivation

• Photorealistic avatars
• Virtual reality
• Teleconferencing

3

Goal - Active Learning using Hard Poses

Desired output:

Hard examples for labelling to improve segmentation model using AL

4

Small labelled dataset of multi-view images (~thousands)

Large unlabelled dataset (~millions)

Challenges

• Problem 1: Long tail
• Solution: Multi-view
• Problem 2: Annotation cost
• Dense prediction problem
• Solution: Active learning

5

Proposed Approach - Active Learning for Hard Poses

6

Hard - Use for next iteration of training with weighted penalty and continue from stage 1 iteratively till annotation budget

​

​

Other - Continue as unlabeled for next iteration

​

​

Easy - Good as pseudo labels, use for self-training in the next iteration.

​

​

CDGNet - Body Part Segmentation

​

​

​

• Part segmentation is a positionally dependent problem.
• CGD backbone learns horizontal and vertical class distribution labels.
• Features as positional guidance signal of human parts.

7

Liu et al. CDGNet - Class Distribution Guided Network for Human Parsing.

Progress I - Learning from Synthetic Humans

​

​

• Create synthetic data using the Surreal dataset.�
• Captured SMPL pose parameters of people performing actions in videos.�
• Given SMPL poses, we randomly sample shapes, randomly sample male/female and randomly add clothing textures to get the mesh. �
• For diversity, we only sample poses if they are different from the previously sampled pose.�
• Next, we render this mesh and using camera poses from the CMU panoptic studio and add the corresponding backgrounds from panoptic studio.

8

Varol et al. SURREAL - Learning from Synthetic Humans.

Progress I - Learning from Synthetic Humans

​

​

​

• We have per vertex segmentation label for SMPL meshes.
• Rendering mesh interpolates pixels.
• Rendering without interpolation to get segmentation maps.

9

Progress I - Learning from Synthetic Humans

​

​

​

• We have per vertex segmentation label for SMPL meshes.
• Rendering mesh interpolates pixels.
• Rendering without interpolation to get segmentation maps.

10

Progress I - Learning from Synthetic Humans

​

​

​

Dataset statistics:

• Images, Part segmentations, Depth maps.
• 10 views.
• ᯈ2.5M images.

11

Progress II - Initial Strategies & Results

​

​

• Baselines: Random, Entropy based
• Common failure: Self-occlusions, Joints crossings, Complex actions.
• Approach: Multi-view active learning (AL) method to identify hard examples.
• Goal: Improve performance, automate annotation process

12

Feng et al. Rethinking the Data Annotation Process for Multi-view 3D Pose Estimation with Active Learning and Self-Training

Finding Hard Poses

• Method 1: Multi-view Consistency
• Multiview data is important for self-occlusions

​

• Method 2: Clustering / Similarity
• Clustering to find most unusual poses
• Predefine hard poses

13

Tulsiani et al. Multi-view Consistency as Supervisory Signal

for Learning Shape and Pose Prediction

Method 1: Multi-view Consistency

​

​

• Some frames have more descriptive information than others due to viewing angles to resolve ambiguities
• Use uncertainty across cross projects as sampling strategy.

14

Method 2: Clustering / Similarity

​

​

​

​

• Clustering on weighted poses (joints).
• Torso and head less likely move.
• Limbs are more flexible.
• Uncommon poses are hard.
• Poses at tail of distribution.
• Generate similar poses.
• Predefine hard poses.
• Generate/Find similar poses.

15

Active Learning Specifics

• Start with 600 initial images.
• For Stages 1, 2,3,4,5…:
• Train segmentation model.
• Run inference on unlabelled dataset and sample hard examples.
• All hard (600 images)
• Mix of easy and hard
• Add these hard examples and corresponding labels to the training set and move to next stage.

16

X% sampled data data

Trained Model

Inference

Other

Hard

Data Pool

Data Pool

Use in next Stage for training

Feng et al. Rethinking the Data Annotation Process for Multi-view 3D Pose Estimation with Active Learning and Self-Training

Qualitative Results

17

Hard examples

Easy examples

Quantitative Results (mIOU scores)

18

Discussion

• Why is random better?
• Mostly (>95%) had easy examples.
• Solution:
• Stratifying test set (Semi-automated).

19

Next Steps

20

Conclusion

• Even after identifying hard examples, learning is not easy.
• However, there is promise in literature and we are working on improvements.
• Stratified test set would give us more signal.

21

Thank you!

Questions?

22