1 of 29

METRO: End-to-End Human Pose

and Mesh Reconstruction with Transformers

CVPR 2021

July 19, 2023

Presenter, Jeongwan On

2 of 29

Abstract

  • reconstruct 3D human pose and mesh vertices from a single image.

​

  • Our method uses a transformer encoder to jointly model vertex-vertex and vertex-joint interactions, and outputs 3D joint coordinates and mesh vertices simultaneously.

​

  • METRO does not rely on any parametric mesh models like SMPL, thus it can be easily extended to other objects such as hands.

​

  • possible to learn non-local relationships among mesh vertices and joints.

​

  • With the proposed masked vertex modeling, our method is more robust and effective in handling challenging situations like partial occlusions

3 of 29

Introduction

Use parametric model

Not use parametric model

4 of 29

Introduction

  • Use parametric model
    • 장점
      • Model에 Encode되어 있는 강력한 prior -> robust
    • 단점
      • parameter space가 제한적임

​

​

  • Not use parametric model
    • 장점
      • parametric model의 단점 극복
    • 단점
      • global한 관계 학습 x

5 of 29

Contribution

  • Transformer encoder를 3D human pose and mesh 분야에 최초로 적용한 연구

​

​

  • MVM (Masked Vertex Modeling) 기법을 사용, occlusion등의 상황에 모델이 좀 더 robust해짐

​

​

  • 3DPW, Human3.6M dataset과 같은 human pose뿐만 아니라 FreiHAND와 같은 Hand 분야에서도 SOTA를 달성함 -> 다른 도메인으로의 확장성이 높음

6 of 29

Method - Architecture

7 of 29

Method - Backbone (CNN)

8 of 29

Method - Backbone (CNN)

Multi - Layer

Transformer Encoder

3D coordinates

9 of 29

Method - Positional Embedding

10 of 29

Method - Positional Embedding

RNN based model

Transformer

A love B != B love A

A love B == B love A

11 of 29

Method - Positional Embedding

SMPL

pose dummy

(1*72)

shape dummy

(1*10)

template joint

( J )

template vertex

( V )

feature vector

( X )

12 of 29

Method - Masked Vertex Modeling (MVM)

13 of 29

Method - Masked Vertex Modeling (MVM)

Masked Language Model

나는

사과를

[mask]

나는

사과를

먹는다

Masked Vertex Model

q_1

q_2

[mask]

3D coordinates

14 of 29

Method - Masked Vertex Modeling (MVM)

15 of 29

Method - Multi-Layer Transformer Encoder

16 of 29

Method - Multi-Layer Transformer Encoder

Progressive

Dimensionality Reduction

17 of 29

Method

18 of 29

Method - Training loss

Dataset

Model output

19 of 29

Method - Training loss

20 of 29

Method - Training loss

21 of 29

Method - Training loss

22 of 29

Method - Training loss

23 of 29

Method - Training loss

24 of 29

Ablation study - MVM

25 of 29

Ablation study - backbone

26 of 29

Experiments

27 of 29

Experiments

28 of 29

Experiments

29 of 29

Thank you!