Gaussian Splatting in Head Avatar Generation

William Gazali

Contents

• Background
• Motivation
• Papers
• Gaussian Splatting
• Gaussian Head Avatar
• Conclusion
• QnA

Background

Novel View Synthesis

https://www.matthewtancik.com/nerf

https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/

Head Avatar Generation

Goal

• Reconstruct a human head with good geometry and texture

​

Application

• Reenactment
• Entertainment
• Forensic
• etc

​

https://www.liuyebin.com/havatar/

Gaussian Splatting? Why should we care?

Motivation

https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/

Gaussian Splatting

What makes it special?

• Faster
• Better results!

Methodology

Variables:

• Position (xyz)
• Covariance matrix (Σ)
• Opacity (alpha)
• Spherical Harmonics (color)

Methodology

We start from point clouds generated by Structure From Motion (SFM)

image taken from: https://www.linkedin.com/pulse/structure-from-motion-manish-joshi/

Methodology

Initialize 3D Gaussian in those point clouds

image taken from: https://youtu.be/T_kXY43VZnk

​

​

​

Projection from 3D to 2D

​

​

Optimize Σ through Rotation and Scale

​

Methodology

Goal

• Populate empty areas
• Reduce large gaussians into smaller one

​

Every 100 iter after warm up

Methodology

Goal: Render quickly

Process:

• Split image to 16x16 tiles
• Instantiate each splats with tiles it overlaps
• GPU Radix Sort
• Accumulate color and alpha for each pixel

image taken from: https://openaccess.thecvf.com/content/CVPR2021/papers/Lassner_Pulsar_Efficient_Sphere-Based_Neural_Rendering_CVPR_2021_paper.pdf

Methodology

Loss function

• Comparing L1 loss of image and gt image

Visualization

The image will slowly take shape

even fine details

image taken from: https://youtu.be/T_kXY43VZnk

​

Results

Integrate to avatar generation?

Gaussian Head Avatar

Problem

• Previous method are not able to model fine details
• Previous methods are not able to model complex and exaggerated facial expressions accurately
• Using Splatting as it is often fail

Methodology

Methodology

Preprocessing

• Background removing

https://github.com/PeterL1n/BackgroundMattingV2

• Fit BFM model to get 2D and 3D landmark and expression coefficient
• Estimate headpose

https://faces.dmi.unibas.ch/bfm/index.php?nav=1-1-0&id=details

https://github.com/1adrianb/face-alignment

​

Methodology

Problem

• Random init gaussian leads to converging failure
• Initializing using FLAME model fails to style model and shoulder

​

Solution

• Signed distance field (SDF)
• Deep Marching Tetrahedra (DMTet)
• Optimize coarse color and deformation MLP
• Use the model as gaussian initialization

https://www.youtube.com/watch?v=vhrvl9qOSKM

Methodology

Pipeline

• SDF MLP output SDF value, η (feature vector)
• Get mesh from features and SDF values
• Color and pose MLP makes the expressive Mesh
• Render using Differentiable rasterization from Gaussian splating

​

Output

• Image
• Mask
• Landmark points from SDF
• Landmark points from deformation

Methodology

Overall goal

Color diff

Mask diff

Landmark diff

Punishment for non zero

Limit SDF to be close to zero

Laplace to smooth

Methodology

Pipeline

• Want to create MLPs to optimize
• Position X
• Feature vector F
• Rotation Q
• Scale S
• Opacitity A

Methodology

Network Overview

• Use 2 MLP
• Expression MLP
• Pose MLP

​

Show how much the points affect the expression and headpose

• λexp (·)
• λpose (·)

​

Methodology

Position X’

​

​

​

​

Color C’

​

​

​

​

Rotation, Scale, and Opacity (Q’, S’, A’)

​

​

​

​

​

​

​

Methodology

Pipeline

• After rendering input to Super Resolution network
• Optimize jointly

Results

Results

Conclusion

• Gaussian Splatting have been a prominent method in 3D reconstruction

​

• Gaussian Splatting is applicable even in Head Avatar Generation

​

• Achieves SOTA results

Q n A

Thank You