Interactive Graphics for a

Universally Accessible Metaverse

Ruofei Du

Senior Research Scientist

Google Labs, San Francisco

www.ruofeidu.com

me@duruofei.com

Self Intro

www.duruofei.com

Self Intro

Ruofei Du (杜若飞)

Self Intro

Ruofei Du (杜若飞)

Self Intro

Ruofei Du (杜若飞)

Human-Computer Interaction

Geollery �CHI '19, Web3D '19, VR '19

Social Street View Web3D '16

Best Paper Award

VideoFields

Web3D '16

SketchyScene

TOG (SIGGRAPH Asia) '19, ECCV '18

Montage4D

I3D '18

JCGT '19

DepthLab UIST '20

13K Installs

Kernel Foveated Rendering�I3D '18, VR '20, TVCG '20

CollaboVR ISMAR '20

LogRectilinear

IEEE VR '21 (TVCG)

TVCG Honorable Mention

GazeChat

UIST '21

Computer Graphics

MDIF�ICCV' 21

HumanGPS�CVPR' 21

HandSight

ECCVW '14

TACCESS '15

Ad hoc UI

CHIEA '22

ProtoSound

CHI ‘22

PRIF�ECCV' 22

Computer

Vision

Self Intro

Ruofei Du (杜若飞)

Interaction and Communication

Geollery �CHI '19, Web3D '19, VR '19

Social Street View Web3D '16

Best Paper Award

VideoFields

Web3D '16

SketchyScene

TOG (SIGGRAPH Asia) '19, ECCV '18

Montage4D

I3D '18

JCGT '19

DepthLab UIST '20

13K Installs

Kernel Foveated Rendering�I3D '18, VR '20, TVCG '20

CollaboVR ISMAR '20

LogRectilinear

IEEE VR '21 (TVCG)

TVCG Honorable Mention

GazeChat

UIST '21

Digital World

Digital Human

HumanGPS�CVPR' 21

HandSight

ECCVW '14

TACCESS '15

ProtoSound

CHI ‘22

Ad hoc UI

CHIEA '22

OmniSyn

IEEE VR '22

SlurpAR DIS '22

Interactive Graphics for a

Universally Accessible Metaverse

Ruofei Du

Senior Research Scientist

Google Labs, San Francisco

www.ruofeidu.com

me@duruofei.com

Metaverse

Neal Stephenson, 1992.

Metaverse

Metaverse

Future of Internet?

Internet of Things?

Virtual Reality?

Augmented Reality?

Decentralization?

Blockchain + NFT?

Mirrored World?

Digital Twin?

VR OS?

Web 3.0?

The Future of Internet

Internet of Things

Virtual Reality

Augmented Reality

Decentralization

Blockchain

NFT

Mirrored World

Metaverse

Digital Twin

VR OS

Web 3.0

Extended Reality (XR)

Accessibility

Avatars

Co-presence

Economics

Gaming

Wearable

AI

Privacy

Security

Vision

Neural

Metaverse

Metaverse envisioned a persistent digital world where people are fully connected as virtual representations,

​

As a teenager, my dream was to live in a metaverse...

​

However, today I wish metaverse is only a tool to make information more useful and accessible and help people to live a better physical life.

Interactive Graphics for a Universally Accessible Metaverse

​

​

Chapter One · Mirrored World & Real-time Rendering

​

Chapter Two · Computational Interaction: Algorithm & Systems

​

Chapter Three · Digital Human & Augmented Communication

​

​

​

Interactive Graphics for a Universally Accessible Metaverse

Chapter One · Mirrored World & Real-time Rendering

Ruofei Du

Senior Research Scientist

Google Labs, San Francisco

www.ruofeidu.com

me@duruofei.com

Interactive Graphics for a Universally Accessible Metaverse

Chapter One · Mirrored World & Real-time Rendering

Geollery �CHI '19, Web3D '19, VRW '19

Social Street View Web3D '16

Best Paper Award

Kernel Foveated Rendering�I3D '18, VR '20, TVCG '20

LogRectilinear, OmniSyn

IEEE VR '21 (TVCG), VRW ‘22

TVCG Honorable Mention

Project Geollery.com & Social Street View: Reconstructing a Live Mirrored World With Geotagged Social Media

Ruofei Du^†, David Li^†, and Amitabh Varshney

{ruofei, dli7319, varshney}@umiacs.umd.edu | www.Geollery.com | ACM CHI 2019 & Web3D 2016 Best Paper Award & 2019

UMIACS

THE AUGMENTARIUM

VIRTUAL AND AUGMENTED REALITY LAB

AT THE UNIVERSITY OF MARYLAND

​

COMPUTER SCIENCE

UNIVERSITY OF MARYLAND, COLLEGE PARK

​

Introduction

Social Media

20

image courtesy: plannedparenthood.org

Introduction

Social Media + Topics

21

image courtesy: huffingtonpost.com

Motivation

Social Media + XR

22

Motivation

Social Media + XR

23

image courtesy:

instagram.com,

facebook.com,

twitter.com

Motivation

2D layout

24

image courtesy:

pinterest.com

Motivation

Immersive Mixed Reality?

25

image courtesy:

viralized.com

Motivation

Pros and cons of the classic

26

Motivation

Pros and cons of the classic

27

Related Work

Social Street View, Du and Varshney

Web3D 2016 Best Paper Award

28

Technical Challenges?

Related Work

Social Street View, Du and Varshney

Web3D 2016 Best Paper Award

30

Related Work

Social Street View, Du and Varshney

Web3D 2016 Best Paper Award

31

Related Work

3D Visual Popularity

Bulbul and Dahyot, 2017

32

Related Work

Virtual Oulu, Kukka et al.

CSCW 2017

33

Related Work

Immersive Trip Reports

Brejcha et al. UIST 2018

34

Related Work

High Fidelity, Inc.

35

Related Work

Facebook Spaces, 2017

36

What's Next?

Research Question 1/3

37

What may a social media platform look like in mixed reality?

What's Next?

Research Question 2/3

38

What if we could allow social media sharing in a live mirrored world?

What's Next?

Research Question 3/3

39

What use cases can we benefit from social media platform in XR?

Geollery.com

A Mixed-Reality Social Media Platform

40

Geollery.com

A Mixed-Reality Social Media Platform

41

42

1

Conception, architecting & implementation

Geollery

A mixed reality system that can depict geotagged social media and online avatars with 3D textured buildings.

43

2

Extending the design space of

3D Social Media Platform

Progressive streaming, aggregation approaches, virtual representation of social media, co-presence with virtual avatars, and collaboration modes.

44

3

Conducting a user study of

Geollery vs. Social Street View

by discussing their benefits, limitations, and potential impacts to future 3D social media platforms.

System Overview

Geollery Workflow

45

System Overview

Geollery Workflow

46

Geollery.com

v2: a major leap

47

System Overview

Geollery Workflow

48

System Overview

2D Map Data

49

System Overview

2D Map Data

50

System Overview

+Avatar +Trees +Clouds

51

System Overview

+Avatar +Trees +Clouds +Night

52

System Overview

Street View Panoramas

53

System Overview

Street View Panoramas

54

System Overview

Street View Panoramas

55

System Overview

Geollery Workflow

56

All data we used is publicly and widely available on the Internet.

Rendering Pipeline

Close-view Rendering

57

Rendering Pipeline

Initial spherical geometries

58

Rendering Pipeline

Depth correction

59

Rendering Pipeline

Intersection removal

60

Rendering Pipeline

Texturing individual geometry

61

Rendering Pipeline

Texturing with alpha blending

62

Rendering Pipeline

Rendering result in the fine detail

63

Rendering Pipeline

Rendering result in the fine detail

64

Rendering Pipeline

Rendering result in the fine detail

65

User Study

Social Street View vs. Geollery

66

User Study

Quantitative Evaluation

67

User Study

Quantitative Evaluation

68

69

I would like to use it for the food in different restaurants. I am always hesitating of different restaurants. It will be very easy to see all restaurants with street views. In Yelp, I can only see one restaurant at a time.

P6 / F

70

[I will use it for] exploring new places. If I am going on vacation somewhere, I could immerse myself into the location. If there are avatars around that area, I could ask questions.

P1 / M

71

I think it (Geollery) will be useful for families. I just taught my grandpa how to use Facetime last week and it would great if I could teleport to their house and meet with them, then we could chat and share photos with our avatars.

P2 / F

72

if there is a way to unify the interaction between them, there will be more realistic buildings [and] you could have more roof structures. Terrains will be interesting to add on.

P18 / M

Rendering Pipeline

Experimental Features

73

Landing Impact

Demos at ACM CHI 2019

74

Landing Impact

Demos at ACM CHI 2019

75

Landing Impact

Demos at ACM CHI 2019

76

Instant Panoramic Texture Mapping with Semantic Object Matching for Large-Scale Urban Scene Reproduction

​

TVCG 2021, Jinwoo Park, Ik-beom Jeon, Student Members, Sung-eui Yoon, and Woontack Woo

Instant Panoramic Texture Mapping with Semantic Object Matching for Large-Scale Urban Scene Reproduction

​

TVCG 2021, Jinwoo Park, Ik-beom Jeon, Student Members, Sung-eui Yoon, and Woontack Woo

A more applicable method for constructing walk-through experiences in urban streets was employed by Geollery [16], which adopted an efficient transformation of a dense spherical mesh to construct a local proxy geometry based on the depth maps from Google Street View

Freeman et al. ACM PHCI 2022

He et al. ISMAR 2020

Park et al. Virtual Reality 2022

Yeom et al. IEEE VR 2021

What's Next?

OmniSyn: Intermediate View Synthesis Between Wide-Baseline Panoramas

​

David Li, Yinda Zhang, Christian Häne, Danhang Tang, Amitabh Varshney, and Ruofei Du, VR 2022

OmniSyn: Intermediate View Synthesis Between Wide-Baseline Panoramas

​

David Li, Yinda Zhang, Christian Häne, Danhang Tang, Amitabh Varshney, and Ruofei Du, VR 2022

How can we further accelerate the real-time rendering procedure?

85

Kernel Foveated Rendering

Xiaoxu Meng, Ruofei Du, Matthias Zwicker and Amitabh Varshney

Augmentarium | UMIACS

University of Maryland, College Park

ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games 2018

86

Original Frame

Buffer

Screen

Sample Map

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

87

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

Kernel Log-polar Mapping

Eye-dominance-guided�Foveated Rendering

Xiaoxu Meng, Ruofei Du, and Amitabh Varshney

IEEE Transactions on Visualization and Computer Graphics (TVCG)

fovea

fovea

more foveation for the non-dominant eye

​

3D-Kernel Foveated Rendering for Light Fields

Xiaoxu Meng, Ruofei Du, Joseph JaJa, and Amitabh Varshney

IEEE Transactions on Visualization and Computer Graphics (TVCG), 2020

UMIACS

A Log-Rectilinear Transformation for Foveated 360-Degree Video Streaming

David Li^†, Ruofei Du^‡, Adharsh Babu^†, Camelia Brumar^†, Amitabh Varshney^†

^† University of Maryland, College Park ^‡ Google Research

UMIACS

TVCG Honorable Mentions Award

Sandwiched Image Compression

Wrapping Neural Networks Around a Standard Codec

Increasing the Resolution and Dynamic Range of Standard Codecs

Onur Guleryuz, Philip Chou, Hugues Hoppe, Danhang Tang, �Ruofei Du, Philip Davidson, and Sean Fanello

2021 IEEE International Conference on Image Processing (ICIP)�2022 Picture Coding Symposium (PCS)

Multiresolution Deep Implicit Functions for 3D Shape Representation

Zhang Chen, Yinda Zhang, Kyle Genova, Thomas Funkhouse, Sean Fanello, Sofien Bouaziz, Christian Häne, Ruofei Du, Cem Keskin, and Danhang Tang

2021 IEEE/CVF International Conference on Computer Vision (ICCV)

Interactive Graphics for a Universally Accessible Metaverse

Chapter Two · Computational Interaction: Algorithm & Systems

Ruofei Du

Senior Research Scientist

Google Labs, San Francisco

www.ruofeidu.com

me@duruofei.com

Interactive Graphics for a Universally Accessible Metaverse

Chapter Two · Computational Interaction: Algorithm & Systems

Ad hoc UI

CHI EA ‘21

DepthLab

UIST '20

13K Installs & deployed in Tiktok, Snap, Teamviewer etc.

​

SlurpAR�DIS ‘22

RetroSphere

IMWUT ‘22

DepthLab: Real-time 3D Interaction with Depth Maps for Mobile Augmented Reality

Ruofei Du, Eric Turner, Maksym Dzitsiuk, Luca Prasso, Ivo Duarte,

Jason Dourgarian, Joao Afonso, Jose Pascoal, Josh Gladstone, Nuno Cruces,

Shahram Izadi, Adarsh Kowdle, Konstantine Tsotsos, David Kim

​

Google | ACM UIST 2020

Introduction

Mobile Augmented Reality

Introduction

Google's ARCore

Introduction

Google's ARCore

Introduction

Mobile Augmented Reality

Introduction

Motivation

Is the current generation of object placement sufficient for realistic AR experiences?

Introduction

Depth Lab

Not always!

​

Introduction

Depth Lab

Virtual content looks like it’s “pasted on the screen” rather than “in the world”!

​

Introduction

Motivation

Introduction

Depth Lab

How can we bring these advanced

features to mobile AR experiences WITHOUT relying on dedicated sensors or the need for computationally expensive surface reconstruction?

​

Introduction

Depth Map

Introduction

Depth Lab

Introduction

Depth Lab

Is there more to realism than occlusion?

Introduction

Depth Lab

Surface interaction?

Introduction

Depth Lab

Realistic Physics?

Introduction

Depth Lab

Path Planning?

Introduction

Depth Lab

Related Work

Valentin et al.

Depth Maps

Depth �from Motion

Depth From a Single Camera

Best Practices

Depth From a Single Camera

Use depth-certified ARCore devices

Minimal movement in the scene

Encourage users to move the device

Depth from 0 to 8 meters

Best accuracy 0.5 to 5 meters

​

Enhancing Depth

Optimized to give you the best depth

Depth from Motion is fused with state-of-the-art Machine Learning

​

Depth leverages specialized hardware like a Time-of-Flight sensor when available

Introduction

Depth Lab

Introduction

Depth Lab

Introduction

Depth Generation

Introduction

Depth Lab

Related Work

Valentin et al.

Introduction

Depth Lab

Introduction

Depth Lab

Up to 8 meters, with

the best within 0.5m to 5m

Motivation

Gap from raw depth to applications

Introduction

Depth Lab

ARCore

​

Depth API

DepthLab

Mobile AR developers

Design Process

3 Brainstorming Sessions

3 brainstorming sessions

18 participants

39 aggregated ideas

Design Process

3 Brainstorming Sessions

System

Architecture overview

Data Structure

Depth Array

2D array (160x120 and above) of 16-bit integers

Data Structure

Depth Mesh

Data Structure

Depth Texture

System

Architecture

Localized Depth

Coordinate System Conversion

​

Localized Depth

Normal Estimation

​

Localized Depth

Normal Estimation

​

Localized Depth

Normal Estimation

​

Localized Depth

Avatar Path Planning

​

Localized Depth

Rain and Snow

​

Surface Depth

Use Cases

Surface Depth

Physics collider

Physics with depth mesh.

Surface Depth

Texture decals

Texture decals with depth mesh.

Surface Depth

3D Photo

Projection mapping with depth mesh.

Dense Depth

Depth Texture - Antialiasing

Dense Depth

Real-time relighting

θ

N

L

Dense Depth

Why normal map does not work?

Dense Depth

Real-time relighting

Dense Depth

Real-time relighting

Dense Depth

Real-time relighting

go/realtime-relighting, go/relit

Dense Depth

Wide-aperture effect

Dense Depth

Occlusion-based rendering

Experiments

DepthLab minimum viable application

Experiments

General Profiling of MVP

Experiments

Relighting

Experiments

Aperture effects

Impact

Deployment with partners

Impact

Deployment with partners

Impact

Deployment with partners

AR Realism

In TikTok

AR Realism

Built into Lens Studio for Snapchat Lenses

Kevaid

Saving Chelon

Quixotical�The Seed: World of Anthrotopia

Snap�Dancing Hotdog

Camera Image

3D Point Cloud

Provides a more detailed representation of the geometry of the objects in the scene.

Raw Depth API

New depth capabilities

Camera Image

Raw Depth Image

Depth Image

Confidence Image

New depth capabilities

Raw Depth API

Provides a more detailed representation of the geometry of the objects in the scene.

Try it yourself!

TeamViewer�LifeAR App

ARCore�Depth Lab App

Depth �Hit Test

New depth capabilities

ARCore�Depth Lab App

Depth API�Codelab

Raw Depth API�Codelab

Limitations

Design space of dynamic depth

Dynamic Depth? HoloDesk, HyperDepth, Digits, Holoportation for mobile AR?

Envision

Design space of dynamic depth

GitHub

Please feel free to fork!

Play Store

Try it yourself!

Impact

Significant Media Coverage

Impact

Significant Media Coverage

More Links

Significant Media Coverage

WebXR + ARCore Depth: https://storage.googleapis.com/chromium-webxr-test/r991081/proposals/index.html

​

Hugging Face Depth: https://huggingface.co/spaces/Detomo/Depth-Estimation

​

ARCore Depth Lab Play Store App: https://play.google.com/store/apps/details?id=com.google.ar.unity.arcore_depth_lab

​

DepthLab: Real-time 3D Interaction with Depth Maps for Mobile Augmented Reality

Ruofei Du, Eric Turner, Maksym Dzitsiuk, Luca Prasso, Ivo Duarte,

Jason Dourgarian, Joao Afonso, Jose Pascoal, Josh Gladstone, Nuno Cruces,

Shahram Izadi, Adarsh Kowdle, Konstantine Tsotsos, David Kim

​

Google | ACM UIST 2020

Thank you!

DepthLab | UIST 2020

Demo

DepthLab | UIST 2020

DepthLab: Real-time 3D Interaction with Depth Maps for Mobile Augmented Reality

Ruofei Du, Eric Turner, Maksym Dzitsiuk, Luca Prasso, Ivo Duarte,

Jason Dourgarian, Joao Afonso, Jose Pascoal, Josh Gladstone, Nuno Cruces,

Shahram Izadi, Adarsh Kowdle, Konstantine Tsotsos, David Kim

​

Google | ACM UIST 2020

Ad hoc UI: On-the-fly Transformation of Everyday Objects

into Tangible 6DOF Interfaces for AR

Ruofei Du, Alex Olwal, Mathieu Le Goc, Shengzhi Wu, Danhang Tang,

Yinda Zhang, Jun Zhang, David Joseph Tan, Federico Tombari, David Kim

​

Google | CHI 2022 Interactivity / In submission to UIST 2022

Applications

“Slurp” Revisited: Using Software Reconstruction to Reflect on Spatial Interactivity and Locative Media

Shengzhi Wu, Daragh Byrne, Ruofei Du, and Molly Steenson

ACM DIS 2022

RetroSphere: Self-Contained Passive 3D Controller Tracking for Augmented Reality

Ananta Narayanan Balaji, Clayton Kimber, David Li, Shengzhi Wu, Ruofei Du, David Kim

ACM Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT) 2022

Ananta Narayanan Balaji, Clayton Kimber, David Li, Shengzhi Wu, Ruofei Du, David Kim

ACM Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT) 2022

Interactive Graphics for a Universally Accessible Metaverse

Chapter Three · Digital Human & Augmented Communication

Ruofei Du

Senior Research Scientist

Google Labs, San Francisco

www.ruofeidu.com

me@duruofei.com

Interactive Graphics for a Universally Accessible Metaverse

Chapter Three · Digital Human & Augmented Communication

HumanGPS �CVPR ‘21

Montage4D

I3D '18

JCGT '19

GazeChat & CollaboVR�UIST ‘21 & ISMAR ‘20

Neural Head Avatar

CVPR ‘22 (in submission)

196

ACM Trans. Graph., Vol. 40, No. 4, Article 1. SIGGRAPH 2021

ACM Trans. Graph., Vol. 40, No. 4, Article 1. SIGGRAPH 2021

GazeChat

Enhancing Virtual Conferences With

Gaze-Aware 3D Photos

Zhenyi He^†, Keru Wang^†, Brandon Yushan Feng^‡, Ruofei Du^⸸, Ken Perlin^†

​

^† New York University�^‡ University of Maryland, College Park �^⸸ Google

Introduction

VR headset & video streaming

219

Related Work

Gaze-2 (2003)

220

Related Work

MultiView (2005)

221

Related Work

MMSpace (2016)

222

Our Work

GazeChat (UIST 2021)

223

Gaze Awareness

Definition

224

Gaze awareness, defined here as knowing what someone is looking at.

Gaze Awareness

Definition

225

gaze correction

gaze redirection

raw input image

GazeChat

Gaze Correction

Definition

226

Gaze Rediction

Definition

227

eye contact

who is looking at whom

Pipeline

System

228

Eye Tracking

WebGazer..js

229

Neural Rendering

Eye movement

230

Neural Rendering

Eye movement

231

3D Photo Rendering

3D photos

232

3D Photo Rendering

3D photos

233

Layouts

UI

234

Networking

WebRTC

235

Zhenyi He^* Ruofei Du^† Ken Perlin^*

​

^*Future Reality Lab, New York University ^†Google LLC

CollaboVR: A Reconfigurable Framework for

Creative Collaboration in Virtual Reality

​

​

ProtoSound: A Personalized and Scalable Sound Recognition System for Deaf and Hard-of-Hearing Users

​

​

ACM CHI 2012 · Dhruv Jain, Khoa Nguyen, Steven Goodman, Rachel Grossman-Kahn, Hung Ngo, Aditya Kusupati, Ruofei Du, Alex Olwal, Leah Findlater, and Jon Froehlich

SketchyScene: Richly-Annotated Scene Sketches

Changqing Zou, Qian Yu, Ruofei Du, Haoran Mo, Yi-Zhe Song, Tao Xiang, Chengying Gao, Baoquan Chen, and Hao Zhang (ECCV 2022)

Language-based Colorization of Scene Sketches

Changqing Zou, Haoran Mo, Chengying Gao, Ruofei Du, and Hongbo Fu (ACM Transaction on Graphics, SIGGRAPH Asia 2019)

Future Directions

The Ultimate XR Platform

244

Wearable Subtitles

Augmenting Spoken Communication with

Lightweight Eyewear for All-day Captioning

Future Directions

The Ultimate XR Platform

246

Future Directions

Fuses Past Events

247

Future Directions

With the present

248

Future Directions

And look into the future

249

Future Directions

Change the way we communicate in 3D and consume the information

250

Future Directions

Consume the information throughout the world

251

Interactive Graphics for a

Universally Accessible Metaverse

Ruofei Du

Senior Research Scientist

Google Labs, San Francisco

www.ruofeidu.com

me@duruofei.com

Interactive Graphics for a

Universally Accessible Metaverse

Ruofei Du

Senior Research Scientist

Google Labs, San Francisco

www.ruofeidu.com

me@duruofei.com

255

Kernel Foveated Rendering

Xiaoxu Meng, Ruofei Du, Matthias Zwicker and Amitabh Varshney

Augmentarium | UMIACS

University of Maryland, College Park

ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games 2018

256

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

257

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

258

* Data from Siggraph Asia 2016, Prediction by Michael Abrash, October 2016

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

259

• Virtual reality is a challenging workload

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

260

• Virtual reality is a challenging workload

​

• Most VR pixels are peripheral

​

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

fovea:

the center of the retina

corresponds to the center of the vision field

261

• Virtual reality is a challenging workload

​

• Most VR pixels are peripheral

​

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

foveal region:

the human eye detects significant detail

peripheral region:

the human eye detects little high fidelity detail

262

• Virtual reality is a challenging workload

​

• Most VR pixels are peripheral

​

foveal

region

foveal region

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

foveal region:

the human eye detects significant detail

peripheral region:

the human eye detects little high fidelity detail

263

• Virtual reality is a challenging workload

​

• Most VR pixels are peripheral

​

96 %

27 %

Percentage of the foveal pixels

4 %

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

* Data from Siggraph 2017, by Anjul Patney, August 2017

264

265

Foveated Rendering

266

• Virtual reality is a challenging workload

​

• Most VR pixels are peripheral

​

• Eye tracking technology available

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

267

Related Work

268

Full Resolution

Multi-Pass Foveated Rendering [Guenter et al. 2012]

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

269

Coarse Pixel Shading (CPS) [Vaidyanathan et al. 2014]

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

270

CPS with TAA & Contrast Preservation [Patney et al. 2016]

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

271

Can we change the resolution gradually?

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

272

Perceptual Foveated Rendering [Stengel et al. 2016]

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

273

Is there a foveated rendering approach

without

the expensive pixel interpolation?

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

274

Log-polar mapping [Araujo and Dias 1996]

Log-polar Mapping

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

275

Log-polar mapping [Araujo and Dias 1996]

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

Log-polar Mapping

276

Log-polar mapping [Araujo and Dias 1996]

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

Log-polar Mapping

277

Log-polar mapping [Araujo and Dias 1996]

Log-polar Mapping

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

278

Log-polar mapping [Araujo and Dias 1996]

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

Log-polar Mapping

279

Log-polar mapping [Araujo and Dias 1996]

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

Log-polar Mapping

280

Log-polar mapping [Araujo and Dias 1996]

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

Log-polar Mapping

281

Log-polar Mapping for 2D Image [Antonelli et al. 2015]

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

282

Log-polar Mapping for 2D Image

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

283

Our Approach

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

284

Kernel Log-polar Mapping

range: [0,1]

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

285

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

Log-polar Mapping

286

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

Kernel Log-polar Mapping

Kernel Foveated Rendering

287

288

Kernel log-polar Mapping

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

289

Kernel log-polar Mapping

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

290

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

291

Original Frame

Buffer

Screen

Sample Map

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

292

Original Frame

Buffer

Screen

Sample Map

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

293

Original Frame

Buffer

Screen

Sample Map

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

294

Fovea

Fovea

Fovea

295

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

296

Original Frame

Buffer

Screen

Sample Map

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

297

Original Frame

Buffer

Screen

Sample Map

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

298

Original Frame

Buffer

Screen

Sample Map

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

299

Fovea

Fovea

Fovea

Eye-dominance-guided�Foveated Rendering

Xiaoxu Meng, Ruofei Du, and Amitabh Varshney

IEEE Transactions on Visualization and Computer Graphics (TVCG)

304

305

Ocular Dominance: the tendency to prefer scene perception from one eye over the other.

Advantage of the Dominant Eye Over the Non-dominant Eye

• better color-vision discrimination ability [Koctekin 2013]
• shorter reaction time on visually triggered manual action [Chaumillon 2014]
• better visual acuity, contrast sensitivity [Shneor 2006]

306

307

Foveated Rendering

• VR requires enormous rendering budget
• Most pixels are outside the fovea

​

308

Foveated Rendering

• VR requires enormous rendering budget
• Most pixels are outside the fovea

​

309

Percentage of the foveal pixels

* Data from Siggraph 2017, by Anjul Patney, August 2017

fovea

fovea

Can we do better?

fovea

fovea

non-dominant eye

​

fovea

fovea

more foveation for the non-dominant eye

​

A Log-Rectilinear Transformation for Foveated 360-Degree Video Streaming

David Li^†, Ruofei Du^‡, Adharsh Babu^†, Camelia Brumar^†, Amitabh Varshney^†

^† University of Maryland, College Park ^‡ Google

UMIACS

TVCG Honorable Mentions Award

Introduction

VR headset & video streaming

316

• 360 Cameras and VR headsets are increasing in resolution.
• Video streaming is quickly increasing in popularity.

Introduction

VR + eye tracking

317

• Commercial VR headsets are getting eye-tracking capabilities.

HTC Vive Eye​

Varjo VR-3

Fove

Introduction

360 videos

318

• 360 cameras capture the scene in every direction with a full 360 degree spherical field of regard.
• These videos are typically stored in the equirectangular projection parameterized by spherical coordinates (𝜃, 𝜑).

360° Field of Regard

Scene

Captured 360 Video

Introduction

360 videos

319

• When viewed in a VR headset, 360° videos cover the entire field-of-view for more immersive experiences.
• However, transmitting the full field-of-regard either has worse perceived quality or requires far more bandwidth than for conventional videos.

Captured 360 Video

Projection to Field of View

Introduction

360 videos

320

• Existing work in 360° streaming focuses on viewport dependent streaming by using tiling to transmit only visible regions based on the user’s head rotation.

Tiling Illustration

Image from (Liu et al. with Prof. Bo, 2017)

Introduction

Foveated rendering

321

• Foveated rendering renders the fovea region of the viewport at a high-resolution and the peripheral region at a lower resolution.
• Kernel Foveated Rendering (Meng et al., PACMCGIT 2018) uses a log-polar transformation to render foveated images in real-time.

Image Credit: Tobii

Log-polar Transformation,

Image from (Meng et al., 2018)

Introduction

Log-Polar Foveated Streaming

322

• Applying log-polar subsampling to videos results in flickering and aliasing artifacts in the foveated video.

323

1

Research Question

Can foveation techniques from rendering be used to optimize 360 video streaming?

324

2

Research Question

How can we reduce foveation artifacts by leveraging the full original video frame?

Log-Polar Foveated Streaming

• Artifacts are caused by subsampling of the original video frame.

Original Frame

Subsampled Pixel

Log-Polar Foveated Streaming

• Artifacts are caused by subsampling of the original video frame.

Original Frame

Subsampled Pixel

Log-Polar Foveated Streaming

• Subsampled pixels should represent an average over an entire region of the original video frame.
• Computationally, this would take O(region size) time to compute for each sample.

Summed-Area Tables

• One way to compute averages quickly is using summed-area tables, also known as integral images.
• Sampling a summed area table only takes O(1) time.

Log-Rectilinear Transformation

• Apply exponential drop off along x-axis and y-axis independently.
• Rectangular regions allow the use of summed area tables for subsampling.
• A one-to-one mapping near the focus region preserves the resolution of the original frame.

Foveated Streaming

Video Streaming Server

Client

socket

socket

FFmpeg

OpenCL

OpenCL

FFmpeg

FFmpeg

OpenCL

Video Streaming Request

socket

Qualitative Results

• Shown with gaze at the center of the viewport

Quantitative Results

We perform quantitative evaluations comparing the log-rectilinear transformation and the log-polar transformation in 360° video streaming.

• Performance overhead of summed-area tables.
• Full-frame quality.
• Bandwidth usage.

​

Quantitative Results

• Pairing the log-rectilinear transformation with summed area table filtering yields lower flickering while also reducing bandwidth usage and returning high weighted-to-spherical signal to noise ratio (WS-PSNR) results.

Quantitative Results

• Pairing the log-rectilinear transformation with summed area table filtering yields lower flickering while also reducing bandwidth usage and returning high weighted-to-spherical signal to noise ratio (WS-PSNR) results.

Conclusion

• We present a log-rectilinear transformation which utilizes foveation, summed-area tables, and standard video codecs for foveated 360° video streaming.

Foveation

Summed-Area Tables

Standard Video Codecs

Foveated 360° Video Streaming

Zhenyi He^* Ruofei Du^† Ken Perlin^*

​

^*Future Reality Lab, New York University ^†Google LLC

CollaboVR: A Reconfigurable Framework for

Creative Collaboration in Virtual Reality

​

​

The best layout and interaction mode?

Research Questions:

• Design: What if we could bring sketching to real-time collaboration in VR?
• Design + Evaluation: If we can convert raw sketches into interactive animations, will it improve the performance of remote collaboration?
• Evaluation: Are there best user arrangements or input modes for different use cases, or is it more a question of personal preferences

CollaboVR: A Reconfigurable Framework for

Creative Collaboration in Virtual Reality

​

​

User Arrangements

(1) side-by-side

(2) face-to-face

(3) hybrid

Input Modes

(1) direct

(2) projection

​

User Arrangements

(1) side-by-side

​

​

(b)

user 1

Interactive boards

tracking range of user 1

user 1

user 2

User Arrangements

(1) side-by-side

(2) face-to-face

​

User Arrangements

(1) side-by-side

(2) face-to-face

​

User Arrangements

(1) side-by-side

(2) face-to-face

(3) hybrid

(d)

(c)

(b)

user 2

teacher

user 3

user 4

Input Modes

(1) direct

(2) projection

​

Input Modes

(1) direct

(2) projection

​

Input Modes

(1) direct

(2) projection

​

C1: Integrated Layout

C2: Mirrored Layout

C3: Projective Layout

C1: Integrated Layout

C2: Mirrored Layout

C3: Projective Layout

C1: Integrated Layout

C2: Mirrored Layout

C3: Projective Layout

C1: Integrated Layout

C2: Mirrored Layout

C3: Projective Layout

Evaluation

Overview of subjective feedback on CollaboVR

Evaluation

Evaluation

Takeaways

1. Developing CollaboVR, a reconfigurable end-to-end collaboration system.
2. Designing custom configurations for real-time user arrangements and input modes.
3. Quantitative and qualitative evaluation of CollaboVR.
4. Open-sourcing our software at https://github.com/snowymo/CollaboVR.

more live demos...

Zhenyi He^* Ruofei Du^† Ken Perlin^*

​

^*Future Reality Lab, New York University ^†Google LLC

CollaboVR: A Reconfigurable Framework for

Creative Collaboration in Virtual Reality

​

​

Fusing Physical and Virtual Worlds into

An Interactive Metaverse

Ruofei Du

Senior Research Scientist

Google, San Francisco

www.ruofeidu.com

me@duruofei.com

Introduction

Depth Map

Introduction

Depth Map

Introduction

Depth Lab

Thank you!

www.duruofei.com

Introduction

Depth Lab

Occlusion is a critical component for AR realism!

Correct occlusion helps ground content in reality, and makes virtual objects feel as if they are actually in your space.

​

​

Introduction

Motivation

Depth Mesh

Generation

Localized Depth

Avatar Path Planning

Dense Depth

Depth Texture

Introduction

Depth Map

Taxonomy

Depth Usage

Introduction

Depth Map

Introduction

Depth Map

OmniSyn: Synthesizing 360 Videos with Wide-baseline Panoramas

David Li, Yinda Zhang, Christian Häne, Danhang Tang, Amitabh Varshney, Ruofei Du

382

Problem

• Interpolate between 360° street view panoramas.

383

≥ 5 meters

baseline

OmniSyn

360° Wide-baseline

View Synthesis

Related Works - Monocular Neural Image Based Rendering with Continuous View Control (ICCV 2019)

• Xu Chen, Jie Song, and Otmar Hilliges
• Predict target depth from source RGB & target pose.
• Do back-projection using the target depth.

384

Related Works - SynSin (CVPR 2020)

• Olivia Wiles, Georgia Gkioxari, Richard Szeliski, Justin Johnson
• End-to-end View Synthesis from a Single Image
• Trained with differentiable point cloud rendering and without depth supervision.

385

Research Goal

• What are the challenges associated with 360 street view synthesis?
• How can we modify the traditional view synthesis pipeline to leverage two 360 panoramas meters apart?

386

Method

• Tailor the traditional view synthesis pipeline to handle 360 panoramas.
• Depth estimation with spherical sweep cost volumes (Sunghoon, 2016).
• Mesh rendering for 360 panoramas.
• Inpainting with CoordConv (Liu, 2018) and Circular CNNs (Schubert, 2019).

387

CoordConv

Spherical Cost Volume

Pipeline

• Our pipeline directly operates on equirectangular panoramas:

388

Panorama 1

Panorama 0

Depth Predictor

Depth Predictor

Fusion

Network

Target Panorama

Stereo Depth with Cost Volume

• Stereo depth estimation uses cost volumes with perspective images.
• Cost volumes compute the image different at various offsets to identify the correct disparity.
• StereoNet compute cost volumes from deep features within a CNN.

389

-

=

-

=

…

Stereo 360 Depth with Cost Volume

• For 360, we can perform a similar sweep testing different depths:

390

Stereo 360 Depth with Cost Volume

• For 360, we can perform a similar sweep testing different depths:

391

-

=

-

=

…

Mesh Rendering

• For 360 images, using point clouds leads to sparse regions:

392

Mesh Rendering

• For 360 images, using point clouds leads to sparse regions:

393

Point Cloud Render

Mesh Render

3 m

2 m

1 m

4 m

OmniSyn (Mesh)

GT Visibility

OmniSyn (Point Cloud)

CoordConv and Circular CNN

• CoordConv - append an additional channel to allow conv layers to know where it is in the image and hopefully adapt to the ERP distortion.
• Distortion in the panorama means the images are not shift-invariant.
• Circular CNN - modify the CNN padding so that the kernel wraps around the left and right edges of the ERP image.

394

CoordConv

Circular CNN

Experiments

• Generated some synthetic street view images using CARLA
• Static scenes without pedestrians or cars.
• Allows us to obtain clean ground-truth depth information.
• Trained our pipeline with explicit depth and RGB supervision.
• Compare mesh and point-cloud rendering.
• Test generalization to real street view images.

395

Results

• We experiment on synthetic static street scenes from CARLA:

396

Generalization to Real Street View Panoramas

397

0 m GT

4.6 m GT

10.1 m GT

0 m GT

9.0 m GT

0 m GT

Synthesized

Limitations

398

Input 0

Input 1

Synthesized

Fusion network does not generalize well to unseen colors.

Depth prediction struggles with tall buildings.

Triangle removal may eliminate thin structures.

Conclusion

• We identified some challenges with performing view synthesis on sets of synthetic 360 street view panoramas.
• We augmented the view synthesis pipeline with components suitable for dual 360 panoramas.

399