1 of 109

DepthLab: Real-time 3D Interaction with Depth Maps for Mobile Augmented Reality

Ruofei Du, Eric Turner, Maksym Dzitsiuk, Luca Prasso, Ivo Duarte,

Jason Dourgarian, Joao Afonso, Jose Pascoal, Josh Gladstone, Nuno Cruces,

Shahram Izadi, Adarsh Kowdle, Konstantine Tsotsos, David Kim

Google | ACM UIST 2020

2 of 109

Introduction

Mobile Augmented Reality

3 of 109

Introduction

Google's ARCore

4 of 109

Introduction

Google's ARCore

5 of 109

Introduction

Mobile Augmented Reality

6 of 109

Introduction

Motivation

Is direct placement and rendering of 3D objects sufficient for realistic AR experiences?

7 of 109

Introduction

Depth Lab

Not always!

8 of 109

Introduction

Depth Lab

Virtual content looks like it’s “pasted on the screen” rather than “in the world”!

9 of 109

Introduction

Motivation

10 of 109

Introduction

Motivation

11 of 109

Introduction

Depth Lab

How can we bring these advanced

features to mobile AR experiences without relying on dedicated sensors or the need for computationally expensive surface reconstruction?

12 of 109

Introduction

Depth Map

13 of 109

Introduction

Depth Lab

Google	•Pixel 2, Pixel 2 XL, Pixel 3, Pixel 3 XL, Pixel 3a, Pixel 3a XL, Pixel 4, Pixel 4 XL
Huawei	•Honor 10, Honor V20, Mate 20 Lite, Mate 20, Mate 20 X, Nova 3, Nova 4, P20, P30, P30 Pro
LG	•G8X ThinQ, V35 ThinQ, V50S ThinQ, V60 ThinQ 5G
OnePlus	•OnePlus 6, OnePlus 6T, OnePlus 7, OnePlus 7 Pro, OnePlus 7 Pro 5G, OnePlus 7T, OnePlus 7T Pro
Oppo	•Reno Ace
Samsung	•Galaxy A80, Galaxy Note8, Galaxy Note9, Galaxy Note10, Galaxy Note10 5G, Galaxy Note10+, Galaxy Note10+ 5G, Galaxy S8, Galaxy S8+, Galaxy S9, Galaxy S9+, Galaxy S10e, Galaxy S10, Galaxy S10+, Galaxy S10 5G, Galaxy S20, Galaxy S20+ 5G, Galaxy S20 Ultra 5G
Sony	•Xperia XZ2, Xperia XZ2 Compact, Xperia XZ2 Premium, Xperia XZ3
Xiaomi	•Pocophone F1
	And growing…�https://developers.google.com/ar/discover/supported-devices

14 of 109

Introduction

Depth Lab

Is there more to realism than occlusion?

15 of 109

Introduction

Depth Lab

Surface interaction?

16 of 109

Introduction

Depth Lab

Realistic Physics?

17 of 109

Introduction

Depth Lab

Path Planning?

18 of 109

19 of 109

Introduction

Depth Lab

20 of 109

Related Work

Valentin et al.

21 of 109

Introduction

Depth Lab

22 of 109

Introduction

Depth Lab

23 of 109

Introduction

Depth Generation

24 of 109

Introduction

Depth Lab

25 of 109

Related Work

Valentin et al.

26 of 109

Introduction

Depth Lab

27 of 109

Introduction

Depth Lab

Up to 8 meters, with

the best within 0.5m to 5m

28 of 109

Motivation

Gap from raw depth to applications

29 of 109

Introduction

Depth Lab

ARCore

Depth API

DepthLab

Mobile AR developers

30 of 109

Design Process

3 Brainstorming Sessions

3 brainstorming sessions

18 participants

39 aggregated ideas

31 of 109

Design Process

3 Brainstorming Sessions

32 of 109

System

Architecture overview

33 of 109

Data Structure

Depth Array

2D array (160x120 and above) of 16-bit integers

34 of 109

Data Structure

Depth Mesh

35 of 109

Data Structure

Depth Texture

36 of 109

System

Architecture

37 of 109

Localized Depth

Coordinate System Conversion

38 of 109

Localized Depth

Normal Estimation

39 of 109

Localized Depth

Normal Estimation

40 of 109

Localized Depth

Normal Estimation

41 of 109

Localized Depth

Avatar Path Planning

42 of 109

Localized Depth

Rain and Snow

43 of 109

Surface Depth

Use Cases

44 of 109

Surface Depth

Physics collider

Physics with depth mesh.

45 of 109

Surface Depth

Texture decals

Texture decals with depth mesh.

46 of 109

Surface Depth

3D Photo

Projection mapping with depth mesh.

3D photo effect is also included in DepthLab with texture projection mapping techniques.

//

\by reprojecting a color texture to a frozen depth mesh with rotating viewpoint.

When the user click the 3D photo capture button, we save the current camera image, depth image, and camera parameters.

Given the screen center, we first resolve the central vertex in the real world, based on the current depth map.

Next, we generate a depth mesh

Based on the camera's world position, we can compute the directional vector from the camera to the central point.

Next, with a cross product, we can compute the plane perpendicular to the direction.

Then we let the camera rotates around in the plane in a circle.

Finally, we recompute the projection matrix based on the camera's position and project the cached texture to the depth mesh.

The details is also opensourced in GitHub.

47 of 109

Dense Depth

Depth Texture - Antialiasing

48 of 109

Dense Depth

Real-time relighting

θ

N

L

49 of 109

Dense Depth

Why normal map does not work?

50 of 109

Dense Depth

Real-time relighting

51 of 109

Dense Depth

Real-time relighting

52 of 109

Dense Depth

Real-time relighting

go/realtime-relighting, go/relit

53 of 109

Dense Depth

Wide-aperture effect

54 of 109

Dense Depth

Occlusion-based rendering

55 of 109

Experiments

DepthLab minimum viable application

56 of 109

Experiments

General Profiling of MVP

57 of 109

Experiments

Relighting

58 of 109

Experiments

Aperture effects

59 of 109

Discussion

Deployment with partners

60 of 109

Discussion

Deployment with partners

61 of 109

Discussion

Deployment with partners

62 of 109

Limitations

Design space of dynamic depth

Dynamic Depth? HoloDesk, HyperDepth, Digits, Holoportation for mobile AR?

63 of 109

Envision

Design space of dynamic depth

64 of 109

GitHub

Please feel free to fork!

65 of 109

Play Store

Try it yourself!

66 of 109

DepthLab: Real-time 3D Interaction with Depth Maps for Mobile Augmented Reality

Ruofei Du, Eric Turner, Maksym Dzitsiuk, Luca Prasso, Ivo Duarte,

Jason Dourgarian, Joao Afonso, Jose Pascoal, Josh Gladstone, Nuno Cruces,

Shahram Izadi, Adarsh Kowdle, Konstantine Tsotsos, David Kim

Google | ACM UIST 2020

67 of 109

Thank you!

DepthLab | UIST 2020

68 of 109

Demo

DepthLab | UIST 2020

69 of 109

Introduction

Depth Map

Introduction

Depth Map

70 of 109

Introduction

Depth Lab

71 of 109

Thank you!

www.duruofei.com

72 of 109

Introduction

Depth Lab

Occlusion is a critical component for AR realism!

Correct occlusion helps ground content in reality, and makes virtual objects feel as if they are actually in your space.

73 of 109

Introduction

Motivation

74 of 109

Depth Mesh

Generation

75 of 109

Localized Depth

Avatar Path Planning

76 of 109

Dense Depth

Depth Texture

77 of 109

Introduction

Depth Map

78 of 109

Taxonomy

Depth Usage

79 of 109

Introduction

Depth Map

80 of 109

Introduction

Depth Map

81 of 109

Google’s developer platform for building mobile augmented reality (AR) experiences�
Allows you to integrate virtual content with �The real world as seen through your phone’s camera via:

Motion Tracking
Environmental Understanding
Lighting Estimation�

Released in 2017 and supported on over�1 billion devices.

What is ARCore?

82 of 109

02 ARCore Depth API

01 AR Realism: Occlusion, Immersion, & Interaction

03 New Depth Capabilities!

83 of 109

83

02 ARCore Depth API

01 AR Realism: Occlusion, Immersion, & Interaction

03 New Depth Capabilities!

84 of 109

Blend the virtual

with the real

AR in Search animals

85 of 109

Blend the virtual

with the real

86 of 109

Enable AR Realism using �the ARCore Depth API

Occlusion

Immersion

Interaction

87 of 109

Create

occlusion effects

Without occlusion

With occlusion

88 of 109

Transform

your scene

89 of 109

Interact �with

your world

That’s not all, depth also enables virtual content to interact with the real world through collision and physics, [PAUSE] which lets users really reach out and touch the world through your app.

[CLICK] Users can make virtual characters aware of the real scene and go over obstacles like you see here.

[CLICK] In this second example [PAUSE] with the depth of the tree known, [PAUSE] you can see how the colored paintballs are colliding and splattering on the branches of the tree.

And when depth based collision is combined with game physics, [PAUSE] the whole experience comes to life!

[CLICK] The virtual objects interact and bounce off the real-world objects [PAUSE] like the exercise bike in the third example, just as they would in reality.

Using these tools, we’ve seen a number of developers make their experiences more immersive and realistic using Depth API.

90 of 109

AR Realism

In TikTok

91 of 109

AR Realism

Built into Lens Studio for Snapchat Lenses

Kevaid

Saving Chelon

Quixotical�The Seed: World of Anthrotopia

Snap�Dancing Hotdog

92 of 109

92

02 ARCore Depth API

01 AR Realism: Occlusion, Immersion, & Interaction

03 New Depth Capabilities!

93 of 109

Depth Maps

94 of 109

94

Depth �from Motion

Depth From a Single Camera

No Depth Yet

95 of 109

Depth �from Motion

Depth From a Single Camera

96 of 109

Best Practices

Depth From a Single Camera

Use depth-certified ARCore devices

Minimal movement in the scene

Encourage users to move the device

Depth from 0 to 8 meters

Best accuracy 0.5 to 5 meters

97 of 109

97

try {

Image depthImage

= frame.acquireDepthImage();

doSomething(depthImage);

} catch (NotYetAvailableException e) {

// Depth is not available yet. This is

// normal at the beginning of a session

// before the user has moved around.

}

98 of 109

98

Session session = new Session(/* context= */ this);

Config config = session.getConfig();

if (session.isDepthModeSupported(Config.DepthMode.AUTOMATIC)) {

config.setDepthMode(Config.DepthMode.AUTOMATIC);

} else {

config.setDepthMode(Config.DepthMode.DISABLED);

}

session.configure(config);

99 of 109

More than 360 million devices support the ARCore Depth API

100 of 109

Enhancing Depth

Optimized to give you the best depth

Depth from Motion is fused with state-of-the-art Machine Learning

Depth leverages specialized hardware like a Time-of-Flight sensor when available

101 of 109

02 ARCore Depth API

01 AR Realism: Occlusion, Immersion & Interaction

03 New Depth Capabilities!

102 of 109

Camera Image

3D Point Cloud

Provides a more detailed representation of the geometry of the objects in the scene.

Raw Depth API

New depth capabilities

103 of 109

Camera Image

Raw Depth Image

Depth Image

Confidence Image

New depth capabilities

Raw Depth API

Provides a more detailed representation of the geometry of the objects in the scene.

104 of 109

104

try {

Image rawDepthImage

= frame.acquireRawDepthImage();

Image confidenceImage

= frame.acquireRawDepthConfidenceImage();

doSomething(rawDepthImage, confidenceImage);

} catch (NotYetAvailableException e) {

// Depth is not available yet.

}

105 of 109

3D Live Scanner App

Try it yourself!

PHORIA�ARConnect App

Jam3�

Raw Depth API

New depth capabilities

106 of 109

Uses depth to augment hitTest results and works just like existing hit tests.

Hit Results with Planes

Works best on horizontal, planar surfaces with

good texture

Hit Results with Depth

Gives more results, even �on non-planar or�low-texture areas

Depth �Hit Test

New depth capabilities

107 of 109

107

List<HitResult> hitResultList = frame.hitTest(tap);

for (HitResult hit : hitResultList) {

Trackable trackable = hit.getTrackable();

if (trackable instanceof Plane

|| trackable instanceof Point

|| trackable instanceof DepthPoint) {

doSomething(hit.createAnchor());

}

108 of 109

Try it yourself!

TeamViewer�LifeAR App

ARCore�Depth Lab App

Depth �Hit Test

New depth capabilities

109 of 109

ARCore�Depth Lab App

Depth API�Codelab

Raw Depth API�Codelab

So what's next? That's up to you! Please check out all the ARCore Depth API experiences shared in the Google I/O Sandbox this year.

We are providing several open-source examples to start with.�

[CLICK] Our Depth Lab is open-sourced in Unity, and has been updated with our latest features, providing a bunch of creative ways to use depth.
And we have two codelabs, [CLICK] including one that teaches you how to retrieve and apply depth images for occlusion, and...
And one that walks you through building an app using Raw Depth for geometric analysis, doing things like detecting objects and finding their bounding box, shown here.

Whether it’s to make fun videos, measure the world, or play a game, developers are using the Depth API to make their AR experiences more realistic and more immersive - and so can you!

We are excited to see what you build with the ARCore Depth API!