Project Geollery.com: Reconstructing a Live Mirrored World With Geotagged Social Media
Ruofei Du†, David Li†, and Amitabh Varshney
{ruofei, dli7319, varshney}@umiacs.umd.edu | www.Geollery.com | Web3D 2019, Los Angeles, USA
UMIACS
THE AUGMENTARIUM
VIRTUAL AND AUGMENTED REALITY LAB
AT THE UNIVERSITY OF MARYLAND
COMPUTER SCIENCE
UNIVERSITY OF MARYLAND, COLLEGE PARK
Project Geollery.com: Reconstructing a Live Mirrored World With Geotagged Social Media
Introduction
Social Media
2
image courtesy: plannedparenthood.org
Introduction
Social Media + Topics
3
image courtesy: huffingtonpost.com
Motivation
Social Media + XR
4
Motivation
Social Media + XR
5
image courtesy:
instagram.com,
facebook.com,
twitter.com
Motivation
2D layout
6
image courtesy:
pinterest.com
Motivation
Immersive Mixed Reality?
7
image courtesy:
viralized.com
Motivation
Pros and cons of the classic
8
Motivation
Pros and cons of the classic
9
Related Work
2D Geospatial Visualization
10
Related Work
2D Geospatial Visualization
11
Related Work
3D Geospatial Visualization
12
Related Work
Social Street View, Du and Varshney
Web3D 2016 Best Paper Award
13
Related Work
Social Street View, Du and Varshney
Web3D 2016 Best Paper Award
14
Related Work
Social Street View, Du and Varshney
Web3D 2016 Best Paper Award
15
Related Work
Social Street View, Du and Varshney
Web3D 2016 Best Paper Award
16
Related Work
VirtualOulu: Toni Alatalo et al.
Web3D 2016
17
Related Work
Virtual Oulu, Kukka et al.
CSCW 2017
18
Related Work
3D Visual Popularity
Bulbul and Dahyot, 2017
19
Related Work
Immersive Trip Reports
Brejcha et al. UIST 2018
20
Related Work
High Fidelity, Inc.
21
Related Work
Facebook Spaces, 2017
22
Geollery
A Mixed-Reality Social Media Platform, CHI 2019
23
Geollery.com
A Mixed-Reality Social Media Platform
24
What's Next?
Research Question ⅓
Du et al. Geollery, CHI 2019.
25
What may a social media platform look like in mixed reality?
What's Next?
Research Question ⅔
Du et al. Geollery, CHI 2019.
26
What if we could allow social media sharing in a live mirrored world?
What's Next?
Research Question 3/3
Du et al. Geollery, CHI 2019.
27
What use cases can we benefit from social media platform in XR?
System Overview
Geollery V1 Workflow
28
Design Space
Geollery vs. Social Street View
Du et al. Geollery, CHI 2019.
29
User Study
Quantitative Evaluation
Du et al. Geollery, CHI 2019.
30
Insights
What we learned
31
High-quality content and seed users play key roles
Insights
What we learned
32
Interactivity and panoramic textures increase immersion.
33
[I will use it for] exploring new places. If I am going on vacation somewhere, I could immerse myself into the location. If there are avatars around that area, I could ask questions.
P1 / M
34
I think it (Geollery) will be useful for families. I just taught my grandpa how to use Facetime last week and it would great if I could teleport to their house and meet with them, then we could chat and share photos with our avatars.
P2 / F
35
What if we could reconstruct a high-quality, all textured, walkable mirrored world with geotagged social media in real time?
Geollery.com
v2: a major leap
36
System Overview
Geollery Workflow
37
System Overview
Geollery Workflow
38
System Overview
Given (latitude, longitude)
39
System Overview
2D Map Data
40
System Overview
2D Map Data
41
System Overview
+Avatar
42
System Overview
+Avatar +Trees +Clouds
43
System Overview
+Avatar +Trees +Clouds +Night
44
System Overview
Street View Panoramas
45
System Overview
Street View Panoramas
46
System Overview
Street View Panoramas
47
System Overview
Geollery Workflow
48
System Overview
Geollery Workflow
49
System Overview
Geollery Workflow
50
System Overview
Geollery Workflow
51
All data we used is publicly and widely available on the Internet.
Rendering Pipeline
Close-view Rendering
52
Rendering Pipeline
Initial spherical geometries
53
Rendering Pipeline
Depth correction
54
Rendering Pipeline
Intersection removal
55
Rendering Pipeline
Texturing individual geometry
56
Rendering Pipeline
Texturing with alpha blending
57
Rendering Pipeline
Rendering result in the fine detail
58
Rendering Pipeline
Rendering result in the fine detail
59
Rendering Pipeline
Rendering result in the fine detail
60
Rendering Pipeline
Close-view Rendering
61
Rendering Pipeline
Gap Alignment
62
Rendering Pipeline
Gap Alignment
63
Rendering Pipeline
Gap Alignment
64
Rendering Pipeline
Seam Blending
65
Rendering Pipeline
Seam Blending
66
Rendering Pipeline
Seam Blending
67
Rendering Pipeline
Street View vs. Satellite Images
68
Rendering Pipeline
Street View vs. Satellite Images
69
Rendering Pipeline
Street View vs. Satellite Images
70
Rendering Pipeline
Gaussian Filtering
71
Rendering Pipeline
Gaussian Filtering
72
Rendering Pipeline
Gaussian Filtering
73
Rendering Pipeline
Occlusion Test
74
Rendering Pipeline
Occlusion Test
75
Rendering Pipeline
Occlusion Test
76
LRU Cache
Least Recently Used
77
5 adjacent street views are cached while users are walking.
Each geometry has 131,074 vertices to be processed by the GPU
Limitations
Inaccurate depth maps etc.
78
Deployment
Geollery.com
79
Rendering Pipeline
Experimental Features
80
81
Geollery/Social Street View has its own set of distinct offerings, as it is anchored within real-world settings, just mapped onto VR, whereas these are definitely more ‘fantasy’ type of arenas. In that way, as you have already done, I think there are multitude game challenges/tasks/feedback, like the balloons, to add in!
Email feedback from pilot users
82
I think it’d be cool if you could see posts by people in real time, along with the establishment they’re in (like someone tweeting from inside McDonald’s or a movie theater), if that makes sense. Sort of like checking in to a place on Facebook
Email feedback from pilot users
83
1
An interactive rendering pipeline of
Fusing 360° Panoramas
at two levels of detail.
84
2
Contributing a large-scale real-time system to
Reconstruct a Mirrored World
without the prior knowledge of any 3D models but only street view images and depth maps, which may be estimated from deep learning pipeline etc.
85
3
Establishing a web-based platform at
Geollery.com
for visualizing geotagged social media in a collaborative mixed-reality setting.
Discussion
Use Case: Audio Tour
86
...
Discussion
Taking the Feedback:
Geollery v2, Web3D & VR 2019
87
...
Discussion
Taking the Feedback
88
Challenge
Global Market Restraint:
Weak Content for XR
89
Research Goal
Fuse the information from physical and virtual world
90
ACM Web3D '19
Future Directions
The Ultimate XR Platform
91
Future Directions
Fuses Past Events
92
Future Directions
With the present
93
Future Directions
And look into the future
94
Future Directions
Change the way we communicate in 3D and consume the information
95
Future Directions
Consume the information throughout the world
96
Acknowledgement
NSF | MPower | UMIACS
97
Acknowledgement
Coauthors
98
Thank you!
Ruofei Du, David Li, and Amitabh Varshney
{ruofei, dli7319, varshney}@cs.umd.edu | www.Geollery.com | CHI 2019
UMIACS
THE AUGMENTARIUM
VIRTUAL AND AUGMENTED REALITY LAB
AT THE UNIVERSITY OF MARYLAND
COMPUTER SCIENCE
UNIVERSITY OF MARYLAND, COLLEGE PARK
Thank you!
Project Geollery.com: Reconstructing a Live Mirrored World With Geotagged Social Media
Ruofei Du, David Li, and Amitabh Varshney
{ruofei, dli7319, varshney}@umiacs.umd.edu | www.Geollery.com | CHI 2019 | Demo at D-2 (INT-40)
UMIACS
THE AUGMENTARIUM
VIRTUAL AND AUGMENTED REALITY LAB
AT THE UNIVERSITY OF MARYLAND
COMPUTER SCIENCE
UNIVERSITY OF MARYLAND, COLLEGE PARK
Project Geollery.com: Reconstructing a Live Mirrored World With Geotagged Social Media
User Study
Background Interview (5 min)
101
User Study
Quantitative Evaluation
102
User Study
Quantitative Evaluation
103
User Study
Post-interview
104
Post Interview
Question 1/3
105
Suppose that we have a polished 3D social media platform like Geollery or Social Street View, would you like to use it? If so, how much time would you like to spend on it?
Post Interview
Question 1/3
106
107
I would like to use it every day when I go to work, or travel during weekends.
P6 / F
108
If it’s not distracting like Facebook and Instagram, I would use it every day on a couple of things.
P17 / F
109
I am a follower on most social media sites. I would only join a 3D social media platform once my friends are there.
P4 / M
110
If my friends are all on this, I can see myself spend a couple of hours every week.
P12 / M
111
I don’t think I will use this. I prefer to use Yelp to see comments [of nearby restaurants]
P12 / M
Post Interview
Question 2/3
112
Can you imagine your use cases for Geollery and Social Street View? What would you like to use 3D social media platforms for?
113
I would like to use it for the food in different restaurants. I am always hesitating of different restaurants. It will be very easy to see all restaurants with street views. In Yelp, I can only see one restaurant at a time.
P6 / F
114
[I will use it for] exploring new places. If I am going on vacation somewhere, I could immerse myself into the location. If there are avatars around that area, I could ask questions.
P1 / M
115
I think it (Geollery) will be useful for families. I just taught my grandpa how to use Facetime last week and it would great if I could teleport to their house and meet with them, then we could chat and share photos with our avatars.
P2 / F
116
… for communicating with my families, maybe, and distant friends, [so] they can see New York. And, getting to know more people, connecting with people based on similar interests.
P2 / F
Post Interview
Question 3/3
117
If you were a designer or product manager for Geollery or Social Street View, what features would you like to add to the systems?
118
A mapping of the texture, high-resolution texture, will be great.
P12 / M
119
if there is a way to unify the interaction between them, there will be more realistic buildings [and] you could have more roof structures. Terrains will be interesting to add on.
P18 / M
120
I would like to see kitties and puppies running around, and birds flying in the air
P13 / F
121
I could also add a bike, add a vehicle, a motorcycle in Geollery, this will add some fun.
P17 / F