1 of 92

The WebVR API

How to design and build virtual reality directly using HTML, CSS, JavaScript

Dr. Pete Markiewicz - Febuary 15, 2017

2 of 92

Scope of the Talk

1. Virtual Reality landscape 2016.

2. The Case for WebVR.

3. Designing for WebVR.

4. WebVR Development.

5. Community.

6. Resources.

3 of 92

About the Author

Dr. Pete Markiewicz worked as a research scientist at UCLA and elsewhere until mid-1993.

Co-founded an early Internet music download & sales service, “Indiespace” in April 1994.

Author of books on the Internet, US Generations, & Tech.

Taught 1000s of students in Web, Graphic Design, Ux and Web Development since 2001.

Focus: WebVR (JavaScript) and Web-VR hybrid experiences.

4 of 92

The Virtual Reality Landscape

Where is VR in 2016?

5 of 92

We’re About get 4 New Media…

Virtual,Augmented and “Mixed” reality move design from graphic design principles to 3D design principles.

Voice-only “Oracle” systems (e.g. Amazon Echo) �eliminate visual design and replace it with �Ai conversations.

2000

2010

2020

Oracles

VR, AR, Mixed

6 of 92

Definitions

Virtual Reality – Create complete, exclusive virtual world via headsets

Augmented Reality – Add useful Ui elements to the real world, converting it to an interactive system.

Mixed Reality – Place virtual world elements seamlessly into real-world scenes to form a hybrid reality.

7 of 92

VR Milestones in 2018

2017 total VR market - about 4 billion.�

12% of consumers will buy VR devices this year.

VR products and demos at Best Buy (but NOT the Apple Store).

Android software and hardware gets built-in VR support.

WebXR API (VR and AR created in web pages via HTML, CSS, JavaScript) standard goes live in desktop and mobile.

WebVR Frameworks (e.g. Google’s new Draco library for compressing streaming VR)

8 of 92

Where are these New Techs Going?

9 of 92

VR will be in “Enlighten-�ment” zone in 2017!

10 of 92

Current VR limitations include...

Hardware cheap to expensive but with low distribution.

User Experience, VR works least well for fast-paced games it is being applied to.

Distribution, you can’t run a native VR app through the web, making consumer conversion difficult.

11 of 92

The “Mac” Problem

No Macintosh in production has a good �enough video card to run an Oculus or �HTC Vive experience

iOS has Ux features that “wreck” VR scenes �in the browser.

Apple is saying “we have 200 people working on the problem” – a bad sign…

Gamer early adopters still scoff at the Mac.

12 of 92

Growth Predictions

Typical research reports

Growth predicted to be lower �than in Jan 2016.

Why? Needs expensive new �hardware.

Why? No distribution model

for content (unlike the web).

Why? No data model

(unlike the web).

Why? No Social Network

(unlike the web).

Why? Content, nobody knows �how to build mass-market VR.

13 of 92

VR Investment in 2016

14 of 92

The Problem with Predictions…

Current studies tend to assume the dominant use of VR will be games via current game channels (Steam/Valve).

Assume high-end hardware will be necessary to participate.

Studies typically don’t even mention web-based solutions.

15 of 92

So, should I, or my Company get Involved?

Can’t I just wait until VR goes mainstream, or leave it to the gamers?

16 of 92

Uh,…You Need to be There!

There’s a strong case for you to begin work in VR, AR, Mixed Reality

17 of 92

Too often, VR is seen as “about games”

“…Only a few X-Treme (barfing) gamers will ever use VR on the Internet…”

18 of 92

But VR , AR, Mixed Reality are Their Own Medium…

Not like the web

Not like a game

Not like a movie

Most like a…book?

19 of 92

VR is a “Long Form” Medium

Feature			VR	Web	Mobile
1.	Engagement	>	Immersive, user is surrounded by media.	Casual, media only occupies part of the field of view.	Similar to web.
2.	Distraction	>	Low, self-contained worlds focus attention locally	High, hyperlinks encourage constant exploring instead of focus.	Medium, small screen limits choice to exit.
3.	Space and Time	>	Temporal, experience unfolds in space and time.	Spatial, information presented in flat layout.	Similar to web
4.	Presence	>	High, you feel “you’re there”.	Low, you are reading or looking at an event from outside.	Lower than web or VR.
5.	Interaction	>	Literal, you reach out to grab something	Abstract, you move a mouse to position a cursor.	Literal, you touch a control to use the control.
6.	Social	>	Low, current system have limited social networking	High, interface maps to real social networks.	Similar to web.

20 of 92

VR Requires Attention (the opposite of web-clicking)

It takes about 30 seconds to orient inside a VR world.

It takes at least 5 minutes to fully appreciate the value of the VR world.

An emotional “message” via VR takes 5-20 minutes to relay.

In other words, VR requires the focus of a good book.

21 of 92

VR Experience = “The Empathy Machine”

Extremely strong emotional engagement with content (unlike the web, but like great books).

Users are willing to “keep exploring” or watching (less desire to click somewhere else).

Stronger “sense of having been somewhere” (the 4^th wall of movies is in VR than web, TV).

Human empathy seems to require high-rez�depictions of events.

VR has more “emotional bandwidth” than �the Web.

22 of 92

VR Enables…

Empathic connection with remote scenes (experience like a knowing ‘ghost’).

Fake body enhancements (fake limbs, body-switching).

Therapy, situation and avoidance.

Acting out real-world behaviors with the �body, instead of just in the head.

Examining a complex systems all at once.

23 of 92

Non-Gaming VR Applications

Task organization (3d operating systems).

Real Estate & Big-ticket items (preview)

Therapy including aversion, role-playing.

Exercise/Training better than the TVs at the gym.

Mediation since the medium works best sloooowwww.

Authoring tools, who will use Maya3d when they can sculpt� by hand in VR?

24 of 92

Well, how do I get in?…

What I know is HTML, CSS, JavaScript, Ux, Ui, Node, ReactJS… not game engines!

HELP ME SPOCK!

25 of 92

The Case for WebVR

Why web-based VR is as important as native apps

26 of 92

WebVR Answers the VR Problem With…

Universal Hardware, since WebVR can run directly in current desktop and smartphone web browsers.

Distribution, since WebVR runs as part of a web page.

Data model, since existing cloud services, Ajax, Fetch work directly with WebVR apps.

Social network, since existing social sites �can be “skinned” with a WebVR UI.

Content, since much of existing web Ux theory�maps into VR and AR spaces.

27 of 92

WebVR Features

NO PLUGINS this is not Flash…

JavaScript API similar to existing device APIs (e.g. Media API).

Device-independent so you don’t need to work in a custom SDK.

Leverages existing WebGL construct VR and AR experiences that lead to more sustainable behavior outside the virtual world.

High-level libraries allow developing 3D apps in a useful framework.

Web-based Authoring apps allow (almost) free tools for 3d development of VR scenes and experiences.

28 of 92

Primary Development APIs

WebGL API enables fast 3D drawing

WebVR API links to VR devices, or defaults to browser window creating stereo images..

Audio API neededs, since sound is a major cue in VR.

Gamepad API supports haptic devices in VR.

Media API for embedded video, hybrid/AR.

29 of 92

Browser Support

Google Chrome Mobile

Samsung Internet Browser (great!)

Firefox 55

Microsoft Edge (windows mixed reality)

Safari (coming soon)

30 of 92

THREE.JS and WebVR

THREE.JS a complete �feature set for 3D �application creation.

A framework, rather than a game engine.

Originally a Flash AS3 library, ported to JavaScript in 2009.

A HUGE community of designers and developers supporting the project.

Supports Oculus, Vive hardware, selecting objects in VR.

31 of 92

Babylon.JS and WebVR

Babylon.js is a game engine written in WebGL.

Provides full support for game development (several titles have been converted into Babylon from native code).

Uses camera (WebVR camera and DeviceOrientation camera) to enable VR scenes.

A large community of designers and developers supporting the project, but few WebVR examples (yet).

VR support currently in state of rapid flux.

32 of 92

PrimroseVR - http://primrosevr.com

A framework based on THREE.js

Built-in “room” model for creating and navigating between multiple rooms.

Fast, can use directly for App development

33 of 92

Declarative WebVR Libraries (Markup)

Use XML Tags to represent objects in 3D and/or VR space.

Easy to learn as a VR experience.

Easily embedded in existing websites as a 3D background scene.

34 of 92

A-Frame

A Mozilla project built on THREE.js.

Uses XML style markup to create �3D scenes.

Optimized for WebVR use.

Component model allows behaviors to �be attached to 3D objects.

Good choice for rapid 3D WebVR �prototyping.

http://aframe.io

<a-scene fog="type: linear; color: #AAB; far: 30; near: 0">

<a-assets>

</a-assets>

<a-entity position="0 0 -10">

<a-image position="0 -.2 5" src="#highlight1" rotation="-90 0 0"

scale="30 30 30"></a-image></a-entity></a-scene>

35 of 92

X3 DOM

Open-source framework based on WebGL.��Manipulate the 3D content by only adding, removing, or changing DOM elements.��Supports most of the HTML events �(like "onclick") on 3D objects.

<shape>

</material>

</appearance>

<box>

</box>

</shape>

</scene>

</x3d>

http://www.x3dom.org

36 of 92

ReactVR

Developed by Facebook for their Oculus platform

Exploits existing ReactJS framework

Uses ‘signed distance field’ fonts for greater clarity

https://facebookincubator.github.io/react-vr/index.html

class WelcomeToVR extends React.Component { render() { �// Displays "hello" text on top of a loaded 360 panorama image. �// Text is 0.8 meters in size and is centered three meters in front of you. �return ( � <View> � <Pano source={asset('chess-world.jpg')}/>� <Text style={{ fontSize: 0.8, layoutOrigin: [0.5, 0.5], transform: [{translate: [0, 0, -3]}], }}> hello� </Text>� </View> �); } }; AppRegistry.registerComponent('WelcomeToVR', () => WelcomeToVR);

37 of 92

X3D (Successor to VRML)

Web 3D Consortium

Formed to handle shortcomings in VRML, the original VR modeling language for the web

In development since 1999(!)

More complex than other 3d markup languages

http://www.web3d.org/working-groups/x3d

<?xml version="1.0" encoding="UTF-8"?>

<!DOCTYPE X3D PUBLIC "ISO//Web3D//DTD X3D 3.3//EN" "http://www.web3d.org/specifications/x3d-3.3.dtd"> <X3D profile='Immersive' version='3.3' xmlns:xsd='http://www.w3.org/2001/XMLSchema-instance' xsd:noNamespaceSchemaLocation =' http://www.web3d.org/specifications/x3d-3.3.xsd '> <head>

</head>

38 of 92

Transpiles From Other Dev Environments

Unity 5 – can transpile code from native Unity to JavaScript using ASM.JS. Working examples available.

Unreal4 – support for ASM.JS. No good examples yet.

Note: resulting code files are HUGE (~90 megs) compared to native JavaScript coding (~1 megs).

http://benchung.com/unity3d-vs-three-js /

https://github.com/gtk2k/Unity-WebVR-Assets

39 of 92

The Promise of WebAssembly

An assembly-language for web browsers

Executes inside JavaScript virtual machine (with same permissions)with near-native speed in execution

Will make Unity and Unreal ‘transpiles’ more quickly, but native WebVR even faster!

http://webassembly.org/

https://medium.com/javascript-scene/what-is-webassembly-the-dawn-of-a-new-era-61256ec5a8f6#.6d27o089r

40 of 92

Inclusive Design in WebVR

A11Y & WebXR - Roland Dubois

https://docs.google.com/presentation/d/1_mpJ1kOHxR1De-0z8G-Qfg8sKNC1DUCTvYw3IkYDyCI/edit?usp=sharing

3D spatial audio mapping - Google Resonance Audio

High quality spatialization (Filtering and Volume Attenuation)
Real-time Reverb (Material Responsive)
1st-3rd Order Ambisonics

A-Frame (HTML-like markup, machine-readable

https://github.com/rdub80/aframe-gui

WAI/ARIA Integration

https://github.com/realitylab-ruben/aframe-aria

41 of 92

Top WebVR Experiences (to date)

Puzzle Rain�https://mozvr.com/puzzle-rain/fallback.html

Awesome WebVR�https://github.com/dziemid/awesome-WebVR

Awesome A-Frame�https://github.com/aframevr/awesome-aframe

42 of 92

Some WebVR Pioneers

Brandon Jones/ Tojiro (Chrome WebVR implementator)�http://www.tomorrowtodaylabs.com/

Boris Sumus (Google, created WebVR Polyfill & Boilerplate) �http://smus.com/

Sean McBeth (Notion Theory, Creator of PrimroseVR framework)�http://www.seanmcbeth.com/ ��Diego Marcos (Mozilla WebVR, A-Frame)�https://github.com/dmarcos

Arturo Paracuellos (Master of WebVR experience & game design)�http://unboring.net�https://blog.mozvr.com/puzzle-rain/

43 of 92

Designing in WebVR

The full-full stack…

44 of 92

How do I remake my existing web/app workflow for WebVR?

Apply Ux and Ui principles already learned from the early history of consumer VR.

45 of 92

VR Hardware in 2016

Smartphone VR uses iOS or Android �devices with low-cost headsets (80%).

Headset VR uses a dedicated desktop �or laptop to render higher-quality�gaze-directed (15%).

Roomscale VR provides tethers, or �mounts the VR device on the user �for greater exploration (5%)

No current Macintosh is powerful enough

To do Headset or Roomscale!

46 of 92

Levels of Interaction

Passive – viewing, but only able to move head

Gaze – select objects by looking at them

Haptic – use a hardware sensor represent hands or other parts of the body to select objects

Full-Body – use sensors to determine the position of the entire body in the scene.

47 of 92

Haptics

Typically hand-held.

Have additional controls �(triggers, buttons).

Existing gamepads are often repurposed�into VR.

Gloves NOT in widespread use.

Need virtual display of haptic �inside VR to be effective.

48 of 92

Responsive VR Breakpoints

Design for low-end haptics, also gaze-only selection, no room-scale.

<= 80deg view,�Head Motions only

90-100deg view, plus hardware �button select on headset

110deg view, plus handheld

haptic devices

> 120deg view, full-body, �room-scale interaction

Google Cardboard

GearVR, Google Daydream

Oculus Rift, Rift Touch

HTC Vive

49 of 92

How do I Author (Traditional)?

Clara.io - http://clara.io

Vizor.io - http://vizor.io

PlayCanvas - http://playcanvas.com

ProntoVR - http://prontovr.com

Mettle - http://www.mettle.com/

Reference: https://virtualrealitypop.com/how-to-develop-apps-for-virtual-reality-hint-low-fidelity-prototyping-adb85c0cea09#.o9qc9fvyy

50 of 92

How do I Author in VR Itself?

Google Tilt Brush - https://www.tiltbrush.com

Quill (Oculus Story Studio) - https://www.oculus.com/experiences/rift/1118609381580656/

Pantheon VR - https://www.pantheonvr.com/

Gravity Sketch - https://www.gravitysketch.com/

51 of 92

Ux in VR Pioneers

Adrienne Hunter (founder, Tomorrow Today Labs)�http://www.tomorrowtodaylabs.com/

Mike Alger (VR designer, Google, Mountain View, CA) �http://mikealger.com/

Timoni West (Ux designer at Unity3d)�http://blog.timoni.org/

Josh Carpenter (Ux in VR for Mozilla/Firefox)�https://www.youtube.com/watch?v=ZOaOYTOpwyM

Jason Jerald (NextGen Interactions, 1^st VR design book)�http://www.thevrbook.net/

Arturo Paracuellos (Unboring.net)�https://blog.mozvr.com/puzzle-rain/

52 of 92

Web to VR Transitions

The WebVR API allows VR to be embedded naturally into web pages.

No real models for transitioning from a web page into VR, or vice-versa.

Embedding 3D HTML elements (A-Frame, X3DOM) may provide a Ui gateway into full VR experiences

You could the one to work this out!

53 of 92

Apparent Fields of View (FOV)

Field of View: 100deg, 60deg up, 40deg down.

Rotation: 55 degrees of head.

Resolution: minimum 960px per eye, 0.1 degrees per pixel.

Source: Mike Alger

54 of 92

Cylinder Ux Model (Josh Carpenter)

Low-Resolution

High-Resolution (Cinema 4D)

55 of 92

User Behavior in VR Space

56 of 92

User Perception in VR Space

Reference: https://blog.mozvr.com/quick-vr-prototypes/

57 of 92

Useful Ui Zones in VR (Mike Alger)

Touch Ui Zone

“No-no” Zone

Content Zone

58 of 92

A Note on Text

Text renders poorly in 3D compared to 2D.

Animated text can be used effectively (banners, or in-place replacement).

Audio can effectively replace text-based cues.

Static text should be a last resort.

New Grid Systems will be required.

59 of 92

Prototyping Techniques

“Spherical Grayboxing” – create grey polygons to substitute for Ui objects on gridded spheres (Alex Chu)

“Cylinder” – Flat 960px Illustrator template, edited, and wrapped around user model at 0.5m (Josh Carpenter)

60 of 92

A 2D VR Template

61 of 92

Sample Cylinder Grid System (Josh Carpenter)

62 of 92

Quick VR Prototypes in Illustrator (Josh Carpenter)

Reference: https://blog.mozvr.com/quick-vr-prototypes/

Template in Illustrator
Save as (large) PNG
Load in THREE.js

63 of 92

Affordances

Hardware buttons on the device.

Circular flat overlay menus or icons.

Local maps embedded in world.

Text input fields (hard).

Object gestures (the object makes a recognizable motion, like sign language).

Objects which look like the thing they manipulate.

Virtual copies of physical haptic devices in the scene.

64 of 92

Triggers

Prolonged gaze – object can expand or change appearance when stared at for a few seconds.

No Button “clicks” – when “touched”, the experience is more like putting finger into a container of water).

Audio Cues – sound changes on motion, selection.

65 of 92

Feedback

Hardware buttons on headset can provide tactile triggers.

Object moves (instead of scene moving).

New object appears in scene.

Scene dissolves (must be slow or will be nauseating).

Audio changes (intensity or type of sound).

Openings (a path, door, gateway).

66 of 92

Experience Metaphors

Curved walls – like a convention installation

Theater in the Round – events happen all around you

The Cave – similar to games, roomscale only

Disabled user (yes) – your limitations match VR limitations

The Workbench – pick and place tools, post-it notes, etc.

The Ghost’s Dream – you’re embedded in a world that can’t see you

‘Gone Fishing – the world flows by in a leisurely way

67 of 92

Object Roles (Jonathan Ravasz)

The ground/land – allows faster orientation to the scene

The sky – gives a sense of scale

Audio fadeins – introduce a change gently

Context reticle – show user where they gaze is going.

Objects – define paths for exploring

Talking objects – �provide detail usually from text

68 of 92

Anti-Patterns (Timoni West)

Moving the user when they aren’t moving (a common error in 360 video).

Static text (use animated text or audio).

Teleports or sudden scene shifts.

No time for orientation when scene changes.

No tracking feedback (users need to see constant motion related to their own.

Sudden motion of an object towards the user (especially dropping down from the sky to the head).

69 of 92

The 2D to 3D Challenge

WebVR access typically starts with a flat, 2d desktop or mobile screen.

Current models don’t include the transition between 2D web and 3D VR as part of the “customer journey”

More work needed to improve Ui and Ux for transitions into and out of VR

Discussion at: https://www.youtube.com/watch?v=78F_HNWuqbw

70 of 92

WebVR Development

It’s not a web page anymore…

71 of 92

Your Road to VR

Get a headset, of Samsung Gear VR/ Google Daydream or higher.

Explore some commercially-produced worlds (Steam community).

Learn something about audio design (it’s important).

Read “Ux in VR” blogs.

Practice long-form storytelling (again).

Start coding in JS or Declarative VR.

I voted for Trump!

72 of 92

Where’s your headset?”…

You’re in Print design, it’s 1994 and somebody is talking about the World Wide Web…

I’m Soooo Excited!

73 of 92

Your Choices

HTC Vive ($700), only system with true “room-scale” VR.

Oculus Rift ($500) most popular, better egronomics.

Samsung Gear VR (~$100) good quality, use existing Note or Edge phone

Google Daydream (~$100) similar to GearVR, but with Google Pixel phone.

Google Cardboard (~$10) with any smartphone (even iOS, lower quality of experiences).

74 of 92

You Need a (Windows) Workstation for High-End

Nvidia GTX 1070 or 1080 video card.

8 gigs or more of RAM.

Lots of USB, HDMI DisplayPort connectors.

Solid-state (SSD) hard drive.

Expect to spend ~$1500 or more for computer.

75 of 92

WebVR Support/Accessibility

Browser	Support	Comments
Samsung Internet Browser	Strong	Has supported WebVR since early 2016. Specific to Samsung Galaxy smartphones.
Google Chrome	Very Strong	Chromium supports most recent WebVR API, in Mobile Chrome in 2017. Oculus/Vive supported.
Mozilla Firefox / Servo	Very Strong	Nightly builds support WebVR now. No Oculus/Vive Support (yet). Servo Browser will support WebVR.
Microsoft Edge	Very Strong	WebVR native promised in 2017. MSFT actively developing W3C spec.
Oculus Carmel	Very Strong	Custom VR browser, plus ReactVR framework from Facebook. Oculus support guaranteed.
Safari	None	Little interest from Apple, no support promised. Missing support for other APIs (e.g. GamePad).
Opera	None	No support announced.

76 of 92

WebVR Polyfill & Boilerplate

Developed by Boris Sumus at Google.

Installs support for WebVR API in most browsers.

Works best with mobile, even Mobile Safari.

Can’t “fire” the device drivers for Vive, Oculus.

WebVR Boilerplate gives a starting �application using the THREE.js library.

https://github.com/borismus/webvr-boilerplate

77 of 92

Start Coding – 1 (Direct)

if (navigator.getVRDisplays) {� // Enumerate the VRDisplays� navigator.getVRDisplays().then(function (displays) {� if (!displays.length) {� VRSamplesUtil.addInfo("WebVR supported, but no VRDisplays found.");� return;� }� for (var i = 0; i < displays.length; ++i) {� document.body.appendChild(buildVRElement(displays[i]));� } � }� );� window.addEventListener("vrdisplayconnect", eventFired, false);� window.addEventListener("vrdisplaydisconnect", eventFired, false);� window.addEventListener("vrdisplayactivate", eventFired, false);� window.addEventListener("vrdisplaydeactivate", eventFired, false);� window.addEventListener("vrdisplayblur", eventFired, false);� window.addEventListener("vrdisplayfocus", eventFired, false);�} else {� VRSamplesUtil.addError("Your browser does not support WebVR. See <a href='http://webvr.info'>webvr.info</a> for assistance.");�}

78 of 92

Start Coding – 2 (Declarative)

<html>�<head>�<meta charset="utf-8">�<title>Hello, World! • A-Frame</title>�<meta name="description" content="Hello, World! • A-Frame">�<script src="../../../dist/aframe.js"></script>�</head>�<body>�<a-scene>� <a-sphere position="0 1.25 -1" radius="1.25" color="#EF2D5E"></a-sphere>� <a-box position="-1 0.5 1" rotation="0 45 0" width="1" height="1" depth="1" color="#4CC3D9"></a-box>� <a-cylinder position="1 0.75 1" radius="0.5" height="1.5" color="#FFC65D"></a-cylinder>� <a-plane rotation="-90 0 0" width="4" height="4" color="#7BC8A4"></a-plane>� <a-sky color="#ECECEC"></a-sky>� <a-entity position="0 0 3.8">� <a-camera></a-camera>� </a-entity>�</a-scene>�</body></html>