1 of 10

360° VIDEO CLOUD STREAMING & HTMLVIDEOELEMENT EXTENSIONS

Louay Bassbouss | Fraunhofer FOKUS

W3C Workshop on Web & Virtual Reality, October 19-20, 2016; San Jose, CA, USA

September 2016

2 of 10

360° video cloud streaming

W3C Workshop on Web & Virtual Reality

https://www.fokus.fraunhofer.de/go/360

2

September 2016

3 of 10

360° STREAMING AND VIDEO PROCESSING OPTIONS

W3C Workshop on Web & Virtual Reality

360° Processing

Server

Client

Video Playback

Streaming

Streaming

360° Processing

Video Playback

user input

user input

​

360° Pre-Processing

Streaming

...

Prepare Video

Video Playback

user input

​

Option1

Option2

Option3a

3

September 2016

4 of 10

360° STREAMING AND VIDEO PROCESSING OPTIONS

W3C Workshop on Web & Virtual Reality

360° Processing

Server

Client

Video Playback

Streaming

Streaming

360° Processing

Video Playback

user input

user input

​

360° Pre-Processing

Streaming

...

Prepare Stream

Video Playback

user input

​

Option1

Option2

Option3b

4

September 2016

5 of 10

ADVANTAGES AND DISADVANTAGES

W3C Workshop on Web & Virtual Reality

​

Option1

Option2

Option3a

Option3b

Additional Storage

No

No

Yes

Yes

360° Video Processing on Client

Yes

No

No

No

360° Video Processing on Server

No

Yes

No1

No1

Bandwidth

High

Low

Low

Low2

Motion-to-Photon Delay

Low

Medium3

Medium3

Medium4

CDN usage

Yes

No5

No5

Yes

Example Target Devices

Head Mounted Displays

Low Capability Devices e.g. HbbTV

Low Capability Devices e.g. HbbTV

Medium Capability Devices e.g. Chromecast

Interaction Types

  • Motion Sensors
  • Touch/Mouse
  • TV RC
  • Keyboard
  • (Touch/Mouse)
  • TV RC
  • Keyboard
  • (Touch/Mouse)
  • TV RC
  • Keyboard
  • Touch/Mouse

5

September 2016

6 of 10

Notes for previous slide

W3C Workshop on Web & Virtual Reality

  1. The original 360° video will be pre-processed and FOVs are stored in separate files. There will be an overlap between the FOVs this is why there is a need for more storage but on the other side no video processing is needed.
  2. since only one FOV is streamed to the client, no additional bandwidth is needed comparing to traditional video streaming. But it is still possible to pre-cache neighboring FOVs e.g. in lower quality to enable fast switch between FOVs in this case additional bandwidth is needed
  3. Motion to Photon delay depends on network latency and protocol used to stream a single FOV (and Buffering on the Player).
  4. Motion to Photon Delay depends on the caching strategy of the player.
  5. it is difficult to use CDN since a persistent connection between client and server is needed (there is a session for each client)

​

6

September 2016

7 of 10

DEMONSTRATION (Option 3b)

W3C Workshop on Web & Virtual Reality

4k origin 360° Video, 30fps, bitrate 40053 kb/s

HD view port, 30fps, bitrate 2435 kb/s, segment=333ms

106.7°

60°

  • HTML5 Video Element (MSE)
  • Intelligent/Efficient Buffering (two dimensions: time and space)
  • No Canvas, WebGL or any other APIs are required

...

...

360°

Pre-Processing

Prepare Stream

(Caching)

X

7

September 2016

8 of 10

Demo video

8

September 2016

9 of 10

HTMLVIDEOELEMENT EXTENSIONS

W3C Workshop on Web & Virtual Reality

  • Two types of players:
      • Native 360° Video Player
      • Using MSE → do we need extensions for the MSE API?
    • Native 360° Video Player:
      • The HTMLVideoElement plays 360° video natively. Set video.src={360_video_url} (or use <source element>)
      • The HTMLVideoElement needs to get all the metadata in order to render the view correctly.
      • New functions to set and get the FOV are needed
      • New events on start, during and after changing the FOV are needed
      • (Maybe also functions and events for Zoom in/out.)
      • Example:
        • video.setFOV(phi, theta, width, height)
        • video.onfovstart, video.onfovend, ...

9

September 2016

10 of 10

HTMLVIDEOELEMENT EXTENSIONS

W3C Workshop on Web & Virtual Reality

    • MSE 360° Player
      • Allows to implement different player algorithms similar for DASH on top of MSE
      • Available viewports can be described in a manifest (e.g. DASH SRD fields)
      • At the start of the playback the currently selected viewport is buffered. When the user triggers a switch request for a different viewport, already buffered segments are removed/replaced by segments of the new viewport.
      • Challenge:
        • How to reduce delay by switching between two viewports?

10

September 2016