1 of 62

1

Kernel Foveated Rendering

Xiaoxu Meng, Ruofei Du, Matthias Zwicker and Amitabh Varshney

Augmentarium | UMIACS

University of Maryland, College Park

ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games 2018

2 of 62

2

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

Application

Resolution

Frame rate

MPixels / sec

Desktop game

1920 x 1080 x 1

60

124

3 of 62

3

Application

Resolution

Frame rate

MPixels / sec

Desktop game

1920 x 1080 x 1

60

124

2018 VR

(HTC Vive PRO)

1440 x 1600 x 2

90

414

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

4 of 62

4

* Data from Siggraph Asia 2016, Prediction by Michael Abrash, October 2016

Application

Resolution

Frame rate

MPixels / sec

Desktop game

1920 x 1080 x 1

60

124

2018 VR

(HTC Vive PRO)

1440 x 1600 x 2

90

414

2020 VR *

4000 x 4000 x 2

90

2,880

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

5 of 62

5

  • Virtual reality is a challenging workload

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

6 of 62

6

  • Virtual reality is a challenging workload

  • Most VR pixels are peripheral

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

fovea:

the center of the retina

corresponds to the center of the vision field

7 of 62

7

  • Virtual reality is a challenging workload

  • Most VR pixels are peripheral

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

foveal region:

the human eye detects significant detail

peripheral region:

the human eye detects little high fidelity detail

8 of 62

8

  • Virtual reality is a challenging workload

  • Most VR pixels are peripheral

foveal

region

foveal region

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

foveal region:

the human eye detects significant detail

peripheral region:

the human eye detects little high fidelity detail

9 of 62

9

  • Virtual reality is a challenging workload

  • Most VR pixels are peripheral

96 %

27 %

Percentage of the foveal pixels

4 %

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

* Data from Siggraph 2017, by Anjul Patney, August 2017

10 of 62

10

11 of 62

11

Foveated Rendering

12 of 62

12

  • Virtual reality is a challenging workload

  • Most VR pixels are peripheral

  • Eye tracking technology available

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

13 of 62

13

Related Work

14 of 62

14

Full Resolution

 

 

Multi-Pass Foveated Rendering [Guenter et al. 2012]

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

15 of 62

15

Rasterizer

Early Z

 

Generate Coarse Quad

Shade

Evaluate Coarse Pixel Size

Input primitives

Coarse Pixel Shading (CPS) [Vaidyanathan et al. 2014]

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

16 of 62

16

CPS with TAA & Contrast Preservation [Patney et al. 2016]

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

17 of 62

17

Can we change the resolution gradually?

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

18 of 62

18

Perceptual Foveated Rendering [Stengel et al. 2016]

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

19 of 62

19

Is there a foveated rendering approach

without

the expensive pixel interpolation?

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

20 of 62

20

 

 

 

 

Log-polar mapping [Araujo and Dias 1996]

 

 

Log-polar Mapping

 

 

 

 

 

 

 

 

 

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

21 of 62

21

Log-polar mapping [Araujo and Dias 1996]

 

 

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

 

 

Log-polar Mapping

22 of 62

22

Log-polar mapping [Araujo and Dias 1996]

 

 

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

 

 

Log-polar Mapping

23 of 62

23

Log-polar mapping [Araujo and Dias 1996]

 

 

 

Log-polar Mapping

 

 

 

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

24 of 62

24

Log-polar mapping [Araujo and Dias 1996]

 

 

 

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

 

 

 

Log-polar Mapping

25 of 62

25

Log-polar mapping [Araujo and Dias 1996]

 

 

 

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

 

 

 

Log-polar Mapping

26 of 62

26

Log-polar mapping [Araujo and Dias 1996]

 

 

 

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

 

 

 

Log-polar Mapping

27 of 62

27

Log-polar Mapping for 2D Image [Antonelli et al. 2015]

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

28 of 62

28

Log-polar Mapping for 2D Image

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

29 of 62

29

Our Approach

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

30 of 62

30

Kernel Log-polar Mapping

 

 

 

 

 

range: [0,1]

 

 

 

 

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

31 of 62

31

 

 

 

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

 

 

 

Log-polar Mapping

32 of 62

32

 

 

 

 

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

 

Kernel Log-polar Mapping

 

33 of 62

Kernel Foveated Rendering

33

 

34 of 62

34

Kernel log-polar Mapping

 

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

 

 

 

35 of 62

35

Kernel log-polar Mapping

 

 

 

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

36 of 62

36

 

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

37 of 62

37

 

Original Frame

Buffer

Screen

Sample Map

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

38 of 62

38

 

Original Frame

Buffer

Screen

Sample Map

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

39 of 62

39

 

Original Frame

Buffer

Screen

Sample Map

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

40 of 62

40

 

 

 

Fovea

Fovea

Fovea

41 of 62

41

 

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

42 of 62

42

 

Original Frame

Buffer

Screen

Sample Map

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

43 of 62

43

 

Original Frame

Buffer

Screen

Sample Map

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

44 of 62

44

 

Original Frame

Buffer

Screen

Sample Map

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

45 of 62

45

 

 

 

Fovea

Fovea

Fovea

46 of 62

46

User Study

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

47 of 62

47

accept

reject

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

 

 

48 of 62

48

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

49 of 62

49

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

50 of 62

50

Kernel log-polar transformation

G-buffer

Inverse kernel

log-polar transformation

& post anti-aliasing

Shading &

internal anti-aliasing

World position

Bit tangent

Normal

Texture coordinates

Albedo map

Roughness, ambient, and refraction maps

 

Screen

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

51 of 62

51

original ray-marching scene

10 FPS

foveated ray-marching scene (σ = 1.8, α = 4)

30 FPS

fovea

* Scene created by Íñigo Quílez.

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

52 of 62

52

original 3D geometries

31 FPS

foveated 3D geometries (σ = 1.8, α = 4)

67 FPS

fovea

fovea

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

53 of 62

53

Scene

3D Textured Meshes

Ray Casting

Resolution

Ground Truth

Foveated

Speed up

Ground Truth

Foveated

Speed up

55 FPS

110 FPS

2.0X

20 FPS

57 FPS

2.9X

31 FPS

67 FPS

2.2X

10 FPS

30 FPS

3.0X

8 FPS

23 FPS

2.8X

5 FPS

16 FPS

3.2X

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

54 of 62

Summary

  •  

54

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

55 of 62

55

Ground Truth

Kernel Foveated Rendering

56 of 62

56

Thanks!

Introduction

Related Work

Our Approach

User Study

Experiments

Conclusion

57 of 62

57

Kernel Foveated Rendering

Xiaoxu Meng, Ruofei Du, Matthias Zwicker and Amitabh Varshney

Augmentarium | UMIACS

University of Maryland, College Park

ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games 2018

video

paper

58 of 62

FOVE Headset

  • DISPLAY
    • WQHD OLED (2560 X 1440)
    • Frame rate: 70fps
    • Field of view: Up to 100 degrees 
  • EYE TRACKING SENSORS
    • Infrared eye tracking system x 2
    • Tracking accuracy: less than 1 degree
    • Frame rate: 120fps

58

59 of 62

User Study: Significance

59

1.2

1.4

1.6

1.8

2.0

2.2

2.4

Cochran’s Q value

1.72

5.76

8.20

8.25

7.49

14.27

5.48

p-value

0.631

0.122

0.042

0.041

0.058

0.002

0.139

60 of 62

Two-level Anti-aliasing

60

Kernel log-polar transformation

G-buffer

Inverse kernel

log-polar transformation

& post anti-aliasing

Shading &

internal anti-aliasing

World position

Bit tangent

Normal

Texture coordinates

Albedo map

Roughness, ambient, and refraction maps

 

Screen

61 of 62

Two-level Anti-aliasing

61

Inverse kernel

log-polar transformation

& post anti-aliasing

Shading &

internal anti-aliasing

Non-uniform Gaussian Blur

Kernel size increase from left (fovea) to right (periphery)

Non-uniform Gaussian Blur

Kernel size increase from fovea to periphery

62 of 62

Video & Paper

62

video

paper