1 of 100

Computer graphics for game development

Part: Graphics pipeline

​

Martin Pernica

(based on presentation from Martin Kahoun)

2 of 100

3 of 100

4 of 100

Table of Content

  • Introduction
  • Graphics Pipeline Overview
  • What are shaders?
  • Vertex Shader
  • Pixel Shader
  • Geometry Shader
  • Tessellation Shaders
  • Compute Shader
  • RayTracing Shaders
  • Mesh Shaders
  • Summary

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

5 of 100

Graphics Pipeline Overview

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

6 of 100

Graphics Pipeline Overview

  • Let’s start with CPU
  • Good for complex operations
    • Many instruction sets
  • Fewer cores
    • With varying performance characteristics (big.LITTLE)
  • Very complicated in terms of design

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

7 of 100

Graphics Pipeline Overview

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

Control

ALU

ALU

ALU

ALU

Cache

DRAM

8 of 100

Graphics Pipeline Overview

  • Why we need GPUs?
  • Pixels, many pixels
    • And numbers are growing over time
  • 640 x 480 = 307,200 pixels
  • 3840 x 2160 = 8,294,400 pixels
  • Instructions are bit simpler
  • Has many cores and memory levels
  • Heavy pressure on parallel processing (saturation)

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

9 of 100

Graphics Pipeline Overview

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

Memory

Memory

Core

Core

Core

Core

Core

Core

Core

Core

Memory

Memory

Core

Core

Core

Core

Core

Core

Core

Core

Memory

Memory

Core

Core

Core

Core

Core

Core

Core

Core

DRAM

10 of 100

Graphics Pipeline Overview

  • Not everything is great
  • Branching
  • State changes
    • e.g. texture, buffer, RTs, …
  • Complex math
    • lookup tables often helps
  • High polygon count
  • And yes, even pixels
    • Because of this we render in lower resolution and upscale afterwards

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

11 of 100

Graphics Pipeline Overview

  • In the past GPU interaction (from dev perspective) was very limited
    • Minimal programmable stages
    • Fixed functions stages
  • GPU hardware was very complicated
    • Every operation had its own “block”

​

  • But it was a start of something great…

​

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

12 of 100

Graphics Pipeline Overview

  • Modern GPUs are extremely programmable
    • Often the vendor proprietary code is implemented as GPU code, not fixed ”block”
  • Specialized hardware blocks are mostly gone (kind of – ray tracing)
    • Rise of CUDA and OpenCL

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

13 of 100

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

NVIDIA Ada AD102 GPU

14 of 100

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

15 of 100

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

16 of 100

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

17 of 100

Graphics Pipeline Overview

  • How do we talk with GPU?
  • Graphics API
    • APIs which bridges communication between CPU and GPU (mostly “driver”)
    • Modern APIs focus on “giving developers as much flexibility as possible with tradeoffs”

​

  • Shaders
    • GPU code, like ”C++ for GPU”
    • The language itself is API dependent
    • Compiled often to “bytecode” not assembly (excluding consoles or fixed hardware systems like Apple M3 and others)

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

18 of 100

Graphics Pipeline Overview

  • Microsoft Direct3D (often referred as DirectX)
  • Direct3D 11
    • older, yet still working, high-level
  • Direct3D 12
    • newer, most development focus goes here, very low-level and complicated

​

  • Works primarily on Microsoft platforms Windows and Xbox
    • yet very well supported on translation layers like Proton

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

19 of 100

Graphics Pipeline Overview

  • OpenGL
  • High-level, older, yet still working and suitable for many applications
  • Cross-platform
    • Open standard managed by Khronos Group
  • As there is no “owner”, implementations and their features sets wildly varies across vendors

​

  • Many features implemented as extensions

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

20 of 100

Graphics Pipeline Overview

  • Vulkan
  • Low-level, modern API, successor to OpenGL
  • Cross-platform
    • Open standard same as OpenGL
  • Main competitor to Direct3D 12

​

  • Historically derived from AMD Mantle (donated by AMD)

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

21 of 100

Graphics Pipeline Overview

  • Others
    • Apple Metal
    • Xbox
    • PlayStation
    • Nintendo Switch

​

  • Console APIs are closed, no public documentation exists

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

22 of 100

Graphics Pipeline Overview

  • The trend is to give developers high control over GPUs
  • But there is huge tradeoff, even small mistakes can lead to crashes, hangs and other issues
  • In the past the driver was the “fixer”
    • Patching shaders and API calls during runtime based on game

​

  • Due the much more complex nature of modern APIs, the “debug layers” are essential for development

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

23 of 100

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

24 of 100

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

25 of 100

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

26 of 100

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

27 of 100

Rendering Basics

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

28 of 100

Rendering Basics

  •  

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

Translation, rotation, scaling and projection can be treated as matrix!

29 of 100

Rendering Basics

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

30 of 100

Rendering Basics

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

31 of 100

Rendering Basics – Coordinate Spaces

  • Model - positions relative to the local origin (aka object space)
  • World - positions relative to the global origin
  • View - positions relative to the viewer (aka camera or eye space)
  • Clip - vertex positions after projection into homogeneous space
  • NDC - clip-space coords divided by homogeneous w component
  • Window - pixel positions relative to window origin

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

32 of 100

Rendering Basics - Vertex

  • Position
  • Normal
  • Texture coord
  • Color
  • …

​

Vertex attributes loaded into arrays – Vertex Buffer Objects

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

33 of 100

Rendering Basics - Textures

  • Structured 1D, 2D, or 3D data
    • Color, Normal, Depth, Height, Cube (Skybox), 3D Volumetric, …
  • Texture coordinates s,t,r ∈ < 0,1 > or u,v,w
  • Loaded into video memory and sampled
  • Often filtered
    • Bilinear, bicubic, anisotropic

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

34 of 100

Rendering Basics - Textures

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

35 of 100

Rendering Basics – Texture Filtering

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

Nearest neighbor (left), bilinear (right)

36 of 100

Rendering Basics – Texture MipMaps

  • Texture LODs
  • Trilinear/anisotropic filtering

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

37 of 100

Rendering Basics – Texture MipMaps

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

38 of 100

Rendering Basics – Shader Stages

  • Vertex shader
  • Tessellation (optional)
  • Geometry
  • Primitive assembly (face culling)
  • Viewport transform
  • Rasterization to pixels/fragments
  • Pixel/fragment shader
  • Sample processing

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

39 of 100

Rendering Basics – Vertex Post-Processing

  • Transform feedback or stream-out (optional)
  • Perspective divide
  • View frustum clipping
  • Depth clamping
  • Viewport transform

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

40 of 100

Rendering Basics – Backface Culling

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

Front face — counter clockwise order Back face — clockwise order

41 of 100

Rendering Basics – Rasterisation

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

42 of 100

Rendering Basics – Attribute Interpolation

  •  

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

43 of 100

Rendering Basics – Perspective correction texture mapping

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

44 of 100

Rendering Basics – Per Sample Operations

  • Multisampling
    • MSAA — HW accelerated anti-aliasing
    • Sample polygon edges N times (depth, stencil, coverage mask)
    • Each sample jittered in screenspace coords
    • Fragment shader runs only once
    • Write color based on coverage mask & depth/stencil test
    • Faster than SSAA (supersampling anti-aliasing)
    • Doesn’t prevent texture aliasing (mip-maps do)

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

45 of 100

Rendering Basics – Per Sample Operations

  • Scissor Test
    • Discard fragments outside viewport defined rectangle
  • Stencil & Depth Test
    • Uses stencil & depth buffer
    • Test fragment stencil/depth value against buffer value
    • Discard fragment if the test fails
  • Early Fragment Test Optimization
    • Tests usually run before the fragment shader, if:
      • FS does not change Z value
      • FS does not discard fragment
      • We are not alpha blending

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

46 of 100

Rendering Basics – Per Sample Operations

  • Others
    • Occlusion query is updated
    • Alpha blending occurs
    • sRGB conversion takes place
    • Dithering & logical ops can be run
    • Fragment written w/ respect to write mask

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

47 of 100

Shaders

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

48 of 100

Shaders

  • Programs running on GPU cores
  • Many instances across hundreds/thousand cores at once
  • Transform geometry, affects the final look
  • Can be used for special effects and post-processing
  • Pretty much anything you want to do on GPU

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

49 of 100

Shaders – Shading Language

  • GLSL
    • OpenGL Shading Language
    • Used with OpenGL and Vulkan
  • HLSL
    • High-Level Shading Language
    • Used with Direct3D
    • Often used as primary language for cross-compilation
  • Metal Shading Language
    • Based on C++14
    • Used with Metal on Apple platforms

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

50 of 100

Shaders – Shading Language

  • NVIDIA Slang
    • New universal language for computer graphics and ML
    • Emits HLSL, GLSL/SPIR-V, CUDA, OptiX and scalar C++ for easier debug

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

51 of 100

Shaders – Scalar Types

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

GLSL

HLSL

Definition

bool

bool

Boolean true or false

float

float

FP-32

double

double

FP-64

int

int

Signed integer, 32-bits

unsigned int

uint

Unsinged integer, 32-bits

52 of 100

Shaders – HLSL Vector & Matrix Types

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

Type

Description

Examples

floatN

Represents a vector with N floating-point components

float4 (colors with alpha)

intN

Vectors with integer components

int3

uintN

Vectors with unsigned integer components

uint2

boolN

Vectors with boolean components

bool2

halfN

Half precision (FP-16) floating point number

half3

doubleN

Double precision (FP-64) floating point number

double2

floatMxN

Represents a matrix with M rows and N columns of floating-point values

float4x4

53 of 100

Shaders – HLSL Vector & Matrix Types

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

54 of 100

Shaders – HLSL Vector & Matrix Types

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

55 of 100

Shaders – Features

  • Matrices — arrays of column/row vectors
  • Arrays (w/ defined length() method)
  • C-like structures, conditionals, loops
  • Lots of built-in functions for math, etc.
  • Samplers for texture access
  • Constants set from the main program

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

56 of 100

Shaders – Debug

  • Visual aids (color coding etc.)
  • RenderDoc - open source, debugging, frame analysis
  • Miscrosoft PIX - frame capture, analysis, perf. analysis
  • MSVS Graphics Frame Analyzer & Shader debugger
  • Intel GPA - frame capture and analysis
  • NVIDIA Nsight – frame capture and analysis not only graphics code
  • GPU Perf studio / CodeXL - open source, debugging, frame analysis

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

57 of 100

Vertex Shaders

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

58 of 100

Vertex Shaders – GLSL Example

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

59 of 100

Vertex Shaders – HLSL Example

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

60 of 100

Tessellation Shaders

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

61 of 100

Tessellation Shaders

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

62 of 100

Tessellation Shaders

  • Treats vertices as control points
  • Runs in batches, once per control point
  • Determines tessellation parameters
  • Passes parameters down the line

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

63 of 100

Tessellation Shaders

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

64 of 100

Tessellation Shaders - Parameters

  • Number of divisions of the patch edges
  • Number of divisions of the patch interior

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

65 of 100

Tessellation Shaders

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

66 of 100

Tessellation Shaders – Evaluation Shader

  • TCS generates vertices w/ coords in u, v space
  • TES runs for each tessellated vertex
  • Calculates positions and params for tessellated vertices

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

67 of 100

Tessellation Shaders – Evaluation Shader

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

68 of 100

Tessellation Shaders – Terrain Rendering

  • Terrain square patch
  • Render it as N×N grid of quads
  • Adaptively tessellate each quad (TCS)
  • Apply displacement by heightmap (TES):
    • Sample heightmap texture
    • Set vertex z coordinate data
    • Effectively displaces vertex along surface normal
  • • All resulting triangles have roughly same screen-space area

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

69 of 100

Tessellation Shaders – Terrain Rendering

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

70 of 100

Tessellation Shaders – Terrain Rendering

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

71 of 100

Tessellation Shaders – Terrain Rendering

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

72 of 100

Tessellation Shaders – Terrain Rendering

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

73 of 100

Tessellation Shaders – Terrain Rendering

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

74 of 100

Tessellation Shaders – Terrain Rendering

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

75 of 100

Tessellation Shaders – Terrain Rendering

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

76 of 100

Tessellation Shaders – Call of Duty: Ghosts

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

77 of 100

Tessellation Shaders – Call of Duty: Ghosts

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

78 of 100

Geometry Shaders

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

79 of 100

Geometry Shaders

  • Runs per primitive (consumes primitive-assembly input)
  • Has access to all its vertex params
  • Can emit vertex or end primitive
  • Can change the primitive mode (e.g. create points from polygons)
  • Approx. 2x slower pipeline
    • Bad warp utilization (batch sizes wary)
    • Unpredictable output
    • Available memory limits number of parallel invocations

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

80 of 100

Geometry Shaders

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

81 of 100

Geometry Shaders

  • Single pass rendering into a cubemap
  • Cascaded shadow maps in one pass
  • Point sprite generation
  • Shadow volume extrusion
  • Pre-CS geometry processing using transform feedback
  • ...

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

82 of 100

Fragment/Pixel Shaders

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

83 of 100

Fragment/Pixel Shaders

  • Runs per fragment
  • Assigns color to fragment
    • Lighting
    • Shadows
    • Ambient Occlusion
    • Reflections
  • Can get easily very expensive
  • Make sure you are limiting overdraw

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

84 of 100

Fragment/Pixel Shaders

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

85 of 100

Fragment/Pixel Shaders

  • Not only used on “models”
  • But often used for post-processing (simplified)
    • Camera is rendered to texture/RenderTarget
    • The RT is mapped to rectangle and processed as another render pass
    • Which gives you ability to run fragment shaders on final image
  • Post-processing examples
    • Color grading
    • Depth of Field
    • Bloom
    • Screen-space reflections

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

86 of 100

Compute Shaders

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

87 of 100

Compute Shaders

  • Form effectively a separate single-stage pipeline
  • Perform arbitrary parallel computations
  • Like CUDA or OpenCL
  • Run in work groups over work items

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

88 of 100

Compute Shaders

  • Simulations (particles, water, ...)
  • Image postprocessing
  • Instance chain creation for indirect drawing
  • GPU scene preparation (frustum culling, etc.)
  • GPU occlusion culling
  • Software rasterization
  • ...

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

89 of 100

Compute Shaders

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

90 of 100

Compute Shaders

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

91 of 100

Compute Shaders

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

92 of 100

Compute Shaders - Example

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

93 of 100

Compute Shaders – Example (CPU)

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

94 of 100

RT & Mesh Shaders

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

95 of 100

RT & Mesh – Example (CPU)

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

96 of 100

RT & Mesh Shaders

  • Available through Vulkan & D3D12 extensions
  • Real magic happens in fast BVH traversal

​

  • Ray generation casts rays into the scene (primary, secondary)
  • Intersection invoked for ray-BV hit, API provides fast ray-triangle
  • Any hit can be used to discard hit using e.g. alpha testing
  • Closest hit evaluates surface BRDF
  • Miss evaluates scene background

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

97 of 100

RT & Mesh Shaders

  • Available through OpenGL, Vulkan & D3D12 extensions
  • Drop-in replacement for vertex, tessellation and geometry stages
  • Allow for more freedom, e.g., no need to supply patches and rely on tessellation engine
  • Better scalability than geometry shaders
  • Using execution model like compute shaders

​

  • Task dynamically generates work (mesh shader workgroups) much like TSC but with more freedom
  • Mesh generates primitives (much like GS)

​

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

98 of 100

Wrap-up

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

99 of 100

Useful books

  • Akenine-Möller T., Haines E., Hoffman N., Pesce A., Iwanicki M., and Hillaire S., Real-Time Rendering, 4th ed., 2018
  • Kessenich J., Sellers G., Shreiner D. OpenGL Programming Guide: The Official Guide to Learning OpenGL, Version 4.5 with SPIR-V (9th Edition), 2016
  • Wright R., Sellers G., Haemel N. penGL SuperBible: Comprehensive Tutorial and Reference (6th Edition), 2013
  • Zink J., Hoxley J., Pettineo M. Practical Rendering and Computation with Direct3D 11, 2011
  • Sellers G. Vulkan Programming Guide: The Official Guide to Learning Vulkan, 2016
  • GPU Gems series
  • GPU Pro series

Martin Pernica | @martindeveloper

Computer graphics for game development: Graphics pipeline

100 of 100

Thank you!