1 of 27

Overcoming latency, How we build a Cloud Game service in Go

Gophercon VietNam 2019

giongto35

2 of 27

Agenda

  • Cloud-Gaming introduction
  • WebRTC in Golang.
  • Write Cloud-Gaming in Go.

3 of 27

What is cloud gaming?

  • Cloud gaming is new generation of Gaming platform
  • Maximize backend control => minimize frontend logic.
  • Backend runs the games, and stream media to frontend.

4 of 27

Future Application

  • Heavy offline application to browser, mobile: Photoshop, operating system
    • Running MS Window on Chrome borwser
    • Photoshop CC on tablet

5 of 27

Challenges

  • So it’s streaming platform. Why it’s different among Twitch, Youtube Gaming?
  • Smooth user interaction: the gap between input and media as small as possible.
  • I decide write Proof Of Concept of Cloud-gaming in Go.
  • Go perfectly fits my usecase: data stream + concurrency

6 of 27

Proof of concept!

  • Github: https://github.com/giongto35/cloud-game
    • If you like the project please give me a star
  • Play at: http://cloudretro.io/

7 of 27

8 of 27

Tech stack

  • Single player
    • No Need for CDN, Centralized server, Stream doesn’t need to be shared.
    • WebRTC: Peer-to-peer communication between a slave server and user.
    • There is a master will be in charge of coordinating all slave servers and user.
  • Full control on Game Emulator
    • Hook frames directly from emulator. Easy state modification.
    • LibRetro
  • Low latency media stream
    • VPX/H264 Video Codec.
    • Opus Audio Codec.
  • Horizontally scalable
    • Self-implemented load-balancing

9 of 27

WebRTC?

The technology behind Cloud Gaming

10 of 27

WebRTC

  • Realtime Communication between mobile and web browser via simple API
    • Video call setup is simple with WebRTC
  • It’s peer to peer communication, optimized for media:
    • Integrated with common codec: VP8, H264
  • Multiplatform: Browser, Mobile

11 of 27

WebRTC

  • NAT Traversal by ICE.
    • Find a public address for direct communication.
    • Works as “Whatismyip.com”.
  • Session Description Protocol

12 of 27

Video Compression

  • VP8, H264
  • Omit non-essential bits of information.
  • Transform for better analysis, hence can omit the most bits (YUV)
  • The frames are inferred from previous and future frames. Only the difference is transmitted.

13 of 27

Audio Compression

  • Omit data in audio that cannot be perceived by human
  • Opus is designed to transport over Real-Time Transport Protocol
  • Opus encoding has very low delay (5 ~ 66.5ms)

14 of 27

Pion, WebRTC library for Go

  • https://github.com/pion/webrtc
  • WebRTC in Native Golang:
    • Not fromCGO
  • Include WebRTC Features:
    • Sub-second latency communication
    • Simple API
  • Web Assembly
    • One Go code deployed on Backend and frontend
  • Versioning with GoModule
    • Different set protocol can be selected
  • QUIC
  • Lively community pion.ly/slack about WebRTC

My project got featured in Pion talk in GopherCon 2019

15 of 27

Pion, WebRTC library for Go Example

// Setup peer connection

api := webrtc.NewAPI(webrtc.WithMediaEngine(mediaEngine))

peerConnection, err := api.NewPeerConnection()

// Create track and add track to connection

videoTrack, err := peerConnection.NewTrack(payloadType, rand.Uint32(), "video", "pion")

peerConnection.AddTrack(videoTrack);

// Write sample to the track

videoTrack.WriteSample(media.Sample{Data: frame, Samples: 90000});

16 of 27

Write Cloud-Gaming in Go

How golang helps my interation is that fast

17 of 27

Architecture

  • Network latency matter the most
    • Infrastructure enables finding closest streaming server to user.
  • Coordinator
    • Pairing + Setup peerconnection between most suitable server and user
  • Worker (Stream server)
    • Running Games + Stream to user
  • Horizontal scaling by adding more worker

18 of 27

Go channel in action

  • GoRoutine and Channel is designed to simplify streaming pipeline and concurrency problems.
  • All components run separately, Each component manages its own state.
  • Components communicate over channel. The Go way: No shared state, No lock.

19 of 27

WebRTC Transmitter (user facing) | Game Emulator | Video Encoder | Audio Encoder | Connection Handler

20 of 27

2. Fan-in, Fan-out

  • Collaborative play: Multiple players play the same game together. (Twitch play pokemon)
  • Golang fan-in fan-out pattern matches this usecase perfectly

21 of 27

2. Fan-in, Fan-out

  • Fan-in input from users, Fan-out media to them.

func (r *Room) listen() {

for _, webRTC := range r.rtcSessions {

go func(webRTC *){

for input := range webRTC.InputChannel {

r.inputChannel <- input

}

}(webRTC)

}

}

// Other side of r.inputChannel update Game state

func (r *Room) broadcast() {

for image := range r.outputChannel {

for _, webRTC := range r.rtcSessions {

webRTC.ImageChannel <- data

}

}

}

// Other side of webRTC.outputChannel send new copy of media

Fan-in game input

Fan-out media stream

22 of 27

3. Handling concurrency

  • Input from WebRTC/user changes game state in Emulator.
  • Save from WebRTC/user requires an unspoiled snapshot in Emulator.
  • Use select statement between 2 events => only one event can run in a time.

func (w *webRTC) save() {

w.event <- struct{}

}

func (w *webRTC) update(key int) {

W.input <- key

}

func (e *gameEmulator) gameUpdate() {

for {

select {

case <-e.save:

save(e.snapshotState())

case key := <-e.input:

e.updateState(key)

case <-e.done:

e.close()

return

}

}

}

func (w *webRTC) save() {

w.room.save()

}

func (w *webRTC) update(key int) {

w.room.update(key)

}

func (r *Room) save() {

save(r.gameEmulator.snapshotState())

}

func (r *Room) update(key int) {

r.gameEmulator.updateState(key)

}

Direct call

Over channel

23 of 27

Golang makes it hard

  • Channel can cause slow
    • Select statement is costly
    • Comparing with lock, Channel is a simpler way to deal with concurrency and event, not the faster way.
  • Try to avoid using channel in hotspot. Avoid selecting too many events.

24 of 27

Golang makes it hard

  • Channel headache.
    • You forgot close Channel. It leaks!
    • Channel belong to both side, so which is the owner of channel?
  • Codec libraries and emulators are written in C.
    • “CGO is not GO”!
    • Profiling shows nothing but CGO.

25 of 27

Profiling can cure your headache

  • GoRoutine Profile:
    • Check if code is stuck at what line
  • Memory Profile
    • Image frames kept allocs.
  • Trace Profile
    • See if all cores are fully utilized.

26 of 27

Future Improvement

  • Encoding pipeline in GPU.
    • Image Encoding, Audio Encoding running on CPU is very costly, current bottleneck
  • Making an interface for not only game but can be anything.
    • For example: Window or Photoshop
  • Decentralize, Sharing platform for application
    • Some suppliers or distributors can host the game or application on their own hosts.
    • Users can run the application directly from any hosts.
    • CloudRetro plays as sharing platform.

27 of 27

  • sergystepanov for Audio re-implementation; bilinear, nearest neighbor interpolation scaling; Window setup document; Frontend refactoring
  • sadlil for massive backend code structure reogranization; log and monitor server introduction.
  • Pion Webrtc team for the incredible Golang Webrtc library and their supports.
  • libretro/kivutar Golang libretro and https://www.libretro.com/.