Building high performance MO Game server
2016.04 GoCon�By @methane
About me
Puzzle Wonderland
https://www.puzlan.com/�https://www.youtube.com/watch?v=PLru7j4Q-1A
Turn based multiplay.
Dozens messages per second.
Backend
Game Server
Game Server
Game Server
Game Server
Lobby Server
Lobby Server
Lobby Server
MySQL
Websocket
ELB
http
Game Server
Lobby Server
Sorry, it's not an OSS
I chose development speed and destructive changes.
Today's topic
Patterns and practices to build high performance network server
BTW, we use gb too
Pros:
Not require go15vendor. (Better tools support)
Cons:
Can't use quickfix (See gb#406)
Game Server
The Go Programming Language
Pragmatic
Especially, Exercises are very pragmatic
Japanese version may be available
GOPL 8.10 Example: Chat Server
https://github.com/adonovan/gopl.io/tree/master/ch8/chat
Modified version for this talk
Step 1. Using buffered channel
Unbuffered channel is blocking when sending or receiving.
When there are many rooms, server can scale for CPU cores. But higher goroutine switching rate has significant performance penalty.
Problem: Nondeterministic order
chat-original/chat.go
broadcaster():
select {
case msg := <-messages:
for cli := range clients {
cli <- msg
}
case cli := <-entering:
clients[cli] = true
handleConn():
messages <- who + " has arrived"
entering <- ch
Solution: Command pattern
chat-step1/chat.go
broadcaster():
for cmd := range cmdQueue {
switch cmd := cmd.(type) {
case cmdEnter:
clients[cmd.c] = true
case cmdMessage:
for cli := range clients {
cli <- cmd.m
}
handleConn():
cmdQueue <- cmdMessage{who + " has arrived"}
cmdQueue <- cmdEnter{ch}
Benchmark
c4.8xlarge, 1room, 100 clients, 1000 msg/clients
before: 39sec
step1: 19.8sec
Step 2. Buffering I/O
I/O system call may be most heavy part of network server.
Read buffering is easier than Write buffering since Write buffering needs flushing.
Larger chunk size when high load. It makes easy to take a balance of latency and throughput.
GOPL chat server
Reader: Buffered already
input := bufio.NewScanner(conn)
for input.Scan() {
messages <- who + ": " + input.Text()
}
Writer: Not buffered
for msg := range ch {
fmt.Fprintln(conn, msg) // NOTE: ignoring network errors
}
chat-step2/chat.go
// restrict concurrent write
var wsem = make(chan struct{}, 2)
func clientWriter(conn net.Conn, ch <-chan string) {
var c chan struct{}
buf := bytes.Buffer{}
defer conn.Close()
for {
select {
case msg, ok := <-ch:
if !ok {
buf.WriteTo(conn)
return
}
fmt.Fprintln(&buf, msg)
c = wsem
case c <- struct{}{}:
_, err := buf.WriteTo(conn)
<-c
if err != nil {
return
}
if buf.Len() == 0 {
c = nil
}
}
Benchmark step2
before: 39sec
step1: 19.8sec
step2: 4.1sec
Lower packet/sec, frames/sec
Step 3. Coping with slow clients
When socket send buffer is full, write may block for a long time. It may affects to other clients.
Solution
Very difficult. I haven't establish "pattern".
Sorry, no sample code...
Other practices and patterns
Use pprof, especially goroutine profile
CPU profile is useful only when CPU usage is problem.
Goroutine profile (debug=1) is useful in many other cases.
Further reading about pprof
I wrote articles in last year's advent calendar. (Sorry, written in Japanese)
http://klabgames.tech.blog.jp.klab.com/archives/pprof1-cpuprofile.html
http://klabgames.tech.blog.jp.klab.com/archives/pprof2-cpuprofile.html
Zero time cache pattern
// N concurrent request : 1 concurrent query
// No consistency issue caused by cache lifetime
// This pattern is used to list rooms waiting new members.
now := time.Now()
c.Lock(); defer c.Unlock()
if now < last {
last = time.Now()
cached = query()
}
return cached
// see also cachedquery pkg in sample code repository
Thank you
INADA Naoki
@methane
KLab Inc.