Realm Royale Server Performance

Since Realm Royale has been released on Steam, we have had engineers working on server performance.  The lag and hitching in the game was not acceptable, and we had to cap players at 85 as a partial stopgap measure.  While we continue to work on further improvements, the 0.4 patch on July 2 has made significant progress and has eliminated the vast majority of hitching.

Background

Our servers are set to run at an ideal frame rate of 24 Hz, which we have found provides smooth gameplay.  From the start, Realm only used server hardware with very late model powerful CPUs to help us reach this goal.  

Hirez has done extensive modifications to our copy of the Unreal Engine since 2010 to get better performance. The work involved has included: multithreading parts of the networking to better utilize CPU capacity, large physics updates, optimizing the approach for relevancy checks, and optimizing replication.

These legacy modifications were built primarily to optimize high-action multiplayer matches involving less than 32 players in a match, as that fit the needs of our previous games.  

As we began work late last year to build a 100-player Battle Royale game, we realized significant additional optimizations would be needed.  Game networking does not scale linearly with players, meaning that going from 10 to 100 players is not 10x the CPU cost, is is more like 30-40x the cost.  This effort was centered around more multi-threading and continued relevancy optimizations.

Improvements in the 0.4 patch

Although the performance improvement has been a continual process, we made some major changes in 0.4 that have yielded a significant benefit.  We reworked our relevancy and line check process for projectiles, since they are a big part of the Realm Royale gameplay but had significant CPU cost.  We also made several processing power vs memory tradeoffs. Our servers are bound by processing time, not memory. One example tradeoff was reducing the cleanup done on items that have left the match, such as loot boxes destroyed by the fog.  It is better to leave that memory used until the match is over and not use the cleanup processor time.  We also ran some very deep profiling on some live matches to find gameplay areas that were taking extensive time, and optimized those code paths.

We are experimenting now in 0.4 with match kick off sizes of 90 and 100. We are also continue to investigate additional potential server improvements.

Representative Data

0.3 Version

0.4 Version

Server tick rate at early match

15-20 FPS

23-24 FPS

Very small hitches (likely not noticeable)

Average 71 / match

Average 40 / match

Significant hitches

Average 5 / match

Average 1 / match

Horrible hitches

Average 3 / match

Average 0.25 / match

Graph

The graph below shows a partial sample of the server tick times for representative matches.  The solid line at 41 msec is the target time when there are not problems.  You can see in the blue data in a typical 0.3 match the early minutes of the game had constant lag and some giant hitches.  The red data (0.4) shows the big improvements. The periodic 30 second small hitches are garbage collection (part of the Unreal engine) and is something we are continuing to optimize.