1 of 95

HORDE BUILD SYSTEM

2 of 95

QUICK INTRO

Regner Blok-Andersen
Splash Damage
Managed Horde at Splash

3 of 95

QUICK INTRO

Regner Blok-Andersen
Splash Damage
Managed Horde at Splash

4 of 95

THE PLAN

What is Horde
How we transitioned
Things to consider

5 of 95

BUILD SYSTEMS

6 of 95

BUILD SYSTEMS

What is a build system

7 of 95

BUILD SYSTEMS

Fundamentally, all build systems have a straightforward purpose: they transform the source code written by engineers into executable binaries that can be read by machines.

8 of 95

BUILD SYSTEMS

What is a build system
Produces a thing players can play from Perforce depot

9 of 95

BUILD SYSTEMS

What is a build system
Produces a thing players can play from Perforce depot
Jenkins, TeamCity, GitHub Actions

10 of 95

BUILD SYSTEMS

What is a build system
Produces a thing players can play from Perforce depot
Jenkins, TeamCity, GitHub Actions
Extras:

11 of 95

BUILD SYSTEMS

What is a build system
Produces a thing players can play from Perforce depot
Jenkins, TeamCity, GitHub Actions
Extras:

Fast

12 of 95

BUILD SYSTEMS

What is a build system
Produces a thing players can play from Perforce depot
Jenkins, TeamCity, GitHub Actions
Extras:

Fast
Automated

13 of 95

BUILD SYSTEMS

What is a build system
Produces a thing players can play from Perforce depot
Jenkins, TeamCity, GitHub Actions
Extras:

Fast
Automated
Informative

14 of 95

DISCLAIMER FIRST

15 of 95

16 of 95

WHAT IS HORDE

17 of 95

18 of 95

19 of 95

20 of 95

21 of 95

22 of 95

SPECIAL SAUCE

23 of 95

SPECIAL SAUCE

Modern application

24 of 95

SPECIAL SAUCE

Modern application
Perforce as first-class citizen

25 of 95

SPECIAL SAUCE

Modern application
Perforce as first-class citizen
Integrated with the Engine

Getting any generic build system to build your Unreal project isn’t that hard, assuming you understand all the commandlets you need to call. Horde doesn’t change that. You still need to understand those things. You need to know how to build an Unreal project.

It’s all the other things you want from a build system.

With most build systems you have the build system, and you have your project. They are separate things, very separate things, and you want to tell them how to work together. You must make the build system understand the projects dependencies, what files that are generated from step 3 are needed on step 7, you must make the build system understand warnings vs errors, or how to process test results.

With Horde and its integration into BuildGraph and the overall Unreal build process these things are just sort of taken care of.

26 of 95

SPECIAL SAUCE

Modern application
Perforce as first-class citizen
Integrated with the Engine

27 of 95

SPECIAL SAUCE

Modern application
Perforce as first-class citizen
Integrated with the Engine
The developer experience

28 of 95

HORDE TOUR

29 of 95

30 of 95

31 of 95

32 of 95

33 of 95

0

34 of 95

35 of 95

Below the categories we can see all the jobs that have been triggered in this category.

This is where we get one of the first things that really helps Horde stand out. Many other build systems that I have used will only show you the overall build status for each job. If you want more information, maybe to understand what was running in that job or what failed you must drill further into it.

Here we are presented with all the information we need to understand what the job was doing, in this case we can see it ran compiles, cooks, and some tests for multiple platforms.

We can also tell that the compiles were so close to being error and warning free. That block of red is because those jobs had errors.

We can also tell that this project currently has some quantity of warnings present in all the cooks and tests as they are yellow.

36 of 95

37 of 95

38 of 95

39 of 95

40 of 95

41 of 95

42 of 95

43 of 95

44 of 95

45 of 95

46 of 95

47 of 95

48 of 95

49 of 95

50 of 95

51 of 95

52 of 95

53 of 95

54 of 95

l

55 of 95

56 of 95

57 of 95

58 of 95

59 of 95

THINGS LEARNED

60 of 95

THINGS LEARNED

Horde build “simply” runs BuildGraph scripts in a distributed fashion.

61 of 95

THINGS LEARNED

Horde runs BuildGraph

62 of 95

THINGS LEARNED

Horde runs BuildGraph

The common

63 of 95

THINGS LEARNED

Horde runs BuildGraph

The common
Lack of IDE

64 of 95

65 of 95

THINGS LEARNED

Horde runs BuildGraph
Start small

66 of 95

THINGS LEARNED

Horde runs BuildGraph
Start small
Components of Horde

67 of 95

SERVER

68 of 95

SERVER

DASHBOARD

69 of 95

SERVER

DASHBOARD

PROXY

70 of 95

SERVER

DASHBOARD

PROXY

USER

71 of 95

SERVER

DASHBOARD

PROXY

USER

AGENT

The Horde agent communicates with the server backend via a gRPC API.

The agent is the thing that does the actual heavy lifting of running BuildGraph scripts. As with most other build systems agents can be grouped into pools which allow you to run different types of jobs on different types of agents, or to split resources between teams.

We use Ansible to deploy and configure the agents along with the rest of our infrastructure. What’s nice here is that Horde understands AutoSDK. When we configure our agents with Ansible all we install is some monitoring software, the horde agent software, and a few utilities such as Perforce, VSCode, and WinDirStat. That’s it. Horde makes sure that machines which need the AutoSDK have it synced from Perforce, it makes sure the correct environment variable is set, and then the engine takes care of installing all the SDKs it needs.

72 of 95

SERVER

DASHBOARD

PROXY

USER

AGENT

WORKER

73 of 95

SERVER

DASHBOARD

PROXY

USER

AGENT

WORKER

MONGODB

74 of 95

SERVER

DASHBOARD

PROXY

USER

AGENT

WORKER

MONGODB

REDIS

75 of 95

SERVER

DASHBOARD

PROXY

USER

AGENT

WORKER

MONGODB

REDIS

STORAGE

76 of 95

SERVER

DASHBOARD

PROXY

USER

AGENT

WORKER

MONGODB

REDIS

STORAGE

77 of 95

BARE METAL

78 of 95

THINGS LEARNED

Horde runs BuildGraph
Start small
Components of Horde

79 of 95

THINGS LEARNED

Horde runs BuildGraph
Start small
Components of Horde
TCP connections

This was a “fun” one to learn. I have no idea why Epic doesn’t run into this problem, but for us with the default settings in Windows Server 2022 the number of available TCP ports was a problem.

Essentially the Horde agents were unable to sync any of our workspaces as they would run out of available TCP ports and just fall over.

I won’t go into all the details, but behind the scenes Horde attempts to improve upon the default parallel sync behaviour in the P4 CLI and the result is that it churns through many thousands of TCP connections to the Perforce server.

Once we understood the problem the solution was simple. We changed two registry keys. Specifically, we decreased TcpTimedWaitDelay from 120 seconds to 60 seconds and we increased MaxUserPort from 15k to 30k. Basically quadrupling the number of available TCP ports on our Horde agents.

80 of 95

81 of 95

Disclaimer

82 of 95

THE DRAGONS

83 of 95

THE DRAGONS

There are no releases

84 of 95

THE DRAGONS

There are no releases
Lack of retention mechanisms

85 of 95

THE DRAGONS

There are no releases
Lack of retention mechanisms
The Epic way, or build it yourself way

86 of 95

87 of 95

THE DRAGONS

There are no releases
Lack of retention mechanisms
The Epic way, or build it yourself way
No visibility of server wide queues

88 of 95

THE DRAGONS

There are no releases
Lack of retention mechanisms
The Epic way, or build it yourself way
No visibility of server wide queues
No way to run specific job on all agents

89 of 95

THE DRAGONS

There are no releases
Lack of retention mechanisms
The Epic way, or build it yourself way
No visibility of server wide queues
No way to run specific job on all agents
No easy dashboards

90 of 95

p

91 of 95

THE DRAGONS

There are no releases
Lack of retention mechanisms
The Epic way, or build it yourself way
No visibility of server wide queues
No way to run specific job on all agents
The list goes on…

92 of 95

THANKS FOR HANGING

93 of 95

THANKS FOR HANGING

Regner Blok-Andersen

@regnerba

94 of 95

THANKS FOR HANGING

Regner Blok-Andersen

@regnerba

careers.splashdamage.com

95 of 95

THANKS FOR HANGING

tinyurl.com/developehorde

Regner Blok-Andersen

@regnerba

careers.splashdamage.com