1 of 27

MetaBrainz Infrastructure

Recent changes and near future

1

|

2 of 27

Servers

29 physical servers

~14 virtual servers

~10 external servers

​

vSwitch feature let us interconnect all those servers

�Low cost, but we are very satisfied after 8 years working with Hetzner.

2

|

3 of 27

Important changes

  • Ansible to manage infrastructure

​

3

|

4 of 27

Important changes

  • Ansible to manage infrastructure
  • New gateways making use of Hetzner load balancer, horizontal scalability

​

4

|

5 of 27

Important changes

  • Ansible to manage infrastructure
  • New gateways making use of Hetzner load balancer, horizontal scalability
  • New backup solution based on Ansible Borgmatic role

​

5

|

6 of 27

Important changes

  • Ansible to manage infrastructure
  • New gateways making use of Hetzner load balancer, horizontal scalability
  • New backup solution based on Ansible Borgmatic role
  • Consul was upgraded, now running on virtual network, deployment was reworked

​

6

|

7 of 27

Important changes

  • Ansible to manage infrastructure
  • New gateways making use of Hetzner load balancer, horizontal scalability
  • New backup solution based on Ansible Borgmatic role
  • Consul was upgraded, now running on virtual network, deployment was reworked
  • New servers using ZFS

7

|

8 of 27

New gateways? Why?

​

  • MetaBrainz projects exponential growth
  • HTTPS/2 is now very common, it requires more resources than HTTP
  • kiki & herb were replaced by faster rex & rudi
  • horizontal scalability vs single entry point

​

​

8

|

9 of 27

9

|

10 of 27

New hardware

  • AMD Ryzen 9 7950X3D based servers

​

10

|

11 of 27

New hardware

  • AMD Ryzen 9 7950X3D based servers
  • pink & floyd are retiring, replaced by new jimmy & hendrix

​

11

|

12 of 27

New hardware

  • AMD Ryzen 9 7950X3D based servers
  • pink & floyd are retiring, replaced by new jimmy & hendrix
  • main database performance increased by a minimal factor of 4, constant cost

​

12

|

13 of 27

New hardware

  • AMD Ryzen 9 7950X3D based servers
  • pink & floyd are retiring, replaced by new jimmy & hendrix
  • main database performance increased by a minimal factor of 4, constant cost
  • ZFS along bigger SSDs provide far more disk space

​

13

|

14 of 27

Ansible

  • Ease the deployment of new servers, and maintenance of existing servers

​

14

|

15 of 27

Ansible

  • Ease the deployment of new servers, and maintenance of existing servers
  • Provide a cartography of the whole infrastructure

​

15

|

16 of 27

Ansible

  • Ease the deployment of new servers, and maintenance of existing servers
  • Provide a cartography of the whole infrastructure
  • Help to harmonize production environment

​

16

|

17 of 27

Ansible

  • Ease the deployment of new servers, and maintenance of existing servers
  • Provide a cartography of the whole infrastructure
  • Help to harmonize production environment
  • Ease upgrades to new major versions of base OS

​

17

|

18 of 27

Ansible

  • Ease the deployment of new servers, and maintenance of existing servers
  • Provide a cartography of the whole infrastructure
  • Help to harmonize production environment
  • Ease upgrades to new major versions of base OS
  • Shell scripts previously used (syswiki) are now obsolete
  • Tracking changes is easier, and tests help to maintain things

18

|

19 of 27

Ansible

  • Ease the deployment of new servers, and maintenance of existing servers
  • Provide a cartography of the whole infrastructure
  • Help to harmonize production environment
  • Ease upgrades to new major versions of base OS
  • Shell scripts previously used (syswiki) are now obsolete
  • Tracking changes is easier, and tests help to maintain things

19

|

20 of 27

Future

Near future goals

20

|

21 of 27

Notable goals for next year

  • Generalize the use of ZFS

​

21

|

22 of 27

Notable goals for next year

  • Generalize the use of ZFS
  • Complete the transition to 10.10.10.0 virtual network and slowly drop physical network interface on 10.2.2.0

​

22

|

23 of 27

Notable goals for next year

  • Generalize the use of ZFS
  • Complete the transition to 10.10.10.0 virtual network and slowly drop physical network interface on 10.2.2.0
  • Move docker-server-configs service startup scripts to Ansible

23

|

24 of 27

Notable goals for next year

  • Generalize the use of ZFS
  • Complete the transition to 10.10.10.0 virtual network and slowly drop physical network interface on 10.2.2.0
  • Move docker-server-configs service startup scripts to Ansible
  • Upgrade SOLR and deploy it via Ansible, likely moving the cluster to physical servers

​

24

|

25 of 27

Notable goals for next year

  • Generalize the use of ZFS
  • Complete the transition to 10.10.10.0 virtual network and slowly drop physical network interface on 10.2.2.0
  • Move docker-server-configs service startup scripts to Ansible
  • Upgrade SOLR and deploy it via Ansible, likely moving the cluster to physical servers
  • New improved Openresty setup, to consolidate LE certs related stuff (WIP)

​

25

|

26 of 27

Notable goals for next year

  • Generalize the use of ZFS
  • Complete the transition to 10.10.10.0 virtual network and slowly drop physical network interface on 10.2.2.0
  • Move docker-server-configs service startup scripts to Ansible
  • Upgrade SOLR and deploy it via Ansible, likely moving the cluster to physical servers
  • New improved Openresty setup, to consolidate LE certs related stuff (WIP)
  • Move to Ubuntu 22.04 (most servers are on 20.04)

26

|

27 of 27

​

Thank you.

​

​

Zas

27

|