1 of 40

Refurbishing the Meyrin Data Centre: Storage Juggling and Operations

Presented by Octavian-Mihai Matei on behalf of the EOS team

EOS WORKSHOP 2025

@

2 of 40

Situation -> Problem -> Plan

2

EOS 2025 Workshop - TechWeekStorage25

3 of 40

The situation

Why are we even talking about this refurbishment?

3

EOS 2025 Workshop - TechWeekStorage25

4 of 40

The situation - cont

  • Over 400 storage nodes with different architectures, components etc
  • 6 production instances and 3 pre production instances
  • All on AlmaLinux 9.5
  • Running 6 versions of EOS (MGMs + FSTs)
    • 5.2.24-1, 5.3.2-1, 5.3.4-1, 5.3.6-1, 5.3.7-1, 5.3.8-1
  • Around 170 PB raw storage to drain
  • 55 PB new pledge logical for experiments
    • In computations we consider replica 2, so we need 110 PB raw for experiments
  • 5B+ files to be moved

​

A lot of work to be done

4

EOS 2025 Workshop - TechWeekStorage25

5 of 40

The problem

  • Old machines
    • Quanta nodes, 10Gbps NICS, 6TB drive etc
  • Decommissioning timelines
    • 2 zones in March 2025
    • 3 zones in July 2025
    • 2 zones in October 2025
  • Capacity pledges to be maintained on �the instances
  • Different instances, �different usage patterns

​

5

EK

CF

EOS 2025 Workshop - TechWeekStorage25

6 of 40

The plan

  1. Test 4/7 production instances to establish a baseline
  2. See what zone we can remove first
  3. See what machines we can drain first
  4. Drain them
  5. Repeat from 2. for multiple racks
  6. Replace old hardware with the new one for instances

6

EOS 2025 Workshop - TechWeekStorage25

7 of 40

The plan was a dream, this is what happened

  1. Start testing
  2. Stop drainings because high usage by users
  3. Continue testing
  4. Stop drainings because high usage by users
  5. Continue testing
  6. See what zone we can remove first
  7. See what machines we can drain first
  8. Drain
  9. Stop drainings because high usage by users
  10. Drain
  11. 🎉Holidays🎉
  12. Stop drainings because high usage by users
  13. Drain
  14. NOT ENOUGH SPACE!
  15. Replace old hardware with pilot machines
  16. Drain

…………………….A few problems later

  1. Drain
  2. Drain
  3. 🎉New hardware came🎉
  4. Distribute hardware to instances
  5. DRAIN
  6. FINISH?

7

EOS 2025 Workshop - TechWeekStorage25

8 of 40

Timeline

8

14/01 Testing Begins

23/01 Testing Ends

6/02 EK Draining Begins

19/02 EK Draining Ends

10/03 More Hardware

18/03 CF Draining Begins

Soonish?

EK & CF done

EOS 2025 Workshop - TechWeekStorage25

9 of 40

Per instance distribution of zones

9

0 PB

5 PB

10 PB

15 PB

20 PB

25 PB

30 PB

35 PB

EOS 2025 Workshop - TechWeekStorage25

10 of 40

Testing

10

EOS 2025 Workshop - TechWeekStorage25

11 of 40

Testing methodology

  • Take 48 filesystems (size of an old node) => 8 filesystems/machine, 6 machines
  • 288 TB per cluster
  • Tweak the draining configuration
    • Drainer.fs.nts
    • Drainer.node.nfs
    • Drainer.node.ntx
    • Drainer.node.rate
  • Record how long it takes

11

EOS 2025 Workshop - TechWeekStorage25

12 of 40

Testing EOS CMS�

Max 39.5 GB/s Mean 15.5 GB/s

Time to drain - 5 Hrs/node

12

EOS 2025 Workshop - TechWeekStorage25

​

13 of 40

Testing EOS ATLAS�

Max 27.4 GB/s Mean 15.8 GB/s

Time to drain - 4.5 Hrs/node

13

EOS 2025 Workshop - TechWeekStorage25

​

14 of 40

Testing EOS PUBLIC�

Max 32.7 GB/s Mean 13.8 GB/s

Time to drain - 5 Hrs/node

14

EOS 2025 Workshop - TechWeekStorage25

​

15 of 40

Testing EOS LHCb�

Max 7.87 GB/s Mean 2.72 GB/s

Time to drain - 1 day/node

15

EOS 2025 Workshop - TechWeekStorage25

​

16 of 40

Draining estimates

16

EOS 2025 Workshop - TechWeekStorage25

​

17 of 40

Actual Draining

17

EOS 2025 Workshop - TechWeekStorage25

18 of 40

PROBLEM - We need space before we can drain

We need to take 9 machines from EOS PILOT with 1.7PB/node to meet the pledges of the experiments:

  • To LHCB - 1 nodes (1.7PB)
  • To ALICE - 5 nodes (8.5PB)
  • To ATLAS - 1 node (1.7PB)
  • To CMS - 1 node (1.7PB)
  • To Public - 1 node (1.7PB)

Plan:

  1. drain EK zone as it has older machines and less capacity
  2. drain CF zone

18

where

  • bold = space is covered by the new nodes
  • underline = it is partially covered

EOS 2025 Workshop - TechWeekStorage25

​

19 of 40

EK racks EOS Atlas�

Max 48.9 GB/s Mean 17.5 GB/s

Estimated to take 2 days, took only 1 day

19

EOS 2025 Workshop - TechWeekStorage25

​

20 of 40

EK racks EOS CMS�

Max 117 GB/s Mean 75.3 GB/s

20

EOS 2025 Workshop - TechWeekStorage25

​

21 of 40

EK racks EOS Public�

Split over multiple days, as the instance was used by users

Max 64.7 GB/s

21

EOS 2025 Workshop - TechWeekStorage25

​

22 of 40

Post EK draining problem

  • At the end of EK draining campaign, we had to stop and could not complete CF, because we had no physical hardware available anymore.
  • In danger not to keep the pledge capacity to the experiments.
  • 4 months delay in hardware arrival.

22

EOS 2025 Workshop - TechWeekStorage25

​

23 of 40

The Juggling part - New Hardware

On 10th March, we received the new capacity, 160PB, which was split between

  • New pledge capacities
  • Replacement for CF
  • Future decommissioning

​

23

Experiment

Meet pledge + CF zone takeover

[nr nodes]

Decommissioning CF

[nr nodes]

ALICE

10

25

ATLAS

6

9

CMS

13

9

LHCb

13

0

PUBLIC

2

6

EOS 2025 Workshop - TechWeekStorage25

​

24 of 40

The Juggling part - New Hardware

On 20th March 2025, we crossed 1 Exabyte mark on our instances in terms of CAPACITY

24

EOS 2025 Workshop - TechWeekStorage25

25 of 40

Move new FSes in EOS Public to default group

25

EOS 2025 Workshop - TechWeekStorage25

​

26 of 40

Move new FSes in EOS Alice to default group

26

EOS 2025 Workshop - TechWeekStorage25

​

27 of 40

Move new FSes in EOS CMS to default group

​

27

EOS 2025 Workshop - TechWeekStorage25

​

28 of 40

CF racks decommissioning

Finished:

  • 9 storage servers in CMS
  • 6 storage servers in Public

Ongoing:

  • 25 storage servers in Alice
  • 9 storage servers in Atlas

28

EOS 2025 Workshop - TechWeekStorage25

​

29 of 40

CF racks EOS PUBLIC�

Max 53 GB/s

29

EOS 2025 Workshop - TechWeekStorage25

​

30 of 40

CF racks EOS CMS�

Max 45.4 GB/s

30

EOS 2025 Workshop - TechWeekStorage25

​

31 of 40

CF racks EOS ATLAS - ONGOING�

Max 14.1 GB/s

31

EOS 2025 Workshop - TechWeekStorage25

​

32 of 40

CF racks EOS ALICE - ONGOING�

Max 12.3 GB/s - Limited bandwidth due to very high usage by end users

32

EOS 2025 Workshop - TechWeekStorage25

​

33 of 40

CF racks EOS ALICE�

33

EOS 2025 Workshop - TechWeekStorage25

​

34 of 40

CF Decommissioning - To be finalized by the end of the month

34

EOS 2025 Workshop - TechWeekStorage25

35 of 40

Conclusions

35

EOS 2025 Workshop - TechWeekStorage25

36 of 40

Storage capacity

At the end, we massively increased the storage capacities of all our clusters:

36

EOS 2025 Workshop - TechWeekStorage25

37 of 40

What have we learned?

​

​

  • Decommissioning activity needs to be planned much ahead of time
    • This first round was very short notice to us
  • Installing brand new server models takes time
  • Draining needs to be balanced with end user activities
  • Thanks to FSCK for fixing leftovers (See the talk)
  • Juggling hundreds of PB of production capacity is more difficult than it sounds

37

EOS 2025 Workshop - TechWeekStorage25

38 of 40

home.cern

39 of 40

Testing EOS ATLAS 2 - MORE INFO�

39

Draining speed

EOS 2025 Workshop - TechWeekStorage25

​

40 of 40

Testing EOS CMS 2 - MORE INFO�

40

Draining speed

EOS 2025 Workshop - TechWeekStorage25

​