MPCDF Raven�Direct Liquid Cooling

Karsten Kutzer

2022 Lenovo internal.

How and why adopt liquid cooling?

• How to adopt liquid cooling?
• Different technologies available – e.g. Direct-Water-Cooling (DWC) and Immersion Cooling
• Different implementations of DWC are available, e.g.
• DWC assemblies by specialized providers added to systems or vendor designed and integrated DWC systems
• Loop materials and fusing (e.g. fused plastic/EPDM vs. welded copper)
• Components on cooling loops (e.g. CPU only vs. all components up to the PSU)
• Direct-water-cooling with welded copper loops is a mature technology for a decade now in large HPC systems

​

• Why to adopt liquid cooling?
• Energy savings on the cooling
• at system level (eliminating fans and air cooled power supplies)
• at data center level (allowing “free cooling” concepts with dry or hybrid recoolers)
• Increasing cooling efficiency, supporting higher TDPs / lower Tcase of latest CPUs and GPUs
• Decreasing the carbon footprint of the data center
• reducing data center consumption of electricity
• opening a commercial viable option to re-use heat energy

​

Story: Max Planck Computing and Data Facility (MPCDF)

• Max Planck Computing and Data Facility (MPCDF)
• Is a cross-institutional competence centre of the Max Planck Society in Germany
• supports computational and data sciences by running HPC systems in their own data center
• updates HPC system to next generation technology at regular procurement intervals

​

• Main goals that drove the decision for introducing direct-water-cooling at MPCDF
• Reducing of the overall operational energy cost of the system
• Minimizing/Eliminating the heat energy going to air in the data center

​

• “Raven” – a direct-water-cooled HPC system at MPCDF (installed 2021)
• All compute nodes are direct-water-cooled
• 1592 dual-socket compute nodes with Intel Xeon processors
• 192 compute nodes with additional GPU acceleration (4 Nvidia A100 GPUs)
• Two Coolant Distribution Units (CDUs) with two separate Tichelmann Loops
• Rear Door Heat Exchangers (RDHX) on the racks remove remaining heat
• 100% water cooled system

​

3

Roadblock and/or key learning from “Raven”

• Operation of the Coolant Distribution Unit (CDU)
• CDUs are part of the system managed by the admins, but the facility management specialists have the skills in working with water
• Key learning: Early involvement of system administrators and facility management specialists is important to clarify organizational questions

​

• Missing secondary cooling loop implementation skills on supplier side
• Special considerations on the materials being used (or to be avoided – e.g not using hemp for sealing connections between pipes and valves)
• Key learning: Tight oversight of supplier work even in small details is important to achieve the quality that is important to operate the system flawlessly

​

• Limited communication during Covid pandemic lead to issues
• Onsite meetings and inspections were not possible during the implementation phase, resulting in misunderstandings and issues
• inlet/outlet water connections from secondary loop to CDU were confused initially, causing bad cooling behaviour
• Primary side pumps were not pressure regulated initially, causing high pressure with the CDU primary side 2-way valve regulating
• Key learning: onsite communication with all involved parties (suppliers, vendor, facility specialists) is very important for details

​

• Overall, the project was a great success
• Smaller challenges and issues were resolved quickly, after they have been identified
• Direct-water-cooling works as expected in operation, main goals for going to direct-water-cooling were achieved
• Next generation of the system will be direct-water-cooled as well

​

4