1 of 5

LIQUID COOLING AT NASA AMES

Chris Tanner

Facility Manager

NAS Advanced Supercomputing

NASA Ames Research Center

1

5/3/23

2 of 5

NASA Ames Supercomputing Facility

  • Building N258, a traditional legacy data center, built in 1986. Home to Pleiades & Cabeus Computers,PUE: 1.36.
    • 4MW IT equipment, approximately 300 racks.
    • 1MW Cooling System (3+1 Chillers & Cooling Tower). Raised floor cooling with rear-door heat exchangers on compute racks.

​

  • R&D088, Prototype Modular Data Center Facility, built in 2016 & 2017. Home to Electra Computer. PUE: 1.04.
    • 1.4 MW - 18 air cooled racks & 16 HPE Apollo 8600 (HPE E-Cells).
    • MDC1 uses filtered outside air & evaporative cooling over 81o F.
    • MDC2 uses water cooled heatsinks on processors – process water cooled by dry air and evaporative cooler to 80o F.

​

  • R&D099, Modular Supercomputing Facility, built in 2019. Home to Aitken Computer. PUE: 1.044.
    • 2.1 MW - 8 Apollo 8600 & 16 HPE Apollo 9000
    • Water cooled heatsinks on processors plus Apollo 9000 has water cooled circuit boards, eliminating fans – process water cooled by dry and evaporative cooler to 90o F.

​

2

5/3/23

3 of 5

NASA Ames Liquid Cooling Technology Changes

with Compute Technology

  • Our Implementation of Liquid Cooling is driven by the compute and vendor’s technology. While air cooling is the easiest to implement, it’s not always the best.
  • The N258 Main Data Center has implemented Liquid Cooling since the late 1980’s with the installation of the Fluorinert cooled Cray 2 computer.
  • As Cluster Computing took over in the mid-2000’s, we transitioned to Rear-Door Heat Exchangers (Cooling Doors).
    • As rack power started to increase above 15kW, our hot air containment was inadequate to maintain the proper rack inlet temperature.
    • Cooling doors closely coupled the cooling water/coils to the heat source and eliminated hot air spill over.
      • Our hot aisle went from 4 feet to 4 inches.
  • In 2017, we installed our first Direct-to-Chip Liquid Cooling system in a Modular Data Center.
    • In addition, the system is Warm Water Cooled.
      • The Primary Water loop is cooled by an Adiabatic Cooler to 80F.
      • The Secondary Water Loop that flows through the Cold Plate mounted on top of the Processor has a 85F supply temperature.
      • Other electrical components (memory, power supplies, etc.) are closed-loop, force-air cooled with the circulating air cooled at an air-to-liquid heat exchanger cooled by the primary loop (80F supply).
  • In 2020, we installed a Liquid-Only Cooled System; NO Fans!
    • The Primary Water loop supply temperature is 90F and the Secondary Water Loop supply temperature is 95F.
    • All electrical components are cooled by the cold plate (95F supply)

3

5/3/23

4 of 5

Our Most Recent Liquid Cooling Installation

  • Installed on October 31, 2023, a 4 rack Liquid-Cooled Compute System in a room previously used for air-cooled systems.
    • Rack power is 108kW per rack.
  • While the room once was all raised floor, half the floor was removed 10 years ago to site a Quantum Computer. With the Quantum Computer decommissioned two years ago, the new system was sited on the concrete subfloor.
    • Liquid-cooled systems are heavy and our data center’s raised floor was not rated for the weight of the new system.
  • Chilled water from existing 3-inch piping previously used by long-removed Computer Room Air Handlers was plumbed from under the leftover raised floor into the compute system’s Cooling Distribution Unit.
    • The chilled water supply temperature is 50F at 150 GPM.
  • Chilled Water Piping runs on the floor while Power and Network are distributed from above.
  • Two 4-Ton mini split air conditioners cool the air-cooled I/O rack.
  • This is a temporary install as the compute system will be moved in about 6 months to our coming new Modular Data Center, where it will have a 90F supply temperature.

4

5/3/23

5 of 5

Benefits & Challenges to Liquid Cooling

Benefits

  • Allows for compute density, shorter interconnects, faster speeds.
  • Higher Density allows for less floor space: our 4-rack Liquid Cooled system has 65,000 cores, the equivalent of 32 racks with cooling doors on our main computer floor.
  • Liquid Cooling is more energy efficient than air cooling. Our Warm-water, Liquid Cooled systems have a PUE of 1.04, while our chilled water systems are at 1.36.
  • Liquid cooling allows high power racks to coexist with low power racks in data centers with less than perfect containment.

Challenges

  • Adding new piping to the subfloor requires available subfloor room. The photo to the right is the piping manifold and valves filling our entire 24-inch subfloor.
  • As rack power increases, flow rates and piping sizes increase too. Large, water filled pipes are heavy, requiring significant support.
  • The subfloor needs to be rated for the addition loading of liquid cooling equipment. The cooling door shown weighs 400 lbs.
  • Liquid-cooled doors take up aisle space. Our typical 4 foot hot aisle shrank to 18 inches when the doors were installed. Racks need to be spread apart to account for door thickness.
  • Water leaks typically occur in the underfloor brass shutoff valve bodies, but the occurence is infrequent.

5

5/3/23