1 of 5

First Steps Toward Adopting Direct Liquid Cooled HPC BoF�Queen’s University Liquid Cooling

Nevil Silverius – Queen’s Center for Advanced Computing

Queen’s University

Kingston Ontario Canada

1

2 of 5

Background on Queen’s

  • Queen’s University Center for Advanced Computing – is located in Kingston, Ontario Canada (North of New York State)
  • Regional funded by (Compute Ontario) , center offers HPC systems, data services, training and consulting for research
  • Queen’s is one of Canada’s most research-intensive universities

CAC supports leading engineering, business and medical schools

  • CAC datacenter facility is located in a former industrial plant space – there is high power availability
  • Our DC is around 4400sqft, raised floor (18”), perimeter air cooled with 1.2MW utility feed.
  • We are a small team with staff of 22

​

3 of 5

Cooling Status and Upgrade

  • We are currently air cooled, undergoing upgrade of air cooling to prepare for liquid cooling
  • Replacing 4 Ecosaire EN 27 ton CRAC units on roof with Liebert DSE DA085 units
  • Goal of replacing existing air cooling to handle legacy equipment while preparing DSE units to support liquid install in phase 2
  • Still need to support researcher owned equipment that needs air cooling for foreseeable future
  • We are planning for liquid cooling install in stages, having air cooling for half of the floor, liquid for the other half
  • Rear door chillers likely approach for phase 2
  • We have floor loading issues, necessitating some floor reinforcement or replacement to handle liquid rack densities

4 of 5

Hybrid cooling challenges

We don’t own all the equipment, site hosts third party equipment

Replacing all at once approach won’t work

Looking at direct liquid cooling

Challenges with direct cooling are numerous, including cost of equipment for third parties

PI’s(Researchers) want dense racks, but can’t always afford them due to funding models

Many small buys ($250K) due to researcher funding model. Decisions up to individual Pis

There is wide gap in system installed capabilities, some DGX racks, some thin blade servers.

5 of 5

Open Questions for Phase 2

  • Water quality consultations
  • First liquid cooling install for center, still limited liquid cooling facilities in Canadian University sector
  • Rear door heat exchangers are current front running solution. Chilled water supply from roof top AC.
  • We haven’t decided on active versus passive rear door
  • Still working on plumbing plan, for now thinking of running water above racks – reworking in floor power/whips is cost prohibitive
  • Need good leakage detection solution, better with readers door than direct to component
  • Using liquid cooling guide from EEHPC-WG
  • Consulting with Dr. Ryan Grant – New faculty in from Sandia
  • We would love feedback and to hear about experiences from other hybrid sites who has already done liquid cooling implementation's