1 of 28

A Partnership Framework for Scaling a Workforce of Research Cyber-professionals

James Wilgenbusch*, Joshua Baller**, Ben Lynch*

*Christy Henzler

*University of Minnesota, Research Computing: Minnesota Supercomputing Institute

**University of Virginia, Research Computing

Minnesota Supercomputing Institute

© 2020 Regents of the University of Minnesota. All rights reserved.

​

2 of 28

Draws Heavily from PEARC ‘22 Paper

Research Computing

© 2021 Regents of the University of Minnesota. All rights reserved.

​

3 of 28

Research Computing

Minnesota Supercomputing Institute

Systems

Informatics

U-Spatial

Outreach and Training

IIB International Institute for Biosensing

DSI Data Science Initiative

MiDB

LM&P

GEMS

ITN

Surgery

APG

Research Computing

© 2021 Regents of the University of Minnesota. All rights reserved.

​

4 of 28

Research Computing

Minnesota Supercomputing Institute

Systems

Informatics

U-Spatial

Outreach and Training

IIB International Institute for Biosensing

DSI Data Science Initiative

8

12

7

1

1

1

MiDB

LM&P

GEMS

ITN

Surgery

APG

Research Computing

© 2021 Regents of the University of Minnesota. All rights reserved.

​

5 of 28

Minnesota Supercomputing Institute (MSI)

Services

  • Self-service Access to Computing and Data Resources
  • In-depth Training, Tutorials, and Workshops
  • Project Staff with Diverse Expertise for Dedicated Projects
  • Test Environments for Bleeding-edge proposals

Systems

World-class Supercomputing

  • 82,000+ CPU Cores
  • 3,412,688 GPU Cores
  • 462 TB Memory

Big Data Storage & Analysis

  • 10 PB High Performance
  • 13 PB Second Tier
  • 30+ PB Archive Tape Library

​

Staff

  • 53 Full Time Staff
  • 21 Ph.D.s
    • 13 biologists
  • Assorted Expertise
    • Research support
    • Application Development
    • User Services
    • Systems Operations

​

User Base and Outcomes

​

  • 15 Colleges and 110 Depts. Represented
  • 14,500+ Unique View of Training Materials
  • 2,868 Publications by MSI PIs
  • $19,623,263 in Sponsored Funding with MSI Participation
  • 38 Proposals Submitted with MSI Staff Participation

For more information go to: https://rc.umn.edu

Minnesota Supercomputing Institute

© 2020 Regents of the University of Minnesota. All rights reserved.

​

6 of 28

Minnesota Supercomputing Institute:

Strong HPC History & Regular Upgrade Cycle

2010

2010

2012

2014

2016

2018

2020

2022

2024

2026

Itasca

1/31

Mesabi

3/1

Agate

4/27

Agate+1

8/5

Mangi

9/13

Blackwell

9/1

Agate+1/2

8/13

UMN Supercomputing History

  • Seymour Cray UMN graduate
  • First university to own a Cray-1 supercomputer -1981
  • Home to Army HPC Research Center - 1989-2006
  • Top500 systems since the beginning of the Top500

Planned:� Minor upgrade

Planned:� Major upgrade

Major upgrade�7000 TeraFlops

HIPAA System�360 TeraFlops

Minor Upgrade�1150 TeraFlops

Major Upgrade�860 TeraFlops

Major Upgrade�96 TeraFlops

Minnesota Supercomputing Institute

© 2020 Regents of the University of Minnesota. All rights reserved.

​

7 of 28

Groups in 2021: 996 User Groups�5,393 Active users

The Long Tail of Research Computing

Biggest increases in Life Sciences

Minnesota Supercomputing Institute

© 2020 Regents of the University of Minnesota. All rights reserved.

​

8 of 28

*Slide Credit: Manish Parashar, Office Director of the Office of Advanced Cyberinfrastructure (OAC) at the National Science Foundation (NSF) presented at the Fall Midwest Big Data Hub All hands Meeting, October 30, 2019.

Minnesota Supercomputing Institute

© 2020 Regents of the University of Minnesota. All rights reserved.

​

9 of 28

Scalable, Multipurpose Systems Infrastructure

Full Cost Recovery

Fully Subsidized

Broad Mission

Single Mission

Things

Scales Well

Scales Poorly

What your leadership wants

What your faculty wants

Dedicated

Shared

  • Shared - Available to everyone on a first come first serve basis
  • Dedicated - Dedicated to single research group or project

Minnesota Supercomputing Institute

© 2020 Regents of the University of Minnesota. All rights reserved.

​

10 of 28

Engagement Facets

Full Cost Recovery

Fully Subsidized

Custom Solution

Broad Solution

People

Economies of scale

Poor economies of scale

What your leadership wants

What your faculty wants

Consulting

Collaborations

  • Consultations - No direct cost. Short engagements (<1 hour)
  • Collaborations - Dedicated research involvement. Weeks to months of commitment per project.

Minnesota Supercomputing Institute

© 2020 Regents of the University of Minnesota. All rights reserved.

​

11 of 28

General Landscape

  • A central service mission that consumes most of our recurring budget (people and hardware).
  • It is important that our involvement in particular research projects does not detract from the core mission
    • Asking for additional funding to hire staff is often not an option
    • Need for alignment with University Strategic Plan

​

Minnesota Supercomputing Institute

© 2020 Regents of the University of Minnesota. All rights reserved.

​

12 of 28

Biomedical Informatics

Lab Medicine & Pathology

  • Since 2013 Analysts have met a variety of bioinformatics needs for LM&P:
    • Clinical diagnostic pipelines for our affiliated hospital.
      • Whole Exome, Whole Genome, Solid tumor cancer panel, Blood cancer panel
    • Variety of bioinformatics needs for basic biomedical research
  • Many projects are years-long, close collaborations between MSI analysts and labs

​

Minnesota Supercomputing Institute

© 2020 Regents of the University of Minnesota. All rights reserved.

​

13 of 28

Neuroscience and Brain Imaging Research

Masonic Institute for the Developing Brain (MIDB)

  • Analysis, as well as the development of tools and infrastructure to analyze large scale brain MRI datasets (2021)

​

Institute for Translational Neuroscience

  • Translational research in neuroscience combining MRI and sequencing datasets (2019).

Minnesota Supercomputing Institute

© 2020 Regents of the University of Minnesota. All rights reserved.

​

14 of 28

Agroinformatics

GEMS-Platform

  • An integrated platform for controlled sharing and analysis of agroinformatics data
  • Deployed at UMN and Stellenbosch University

GEMS-Sensing

  • Projects for capturing and collating data from agricultural sites

GEMS-Exchange

  • Land use, weather, water quality, and other agroinformatics APIs
  • 27 APIs planned or in development

Minnesota Supercomputing Institute

© 2020 Regents of the University of Minnesota. All rights reserved.

​

15 of 28

General Challenge

  • How do we staff both a core set of services, and specific large scale needs.

​

  • In particular: How do we scale our pool of experts to meet the increased and new demands of data intensive research?

​

Minnesota Supercomputing Institute

© 2020 Regents of the University of Minnesota. All rights reserved.

​

16 of 28

Engagement Facets (extended)

Full Cost Recovery

Fully Subsidized

Custom Solution

Broad Solution

People

Economies of scale

Lose economies of scale

What your leadership wants

What your faculty wants

Partnership

Consulting

Collaborations

  • Consultations - No direct cost. Short engagements (<1 hour)
  • Collaborations - Dedicated research involvement. Weeks to months of commitment per project.
  • Partnerships - Cost recovery, well defined agreements, and longer term commitments for funding areas of research that are generally considered to be core UMN strengths.

Minnesota Supercomputing Institute

© 2020 Regents of the University of Minnesota. All rights reserved.

​

17 of 28

Engagement Framework

​

​

Partnerships

Collaborations

Consultations

Self Serve Services

Costs Recovered

Costs Subsidized

Minnesota Supercomputing Institute

© 2020 Regents of the University of Minnesota. All rights reserved.

​

18 of 28

Partnership Basics

  • Simply put:
    • A department or college provides the funding for specific staff with a multi-year recurring commitment
    • MSI hires and ‘houses’ the staff members
    • The staff members are dedicated to projects from that unit

​

​

  • But of course, this hides a lot of complexity

​

Minnesota Supercomputing Institute

© 2020 Regents of the University of Minnesota. All rights reserved.

​

19 of 28

Memorandum of Understanding

  • Definition of Work Priorities
  • Assignment of Workloads/Performance Evaluations
  • Handling of Grant Funding
  • Scientific and Intellectual Ownership
  • Hiring and Termination of Positions or Agreement

Minnesota Supercomputing Institute

© 2020 Regents of the University of Minnesota. All rights reserved.

​

20 of 28

Definition of Work Priorities

  • Defines a decision-making body within the partner unit that is responsible for prioritizing tasks and projects.

​

    • It is up to the partner to determine what their priorities should be
    • MSI stays out of departmental politics
    • This aligns with the standard division of responsibilities at a University.

​

Minnesota Supercomputing Institute

© 2020 Regents of the University of Minnesota. All rights reserved.

​

21 of 28

Assignment of Workloads

  • Within a partnership, MSI determine who will complete tasks, how to complete them, and ultimately to communicate back to the partner the general project timeline and the status of key milestones.

​

    • Performance reviews are handled within MSI (with partner feedback)

Minnesota Supercomputing Institute

© 2020 Regents of the University of Minnesota. All rights reserved.

​

22 of 28

Handling of Grant Funding

  • Assignment of partnership staff effort to grants offsets the cost to the department or college
  • MSI retains ‘Indirects’ associated with that effort.

​

    • Financially incentivizes both MSI and our partners to strive for grant participation.
    • Helps MSI scale costs of offices, training and administration

Minnesota Supercomputing Institute

© 2020 Regents of the University of Minnesota. All rights reserved.

​

23 of 28

Scientific and Intellectual Property

  • Cedes decision making on publication and authorship to faculty partners, but sets as a default joint ownership of products produced under the partnership.

​

    • Algorithms or other IP that are developed as part of a project may be applicable to the broader set of projects that MSI staff support.
    • Makes clear that MSI staff should be considered for authorship credit for work performed. The existence of the partnership does not remove the need for recognition of intellectual contributions.

Minnesota Supercomputing Institute

© 2020 Regents of the University of Minnesota. All rights reserved.

​

24 of 28

Hiring and Renewal

  • Commit a minimum of two years of ‘hard’ funding support for all positions that are hired under MoU and hiring only occurs with the consent of both parties.
  • Partners are required to continue reaffirming 1 year funding commitments for each staff member from that point forward.

Minnesota Supercomputing Institute

© 2020 Regents of the University of Minnesota. All rights reserved.

​

25 of 28

Variable Incentives/Variable Goals

Research Computing

  • Scalability
  • Stability
  • “Spotlight” Projects
  • Develops Advocates

​

Institutional Partners

  • CI expertise
  • Confidence in Product
  • Retention/Recruitment Perk

CI Professionals

  • Stable work environment
  • Development Opportunities
  • Community of Practice
  • Career Path

Minnesota Supercomputing Institute

© 2020 Regents of the University of Minnesota. All rights reserved.

​

26 of 28

Ongoing Challenges

  • Office Space
    • COVID has helped
      • More remote workers at MSI
      • Other Units are potentially vacating spaces
  • Managerial and Administrative Infrastructure
    • More people and more distinct projects require oversight
  • Reminding partners of the MOU
    • Once the projects get going there can be scope creep.
  • Gathering a Critical Mass
    • We do not sign Partnership MoUs with individual PIs

Minnesota Supercomputing Institute

© 2020 Regents of the University of Minnesota. All rights reserved.

​

27 of 28

Conclusions

  • Common services are the foundation on which trust is built and dedicated partnerships depend. Trust takes time to build.
  • Self-service vs dedicated engagements with CI professionals will be institution dependent
  • Dedicated engagements à la our partnership model
    • 30 total. LM&P 7, ITN 1, MIDB 12, GEMS 8, Surgery 1, Agronomy & Plant Genetics 1
    • Increase opportunities for CI staff development
    • Lower staff turnover
      • 9 years, no positions withdrawn.
      • Turnover: 3 staff (1 faculty, 1 family, 1 new position)

Minnesota Supercomputing Institute

© 2020 Regents of the University of Minnesota. All rights reserved.

​

28 of 28

Question & Comments Please

James (Jim) Wilgenbusch

Contact me:

Some links:

MN Supercomputing Institute:

  • https://msi.umn.edu/

Research Computing:

  • https://rc.umn.edu

Research Computing

© 2019 Regents of the University of Minnesota. All rights reserved.

​