2 of 34

Introduction

Today we want to present and discuss the newly funded HORUS Project and get feedback from our user community.

We will provide an overview of the project as proposed, give some near term plans and seek feedback from our collaborators about what we present and intend to build.

The project page at NSF has an abstract describing HORUS, but the overview is that we are augmenting the existing storage from the previous OSiRIS project with a diverse set of computing resources to create new capabilities for a diverse range of science domains, especially for under-served institutions.

Please use/add-to the online notes during the webinar

HORUS

3 of 34

Acronym Definitions

OSiRIS: Open Storage Research Infrastructure, NSF 5 year project 2016-2021

HORUS: Helping Our Research Upgrade their Science, NSF 2 year project 2022-2024

CE: An OSG Compute Entrypoint (CE) is the door for remote organizations to submit requests to temporarily allocate local compute resources.

NSF: National Science Foundation which funds many types of research projects.

RHEL: Red Hat Enterprise Linux, the de facto enterprise linux operating system.

OSG: Used to mean Open Science Grid, now just “OSG”

Ceph: A popular storage infrastructure uniquely delivering object, block, and file storage in one unified system.

PATh: Partnerships to Advance Throughput Computing, NSF funded project

GPU: Graphics Processing Unit, dedicated hardware for rapid image processing, but now widely used for AI (artificial intelligence) / ML (machine learning).

HORUS

4 of 34

Agenda for Today

During this hour we want to cover the following topics:

HORUS presentation (this is it :) )
Questions and possible answers

If we don’t have detailed answers now, we can certainly work on getting them moving forward

Expectations and concerns from our science collaborations

How are you doing your science now?
How would your benefit from co-located storage and compute?
What are your questions and concerns?

Future meetings: what should we do about keeping in touch?

HORUS

5 of 34

The HORUS Project in a Nutshell

The HORUS project was conceived as a means to add a heterogeneous set of computing resources to augment the existing storage provided by the previously funded NSF OSiRIS grant (more information on OSiRIS coming up).

We proposed three distinct computing server configurations to try to match the wide range of needs described by our science collaborators:

Large memory servers: 1TB / server, 10.7 GB/job-slot
Large core-count servers: 2 processors, 64-core/128-thread per processor
GPU servers: 4 A100s per server, 32 job slots / server

HORUS intends to integrate easy access to the existing OSiRIS storage to allow our collaborators to stage and process their datasets using HORUS computing systems.

HORUS is also collaborating with the OSG / PATh projects to provide access for 20% of our capacity to other national infrastructure users.

HORUS is funded for 2 years but we intend to operate the project for at least 4 years.

HORUS

6 of 34

OSiRIS - The Precursor to HORUS

The OSiRIS proposal targeted the creation of a distributed storage infrastructure, built with inexpensive commercial off-the-shelf (COTS) hardware, combining the Ceph storage system with software defined networking to deliver a scalable infrastructure to support multi-institutional science.

Current: Single Ceph cluster (Octopus 15.2.17 ) spanning U-M, WSU, MSU - 1500 OSD / 13.7 PiB

https://www.osris.org

HORUS

7 of 34

Building Upon OSiRIS

The OSiRIS project was funded in 2016 (a 5 year award) to create a software defined storage infrastructure to support multiple diverse science domains for their scientific storage needs.

The project ended in 2021 but has continued to be maintained on a best effort basis.

The OSiRIS proposal targeted using data “in-place” from well connected locations across our member institutions but that turned out not to be the primary way our users leveraged OSiRIS.

The more important use case seemed to be the ability to more broadly and easily share large-scale data with collaborators and others interested in the data.

One of the issues that some of our OSiRIS users brought up was having local computing resources able to process the stored and shared data in OSiRIS.

OSiRIS was able to create significant capabilities to allow scientific collaborations to register users using their institutional identity and self-manage access.

HORUS

8 of 34

The Project Team

The HORUS project team incorporates the previous core OSiRIS team and new involvement of Merit Network.

The PI and Co-PIs: Shawn McKee / University of Michigan (PI), Bob Stovall / Merit Network (Co-PI), Rob Thompson / Wayne State University (Co-PI)

Senior Personnel: Andy Keen, Brian O’Shea / Michigan State University, Michael Thompson / Wayne State University

Project Team Members: Nick Grundler, Muhammad Akhdhor, Wendy Dronen / University of Michigan, Aragorn Steiger, Matt Lessins, Patrick Gossman / Wayne State University, Pierrette Renée Dagg, Merit Network.

HORUS

9 of 34

HORUS Equipment Status

We have already ordered our first year equipment based upon the specs shown below and most of it is already racked at UM, MSU and Wayne State.

Users will not need to worry about the physical location of these servers since jobs will be scheduled across all possible servers based upon job requirements.

The HORUS team is just beginning to test provisioning, server configuration, OSiRIS integration and user interfaces.

HORUS Systems	Model	Mem (G) / Host	CPU	CPU Cnt	GPU	GPU Mem	GPU /Host	NICs	Yr1 Host Cnt	HT Job Slots (total)	Mem (G) / Slot
GPU Node	Dell R750xa	512	Xeon Gold 6334 8C/16T	2	A100	80G	4	2x25G	2	64	16.0
Large Memory Node	Dell R6525	1024	AMD Epyc 7F72 3.2GHz, 24C/48T	2	-	-	-	2x25G	6	576	10.7
Compute Node	Dell R6525	512	AMD Epyc 7H12 2.60GHz, 64C/128T	2	-	-	-	2x25G	6	1536	2.0

HORUS

10 of 34

HORUS Networking

We will leverage the excellent networking put in place for OSiRIS / Research

The network infrastructure shown on the right was created by a combination of OSiRIS and a CC* Grant received by Michigan State University

HORUS

11 of 34

Merit Network - Michigan’s Research and Education Provider

Resilient Optical and Layer3 network within Michigan

100 Gbps connections to Internet2 in Chicago and Toledo

100 Gbps connections to Internet service providers in Grand Rapids, Southfield and Detroit

Connected to Michigan’s public universities via optical transport and routing infrastructure

HORUS

12 of 34

HORUS Building Blocks / Open Source Components

The planned HORUS software architecture is built upon a number of open source tools and applications and we group them in the following list by the particular role they are used in:

Authentication and Authorization: InCommon, a set of community designed identity and access management services, CoManage, used to manage identity lifecycle, Grouper, used to create and manage roles, groups and permissions and CILogon to enable researchers to log on to cyberinfrastructure.
Resource allocation and management: HTCondor, used to manage fair-share access for computational tasks to a set of computing resources, HTCondor-CE, a meta-scheduler used as a “door” to a set of resources, NVIDIA MIG, used to subdivide a GPU (cores and memory) into up to 7 smaller instances, allowing more jobs to share a GPU and Ceph with quotas to manage storage use in OSiRIS.
Monitoring and Accounting: Elasticsearch, used to gather data from syslogs and other sources for aggregation, visualization, analytics and correlation, CheckMK, a intelligent server and host monitoring system capable of validating service states and tracking resource usage, perfSONAR, used to test and monitor network behavior across our infrastructure and RHEL8 accounting and auditing to augment usage and security information about our systems.

HORUS

13 of 34

Expected HORUS User Interface

The HORUS user interface is one of the main areas the project team is working on.

We don’t yet have the final details but we can describe the general ideas

We will user CILogon to support users “logging into” HORUS using their own, existing institutional credentials.

We are targeting Open OnDemand as the primary user interface.
Command line (ssh + terminal) should also be an option

Users will have their own assigned project space, potentially shared with other users from the same project and we intend to provide Globus Online access and scp for transferring data into and out of the project space.

This will allow users to bring in their own software, assuming it doesn’t need ‘root’
If software is needed by more than one group, the HORUS team may be able to centrally provision it or add specific software that requires ‘root’ access to install.

HORUS

14 of 34

Project Timeline and Milestones

Shown on the right is the project timeline.

Given recent issues with product supply chains, we may have some challenges in getting components procured, delivered and installed as we originally planned.

Part of the plan for next quarter is identifying the early adopters from our science collaborations

Let us know if your want to be a guinea pig for the project !!

HORUS

15 of 34

Overview of the Proposed Science Collaborations

HORUS

16 of 34

HORUS and OSG: On-ramp to National Scale Resources

As part of the NSF award requirements, the HORUS project needs to make available at least 20% of its capacity for user outside of our region:

“Proposals are required to commit to a minimum of 20% shared time on the cluster and describe their approach to making the cluster available as a shared resource external to the state/region and the set of institutions being primarily served. Proposals are strongly encouraged to address this requirement by joining the Partnerships to Advance Throughput Computing (PATh) campus federation (https://opensciencegrid.org/campus-cyberinfrastructure.html) and adopting an appropriate subset of PATh services to make the cluster available to researchers on a national scale.”

We proposed to do this via the OSG/PATh project and have a close working relationship with them (PI leads OSG Networking, UM/WSU already contribute cycles to OSG).

Our plan is to use an HTCondor meta-scheduler and integrate with an OSG CE (Compute Element) to allow users outside our region to access HORUS.

A beneficial side effect of this integration is that the HORUS team has an opportunity to make OSG/PATh resources more easily accessible for HORUS users at the same time.

HORUS

17 of 34

What Will Be Available and When?

As shown in the timeline slide, we hope to have initial hardware in place and configured in about spring 2023.

Part of our task is to create an easy to use interface for our science collaborations and this may take some time to tune and debug. We will want some early adopters who are willing to help us work through some of the start up issues.

For most science users, it may take us a year to get a production infrastructure in place but we could be ready in late summer 2023, depending upon how things go.

One of the reasons we left ~1/3rd of the equipment to be purchased in the second year of the project was to allow us to evaluate what the real needs are and tune the purchased configuration to best match. This will require us to have some experience and feedback from the majority of our users.

If there is interest, we can discuss timing and details later during Q&A.

HORUS

18 of 34

Next Steps By Role

The HORUS Team

Finalizing architecture details

Upgrading OSiRIS (RHEL8, current Ceph, refresh toolsets)
Prototype, test and evolve HORUS user interfaces

Enable OSG 20% access and associated accounting systems
Activating website

Science Collaborators

Review and document current workflows
Determine best timeline to join HORUS
Identify needed software, libraries and storage requirements

Interested Potential Users

Write up use cases and needs and contact the HORUS team

Campus IT and Administrators

Plan for HORUS user needs: connectivity, support and integration with local IT

HORUS

19 of 34

Questions or Comments

Questions / Discussion?

Please use/add-to the online notes as we discuss:

https://docs.google.com/document/d/1_zYgVzXyrjWi2evOgWfhcNU3JzWFuCPcn4d69gRYVvo/edit#heading=h.71cs8mw1wvfb

Email us with questions: horus-help@umich.edu

Website: http://www.horus-ci.org/ (not yet ready!)

HORUS

20 of 34

Acknowledgements

We would like to thank our HORUS science partners and our host institutions for their contributions to work described.

In addition we want to explicitly acknowledge the support of the National Science Foundation which supported this work via:

HORUS grant, NSF OAC-2232628
Award information at https://www.nsf.gov/awardsearch/showAward?AWD_ID=2232628&HistoricalAwards=false

HORUS

21 of 34

Further Information

OSiRIS

http://www.osris.org project website

Details in various presentations at http://www.osris.org/publications

OSG

https://osg-htc.org/ project website

HTCondor

https://htcondor.org/ project website

Open OnDemand

https://openondemand.org/ project website

NSF CC* Program

https://beta.nsf.gov/funding/opportunities/campus-cyberinfrastructure-cc

HORUS

22 of 34

Additional Slides Follow

Backup Slides

HORUS

23 of 34

OSiRIS Storage Summary

We have deployed 13.7 pebibytes (PiB) of raw Ceph storage across our three research institutions in the state of Michigan.

Older storage node is a 2U headnode and SAS attached 60 disk 5U shelf with either 8 TB or 10 TB disks, 4x25G network links (two dual 25G cards)
New year-4 hardware installed

Dell R7425 (Dual AMD 7301) 2U, 16x12TB disks, 128G RAM, 2x25G NIC, 2x10G, 1x1G, 4 x Samsung 970 Pro 512G NVMe, BOSS card
Added 6.3 PiB to OSiRIS by January 2020 (now 1368 total disks)

Ceph components and services are virtualized

The OSiRIS hardware is monitored by Prometheus and configuration control is provided by Puppet

Institutional identities are used to authenticate users and authorize their access via CoManage and Grouper

Augmented perfSONAR is used to monitor and discover the networks interconnecting our main science users.

HORUS

24 of 34

OSiRIS Science Domains

The primary driver for OSiRIS was a set of science domains with either big data or multi-institutional challenges.

OSiRIS is supporting the following science domains:

ATLAS (high-energy physics), Bioinformatics, Jetscape (nuclear physics), Physical Ocean Modeling, Social Science (via the Institute for Social Research), Molecular Biology, Microscopy, Imaging & Cytometry Resources, Global Night-time Imaging
We are currently “on-boarding” new groups in Genomics and Evolution and Neural Imaging (next slide)
Primary use-case is sharing working-access to data

HORUS

25 of 34

Recent Science Domains

Brainlife.io (Neuroimaging) - Brainlife organizes neuroimaging data and data derivatives using their registered data types. No single computing resources has enough storage capacity to store all datasets, nor reliable enough so that user can access the data when they need them. They will depend on OSiRIS to store datasets and transfer data between computing resources.

Oakland University - Already a user of MSU iCER compute resources, OU will leverage OSiRIS to bring their data closer for analysis and for collaboration with other institutions.

Evolution - Large-scale evolutionary analyses, primarily phylogenetic trees, molecular clocks, and pangenome analyses

Genomics - High volume of human, mammal, environmental, and intermediate analysis data

HORUS

26 of 34

New and Ongoing Collaborations

Open Storage Network - We will be providing ~1 PB to be included in the Open Storage Network (https://www.openstoragenetwork.org)

Timeline depends on OSN readiness to engage, some discussions at OSN group meeting at TACC in Fall 2019

FABRIC - This is a newly funded NSF project to create a network testbed at-scale (1.2 Tbps across the US). OSiRIS will be an early adopter/collaborator, providing ~1 PB to support science use-cases
Library Sciences - OSiRIS roadmap plans for data lifecycle mgmt

Following detailed analysis of two specific datasets, library scientists at UM are working on automated metadata capture and indexing
Integration with U-M ‘Deep Blue Data’ archival system also planned

HORUS

27 of 34

Network Upgrades - 100Gb MiLR

MiLR is a high-speed, special purpose, data network built jointly by Michigan State University, the University of Michigan, and Wayne State University, and operated by the Merit Network.

Thanks to combined effort from campus network teams and Merit we were able to deploy direct 100Gb links via MiLR fiber landing directly on our OSiRIS rack switches

Now we have more options for network management without campus network disruptions and this provides options for experimenting with SDN via NMAL

In our current phase of implementation, they carry only the Ceph ‘cluster network’ used for OSD replication data (Ceph self-healing)

Normal ceph recovery/backfill operations could easily overwhelm smaller links with this traffic, so removing it was a huge difference that let us completely remove throttles on Ceph recovery (see next slide)

HORUS

28 of 34

Michigan Innovation Network

HORUS

29 of 34

COmanage Credential Management

COmanage Ceph Provisioner plugin provides user interface to manage S3 credentials and default bucket placement

Work is underway to include a full GUI for managing buckets: Create, rename, download, set ACL from OSiRIS groups or specific user, etc.

HORUS

30 of 34

Globus and Gridmap

We provide Globus access to CephFS and S3 storage

Ceph connector uses radosgw admin API to lookup user credentials and connect to endpoint URL with them

Credentials: CILogon + globus-gridmap

CILogon DN in LDAP voPerson CoPersonCertificateDN attribute

We wrote a Gridmap plugin to lookup DN directly from LDAP (student project)

Having the subject DN and lookup entirely in LDAP means it will be easy to add capabilities to COmanage so users can self-manage this information

Users already self-manage SSH login keys in COmanage (also in LDAP)

HORUS

31 of 34

Network Management

The OSiRIS Network Management Abstraction Layer is a key part of the project with several important focuses:

Capturing site topology and routing information from multiple sources: SNMP, LLDP, sflow, SDN controllers, and existing topology and looking glass services
Converge on common scheduled measurement architecture with existing perfSONAR mesh configurations
Correlate long-term performance measurements with passive metrics collected via other monitoring infrastructure

Recently wrote new Prometheus exporter to collect perfSonar test results from central ESmond store for alerting and visualization

We demo’ed SDN architecture for traffic routing and traffic shaping / QOS (prioritize client / cluster service traffic over recovery) at SC19

NMAL work is led by the Indiana University CREST team

HORUS

32 of 34

OSiRIS Topology Discovery and Monitoring

UNIS-Runtime release integrated into ZOF-based discovery app

Increased stability and ease of deployment
Added extensions for Traceroute and SNMP polling

Web Development has focused on bringing measurements to dashboard

Link and node highlighting with thresholds determined by link capacities
Overlay for regular testing results to bring “at-a-glance” diagnostics

Filtering to show layer-2 topology versus layer-3 and virtualized components

Fault localization, clustering, and zoom are work-in-progress

HORUS

33 of 34

Summary

OSiRIS continues to improve on our user experience and engage with new collaborators

ATLAS/High-energy Physics has been a long time user for Event Service data

Our new hardware purchases this year will increase our node count and make EC pools feasible (no clients yet for this)
We look forward to participating in more national scale projects such as the Open Storage Network, FABRIC, Eastern Research Network
On our roadmap this year:

Upgrade to Octopus (15.2.1)
Make our S3 services more highly available with LVS failover endpoints on each campus
Make S3 services more performant by greatly increasing instance count behind the proxy endpoints
Improve user GUI for managing storage access
Build more convenient client bundles, modules, etc to make OSiRIS usage as easy as possible
Adding ATLAS dCache storage to explore using Ceph to manage back-end storage.

HORUS

34 of 34

FABRIC Topology

FABRIC (https://fabric-testbed.net/ ) is a newly funded network testbed spanning the US

Michigan is an early

adopter (2021)

https://whatisfabric.net/events/fabric-community-workshop-2020

HORUS