1 of 33

Regional Network Value:

Data Mobility

Ken Miller, ken@es.net

Jason Zurawski, zurawski@es.net

National Science Foundation Award #1826994

2 of 33

Regional Network Value : Data Mobility

  1. Introduction
  2. PetaScale Motivation
  3. Data Mobility Exhibition
  4. Engagement with your subscribers

2

3 of 33

R & E Network Value for Data Mobility

  • Performance is typically measured as much as utilization and capacity (perfSONAR, NetSage)
  • More Engineered toward elephant flows
  • Awareness of performance problems involves the community
  • Troubleshooting problems involves a handful of people/networks
  • Large buffering is a requirement.
  • Dropped Packets are bad, very bad.

3

4 of 33

4

5 of 33

Regional Network Value : Data Mobility

  1. Introduction
  2. PetaScale Motivation
  3. Data Mobility Exhibition
  4. Engagement with your subscribers

5

6 of 33

6

Note significant improvement between NCSA and ANL

7 of 33

7

ALCF/NERSC and ALCF 🡪 NCSA improvements

NCSA 🡪 ALCF temporary setback

8 of 33

DTN Cluster Performance – HPC Facilities (2017)

8

9 of 33

Improvements In Multiple Places

  • DTN clusters expanded
    • ALCF: 8 DTNs to 12
    • NERSC: 4 DTNs to 8
    • ORNL: 8 DTNs to 16 (for Globus pool)
  • Network Upgrades
    • ORNL: Nx10G to Nx40G
    • NCSA: Nx10G to Nx100G
    • NERSC: Nx10G to Nx40G
  • Unrelated Globus change gave a big performance boost
    • Change in pipelining behavior of transfers with large file counts

9

10 of 33

Petascale DTN Lifts All Boats

  • Petascale DTN project benefits all projects which use the HPC facility DTNs
  • Modern science data portal architecture
    • Data portals which use modern architecture benefit from DTN improvements
    • DTN scaling/improvements benefit all data portals which use the same pool
  • Globus API supports this – see Globus World Tour

​

10

11 of 33

Regional Network Value : Data Mobility

  1. Introduction
  2. PetaScale Motivation
  3. Data Mobility Exhibition
  4. Engagement with your subscribers

11

12 of 33

Motivation/Background

  • Networks are an essential part of data-intensive science
    • Connect data sources to data analysis
    • Connect collaborators to each other
  • Performance is critical, but often overlooked
    • Exponential data growth
    • Constant human factors
    • Data movement and data analysis must keep up
  • Effective use of wide area (long-haul) networks by scientists has historically been difficult
  • So, how long does it take to transfer 1TB of data on your network?

12

13 of 33

Current DME Testing Framework Endpoints

  • Sites:
    • Cac_dtn_test - Cornell Center for Advanced Computing in Ithaca, NY
    • Pamela Hill Data Share - NCAR GLADE (IBM Spectrum Scale) filesystem in Boulder, CO
    • DME PerfTest - Argonne - IBM Spectrum Scale filesystem on the Petrel system at Argonne National Lab
    • ESnet DTN in Washington,DC and Chicago (available now)
    • DME PerfTest - Argonne - EAGLE DTN Pool at Argonne National Lab
  • Cloud:
    • DME PerfTest - Google Drive
    • DME PerfTest - Box

13

14 of 33

Transfer Performance At Different Data Scales

14

This table available at: http://fasterdata.es.net/fasterdata-home/requirements-and-expectations/�

10G DTN min

15 of 33

Data Mobility Exhibition -

Raising Your Data Transfer Expectations

  • To baseline, how long does it take your site to transfer 1TB of data?

​

  • Our goal is a 10G connected DTN capable 1TB/hour, or 2.22Gb/s. or 277 MB/s disk-to-disk, as a minimum
  • On a campus 1G connected desktop, data transfer speeds should be around 1TB in 3 hours, or 800 Mb/s, or 100 MB/s

​

  • Know of anyone shipping hard drives or carrying jump drives across campus?
  • Are you getting the data transfer performance with the hardware capacity you paid for?

​

© 2022, Engagement and Performance Operations Center (EPOC)

15

16 of 33

Data Mobility Exhibition (DME)

  • Transferring reference data sets from well tuned DTNs
  • Using existing campus CI components, participants will work on scientific data movement capabilities:
    • Download data sets
    • Upload data sets
    • Measure and baseline against 1TB/hr transfer rate or specific requirement
    • Seek to improve transfer results locally from an end-to-end test
    • 10793 tests completed through 2/1/2022

16

17 of 33

DME Transfers from Unique Sources

average by university

Each bar is the average data transfer rate from a unique site/campus DTN across any and all data sets

17

18 of 33

Downloads: DME Transfers to Unique Destinations

Each bar is the average data transfer rate to a unique site/campus DTN across any and all data sets

18

19 of 33

DME participation via Regionals

  • CENIC
  • FLR
  • Starlight/MREN
  • MARIA
  • BOREAS
  • MAX
  • CEN

​

EPOC Partners:

  • KINBER
  • NYSERnet
  • GPN-KANREN,ARE-ON,ONENET,MORENET,SD-REED
  • I-Light
  • FRGP
  • LEARN
  • Sun Corridor
  • NCAR
  • N-WAVE
  • SOX

19

20 of 33

Data Transfer Rates by Audience

A benchmark table is provided to gauge data architecture performance, which can vary depending on number of files, folders, size of files, distance between sites, CI performance (network, server, disk/filesystem), as well as data transfer tool.

20

Host Transfer Rates

⅙

PetaScale

(Minimum)

⅓

PetaScale

½

PetaScale

�

PetaScale: 

1 PB/wk 

PetaScale: 

1 PB/day

�

�

10G Capable DTN

�

10xG, 25G, 40G, 100G DTNs

Data Transfer Rate/Volume (Researcher)

1 TB/hr

2 TB/hr

3 TB/hr

�

5.95 TB/hr

41.67 TB/hr

Network Transfer Rate (Network Admin)

2.22 Gb/s

4.44 Gb/s

6.67 Gb/s

�

13.23 Gb/s

92.59 Gb/s

Storage Transfer Rate (Sys/Storage Admin)

277.78 MB/s

555.54 MB/s

833.33 MB/s

�

1.65 GB/s

11.57 GB/s

21 of 33

Regional Network Value : Data Mobility

  1. Introduction
  2. PetaScale Motivation
  3. Data Mobility Exhibition
  4. Engagement with your subscribers

21

22 of 33

Roadside Assistance Process

  • “This file transfer worked last week, but it doesn’t anymore?”
    • Think of this like a flat tire, crash repair
    • Anyone can submit
  • Contact epoc@iu.edu
    • Within 24 hours, gets triaged
    • Some initial investigation to verify the issues
    • A Case Manager and Lead Engineer are assigned
    • Shareable infrastructure set up
  • Centralization of Researcher Assistance

​

22

23 of 33

EPOC Six Main Focus Areas

  1. Roadside Assistance and Consulting
  2. Application Deep Dives
  3. Network Analysis (NetSage)
  4. Data Mobility Exhibition/Baseline Testing
  5. Managed Services
  6. Training

​

24 of 33

Why care? Vehicle perspective…

Roadside Assistance with EPOC and DME

Vehicle

  • < 20 MPH
    • Pull off and call a tow truck
    • Get a tune up
  • 40 MPH
    • Turn on hazards
    • Should have a tune-up
  • 60+ MPH
    • Highway Speeds

Data Transfer Node (10G DTN)

  • < 2 Gb/s ( < 1 TB/hr ) or 20%
    • Please request assistance from epoc@iu.edu
  • ~ 4 Gb/s ( ~ 2 TB/hr ) or 40%
    • Ok, but measure with perfSONAR, then DME
  • ~ 6 Gb/s ( ~3 TB/hr ) or 60%
    • Ok
  • 10G Line rate

© 2022, Engagement and Performance Operations Center (EPOC)

24

25 of 33

Data Mobility Exhibition (DME)/

Baseline Performance Testing

  • One TeraByte of data in an hour
    • Equivalent to 2.22 Gb/s average network throughput
    • Most academic institutions connected to R&E Networks at 10G
  • How to find out?
    • DME has known good endpoints to test against
    • 5 on R&E Networks, 2 in cloud
    • Variety of data sizes
    • Standard Globus set up
  • And if you can’t?
    • Work with EPOC to find the bottlenecks!

26 of 33

DME Data Sets

26

ds01

100MB

10,000 x 10KB files in single directory

ds04

10GB

10,000 x 1MB files in 100 non-nested directories, 100 files/directory

ds06

100GB

100,000 x 1MB files in single directory

ds08

1TB

50 x 10GB; 350 x 1GB; 1,000 x 100MB; 5,500 x 10MB; 23,176 x 1MB files in single directory

ds10

1TB

100 x 10GB files in single directory

ds12

100GB

1 x 100GB file in single directory

ds14

5TB

50 x 100GB files in single directory

ds16

1TB

4 x 250GB files in single directory

27 of 33

DTN Reference Architecture - wash-dtn1.es.net

  • CPU
    • 2 x Intel Cascade lake Xeon Gold 6246
    • 12 cores each
    • 3.3GHz 165W TDP processor
  • Memory
    • 12 x 16G DDR4 2933 ECC RDIMM (192G total)
  • Disk
    • 10 x Intel P4610 1.6TB U.2/2.5” PCIe NVMe 3.0 x4 Drives
    • 2 x Enterprise 960G 2.5“ SATA SSD (OS, onboard Intel SATA Raid 1)
  • Network
    • Mellanox ConnectX-5 EN MCX516A-CCAT 40/50/100GbE dual-port QSFP28 NIC
  • Application

27

28 of 33

Data Mobility Benchmark

  • Try to benchmark your DTNs and Data Architectures monthly or after any changes.
  • Download ESnet data Climate Data Sets from Wash-DTN1.es.net or another ESnet server to test your write speeds
  • For larger systems, try the DME datasets:
  • Once downloaded, you can re-upload to test your read speeds.

28

29 of 33

Science DMZ Tuning Recommendations

  • Verify your MTUs match for your Science workflow as well as data transfer paths, storage, and nodes: EPOC paper on MTUs
  • Adjust BGP to prefer Research and Education Networks
  • Setup perfSONAR at your border router as well as near the edge of your science resources (near an instrument)
  • Research the Modern Research Data Portal for data distribution
  • Setup Weekly Top 10 Source and Destination reports to keep an eye on large transfers. This also help find researchers to engage with. If tools are limited, check out NetSage with your regional network provider.
  • Well tuned 10G DTNs go a long way vs a single 100G DTN.
  • If large transfer must go through border firewall for Restricted or Controlled data, research if your firewall has an ASIC path (FastPath/ExpressPath)
  • DTN Tuning: https://fasterdata.es.net/DTN/tuning/

29

30 of 33

Regional Network Value : Data Mobility

  1. Regionals have a great opportunity as the link between campuses and national networks
  2. Use NetSage or another tool verify data transfer performance and add DTNs to Science Registry
  3. Being engineered for elephant flows, regionals typically transfer 4x faster or more
  4. Do you have a value-add page for your network vs commodity? If not, check out REANNZ’s page.
  5. Please check your MTUs for consistency or default jumbo
  6. You do not need 100G perfSONAR or 100G DTNs to be performant. Capacity doesn’t always equal performance.
  7. If you need any assistance in testing or verifying network or data transfer performance, please contact epoc@iu.edu

30

31 of 33

NetSage Links

​

​

31

32 of 33

EPOC Takeaways - Questions, Comments, Jaded Opinions?

  • EPOC is an NSF-funded operations center to help scale science engagement and problem resolution
  • Single point of contact for end-to-end performance issues
    • epoc@iu.edu
  • More about EPOC:
    • http://epoc.global

​

​

33 of 33

DME Links:

33