1 of 38

ESnet & EPOC: Support for Remote Scientific Use Cases

Energy Sciences Network (ESnet)

Lawrence Berkeley National Laboratory

Jason Zurawski, Science Engagement Engineer

zurawski@es.net

March 29th, 2023

© 2022, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy). All rights reserved.

​

NOTICE. This material is owned by the U.S. Department of Energy. As such, the U.S. Government has been granted for itself and others acting on its behalf a paid-up, nonexclusive, irrevocable, worldwide license in the material to reproduce, prepare derivative works, and perform publicly and display publicly.

2 of 38

Topics

  • What / Who is ESnet
  • What / Who is EPOC?
  • Science Engagement Basics
  • Three Scientific Case Categories
  • Conclusions

​

​

2 – ESnet Science Engagement (engage@es.net) - March 2023

3 of 38

ESnet is DOE’s high performance network user facility

3 – ESnet Science Engagement (engage@es.net) - March 2023

4 of 38

4 – ESnet Science Engagement (engage@es.net) - March 2023

ESnet6

ESnet6 Project Scope

To satisfy Mission Need, the ESnet6 project scope resolves capability gaps:

  1. Increasing basic network capability and capacity while replacing end of life equipment
  2. Automating network operations and improving security services
  3. Providing programmable network flexibility to support evolving computation and data models

5 of 38

Topics

  • What / Who is ESnet
  • What / Who is EPOC
  • Science Engagement Basics
  • Three Scientific Use Case Categories
  • Conclusions

​

​

5 – ESnet Science Engagement (engage@es.net) - March 2023

6 of 38

Engagement and Performance Operations Center

  • Joint project between TACC and ESnet
  • Part of CC* program for domestic science support
  • Award #1826994, $3.5M over 3 6 years
  • Partnerships with regional, infrastructure, and science communities that span the NSF and DOE continuum of funding

© 2019, Engagement and Performance Operations Center (EPOC)

6 – ESnet Science Engagement (engage@es.net) - March 2023

7 of 38

Topics

  • What / Who is ESnet
  • What / Who is EPOC
  • Science Engagement Basics
  • Three Scientific Use Case Categories
  • Conclusions

​

​

7 – ESnet Science Engagement (engage@es.net) - March 2023

8 of 38

Move Slow, and Build Relationships

8 – ESnet Science Engagement (engage@es.net) - March 2023

9 of 38

What Does Engagement Do?

  • We help scientists use networking as an effective scientific tool – researchers, facilities, programs
  • We seek to understand, to the greatest extent possible, the ways in which scientists make use of technology in their work (focused on but not limited to networking)
  • We develop and promote best practices to improve the state of the art in science networking globally (e.g. fasterdata knowledgebase)
  • We help others find and fix network performance problems to improve the performance of science networking globally
  • All the above is directly in service of the ESnet vision:
    • Scientific progress will be completely unconstrained by the physical location of instruments, people, computational resources, or data.
    • Collaborations at every scale, in every domain, will have the information and tools they need to achieve maximum benefit from scientific facilities, global networks, and emerging network capabilities.
    • ESnet will foster the partnerships and pioneer the technologies necessary to ensure that these transformations occur.

​

​

9 – ESnet Science Engagement (engage@es.net) - March 2023

10 of 38

The Bad Old Model

  • View scientist as integrator
    • Requires scientists to discover new technologies
    • Requires scientists to become expert in new technologies
    • Requires scientists to assemble separate and distinct technologies into an integrated solution that works for them
    • Some scientists do this brilliantly – most do not
  • If you make the scientist the integrator, they will use wget/curl and rsync over ssh forever
  • They can’t effectively use ESnet, because they are using the wrong tools, because the right tools are too hard to use

10 – ESnet Science Engagement (engage@es.net) - March 2023

11 of 38

Science Engagement: New Model

  • View scientist as collaborator, not integrator
  • We understand how networking and computing work for scientists
    • What tools and services are available? How can they be composed?
      • Remember, we don’t want to make the scientist the integrator
    • How must the services change for scientists to be more productive?
  • We understand how systems are built, how they run
    • Networking and network protocols
    • Computing systems
    • Storage systems
    • Software toolset (e.g. data transfer tools)
  • We understand enough of the science to be effective, and to bridge gaps
    • Meet the scientists where they are
    • Help the scientists work better
    • Help the networks provide better services

​

11 – ESnet Science Engagement (engage@es.net) - March 2023

12 of 38

Topics

  • What / Who is ESnet
  • What / Who is EPOC
  • Science Engagement Basics
  • Three Scientific Use Case Categories
  • Conclusions

​

​

12 – ESnet Science Engagement (engage@es.net) - March 2023

13 of 38

Science Use Case Drivers (short term)

  • Convenience of IoT and the ability to deploy instruments in non-laboratory settings and hostile or constrained environments

​

​

  • Within facilities, ease of relocation, equipment movement, and management

​

​

  • Explosion of wireless options, each with different capabilities, cost models and uses makes data more fluid, less expensive to transport
    • 5G, Private Wireless/CBRS
    • LoRa
    • Zigbee
    • Starlink, Orbcomm, Non Terrestrial Networks
    • mmWave

​

​

​

​

​

13 – ESnet Science Engagement (engage@es.net) - March 2023

14 of 38

Longer term use case driver: Self Guided Field Laboratories and an automation revolution for scientific activity

14 – ESnet Science Engagement (engage@es.net) - March 2023

15 of 38

Topics

  • What / Who is ESnet
  • What / Who is EPOC
  • Science Engagement Basics
  • Three Scientific Use Case Categories
    • Supporting Remote Sensors with 5G/Wireless Edge
    • The Atmospheric Radiation Measurement (ARM)
    • NSF’s United States Antarctic Program
  • Conclusions

​

​

15 – ESnet Science Engagement (engage@es.net) - March 2023

16 of 38

Science Motivations

ESnet is researching how to incorporate 5G/Wireless and better support field science.

  1. What are the right support models for scientists engaged in field research?

16 – ESnet Science Engagement (engage@es.net) - March 2023

  1. As we “grow into” ESnet6 capabilities, how do we want to posture Wireless for:
    1. Edge compute / HPC backhaul
    2. Intelligent network management
    3. Automation, etc
    4. other….

​

​

17 of 38

Science Engagement Case 1: urban radiation detection

17 – ESnet Science Engagement (engage@es.net) - March 2023

LBL-NSD Applied Nuclear Physics Program’s PANDA project is developing an integrated smart radiation sensor based on ANL’s Waggle chassis.

  • ML at the edge for multi-mode radiation and context data fusion
  • Need to be self contained, wireless, supporting movement of spectroscopic/optical/Lidar data
  • Need to operate collaboratively as part of a network
  • Working now to deploy several test units at LBL, before hopefully expanding to test in a wider setting

18 of 38

Science Engagement Use Case 2: Env. Field Science in Colorado

  1. Sensor fields (in white) covered by CBRS antenna/base station placed near Mount Crested Butte SAIL X-band radar site (marked in yellow).
  2. Sensors connected either through cellular directly, or via 1-2 solar cellular-wifi routers deployed by this project in the sensor fields.
  3. Backhaul at the SAIL radar site through fiber and Starlink terminals (in collaboration with Starlink)

18 – ESnet Science Engagement (engage@es.net) - March 2023

19 of 38

Efforts last Year

Deployment of our ESnet NDAC Core at LBL, with a remote BBU/RH serving in CO.

+Lets ESnet test operation of a multi-site cell service supporting field science

+Improves coverage for sensors at WFSFA

+Allows exploration of integration and fielding challenges with LoRA, Starlink, mmWave and other related technologies

​

19 – ESnet Science Engagement (engage@es.net) - March 2023

20 of 38

Starlink

  • Procured 3 MT, and deploying in CO to provide CBRS backhaul
  • Working on peering and ensuring high quality connections to ESnet as DOE/Science use grows
  • CBRS backhaul of mutual interest, as some telcos begin to use CBRS to support remote area build-out via Starlink as well. Many interesting lessons to learn.

20 – ESnet Science Engagement (engage@es.net) - March 2023

Lining up …

21 of 38

21 – ESnet Science Engagement (engage@es.net) - March 2023

22 of 38

Topics

  • What / Who is ESnet
  • What / Who is EPOC
  • Science Engagement Basics
  • Three Scientific Use Case Categories
    • Supporting Remote Sensors with 5G/Wireless Edge
    • The Atmospheric Radiation Measurement (ARM)
    • NSF’s United States Antarctic Program
  • Conclusions

​

​

22 – ESnet Science Engagement (engage@es.net) - March 2023

23 of 38

The Atmospheric Radiation Measurement (ARM)

  • ARM is the world’s premier ground-based observations facility advancing atmospheric and climate research.
  • The ARM user facility is a multi-laboratory, DOE scientific user facility, and a key contributor to national and international climate research efforts.
  • ARM data are currently collected from six atmospheric observatories:
    • Southern Great Plains
    • North Slope of Alaska
    • Eastern North Atlantic
    • 3 Mobile facilities and aerial facilities
  • Data from these atmospheric observatories, as well as from past research campaigns and the former Tropical Western Pacific observatory, are available through the ARM Data Center

23 – ESnet Science Engagement (engage@es.net) - March 2023

24 of 38

ARM Maps for Barrow

  • Main Site:
    • 71° 19′ 22.8″ N, 156° 36′ 54″ W
  • BARC:
    • 71°19'29.80"N, 156°40'5.01"W
  • Duplex:
    • 71°19'33.81"N, 156°40'38.13"
  • E12:
    • 71°16'28" N, 156°38'19" W

24 – ESnet Science Engagement (engage@es.net) - March 2023

25 of 38

ARM Maps for Barrow

25 – ESnet Science Engagement (engage@es.net) - March 2023

26 of 38

Use Case: Barrow AK

  • ARM Barrow AK Measurement Facility:
    • Connected via Arctic Slope Telephone Association Cooperative (ASTAC)
    • Current ISP connection terminates in the BARC (15 Mbps).
    • Fiber run to main site from BARC
    • Radio hop to our office space (Duplex) from BARC
  • Future Plans:
    • ASTAC has been promising fiber to the Duplex for a couple of years, but has not materialized.
    • ARM adding a smaller extended facility (E12) for students that ASTAC is quoting out a wireless/radio connection for

26 – ESnet Science Engagement (engage@es.net) - March 2023

27 of 38

Use Case: Barrow AK

  • Data Volumes & Experience:
    • Typically transmit about 3.5 TB/mo., but this doesn’t allow for much overhead or expansion.
    • It is quite cost prohibitive to add more bandwidth.
    • Appreciate the low latency now that there’s fiber there, but the cost is still exorbitant
    • Support experience with ASTAC not always positive, especially on the wireless/radio link

​

27 – ESnet Science Engagement (engage@es.net) - March 2023

28 of 38

Possible Future Plans

  • Starting to see Starlink service in higher latitudes
  • There is a service cell covering the area, and a downlink at Kuparuk
  • Starlink indicates service will be commercially offered Q1 2023

29 of 38

Topics

  • What / Who is ESnet
  • What / Who is EPOC
  • Science Engagements Basics
  • Three Scientific Use Case Categories
    • Supporting Remote Sensors with 5G/Wireless Edge
    • The Atmospheric Radiation Measurement (ARM)
    • NSF’s United States Antarctic Program
  • Conclusions

​

​

29 – ESnet Science Engagement (engage@es.net) - March 2023

30 of 38

NSF’s United States Antarctic Program

30 – ESnet Science Engagement (engage@es.net) - March 2023

•Preliminary analysis of McMurdo Station, AQ StarlinkBeta Test

•Data provided by NSF and Leidos/ASC

Data gathered via perfSONAR Measurement Framework

31 of 38

NSF’s United States Antarctic Program

32 of 38

Download Testing

32 – ESnet Science Engagement (engage@es.net) - September 2022

32 – ESnet Science Engagement (engage@es.net) - March 2023

33 of 38

Upload Testing

33 – ESnet Science Engagement (engage@es.net) - September 2022

33 – ESnet Science Engagement (engage@es.net) - March 2023

34 of 38

Latency Testing

34 – ESnet Science Engagement (engage@es.net) - March 2023

35 of 38

Topics

  • What / Who is ESnet
  • What / Who is EPOC
  • Science Engagement Basics
  • Three Scientific Use Case Categories
  • Conclusions

​

​

35 – ESnet Science Engagement (engage@es.net) - March 2023

36 of 38

IRI – Integrated Research Infrastructure

  • Effort led by Ben Brown, ESnet’s previous program manager at DOE
  • Seeks to create an ecosystem of interoperable computing and data
    • Lots of work to do this, across many facilities
    • ESnet has been involved since the beginning
      • ESnet requirements program provided process and technique basis
      • ESnet staff have participated in all phases
  • I’m not going to go into lots of depth here, but there are some key points:
    • IRI effort is likely to result in a long-term strategic program or set of programs at DOE, focused on “data” for multi-facility workflows
    • ESnet is central to the entire effort – we are the means by which the rest of the facilities are integrated
    • ESnet6 capabilities, especially automation, will be very valuable
    • Lots of good stuff for Network Services and Software to do in this space

​

36 – ESnet Science Engagement (engage@es.net) - March 2023

37 of 38

The vision: A DOE/SC integrated research ecosystem that transforms science via seamless interoperability

37 – ESnet Science Engagement (engage@es.net) - March 2023

Strategic goal: Broadening participation

AI-enabled insight from dynamic, vast

multi-modal data

Rapid data analysis and steering of experiments

Seamless user interconnectivity via federated IDs

Novel models for multi-facility allocation/utilization

New modes of integrated science

Researchers

Sensors

at the edge

Laboratory

Compute Assets

Testbeds

Experimental and Observational

User Facilities

Advanced Computing

Advanced Networking

Data Repositories

PuRE Data Assets

AI Tools

Digital Twins

Software and Applications

Advanced Data

Capabilities

Cloud Computing

Software

Advanced Data Management

Slide Credit: Ben Brown

38 of 38

ESnet & EPOC: Support for Remote Scientific Use Cases

Energy Sciences Network (ESnet)

Lawrence Berkeley National Laboratory

Jason Zurawski, Science Engagement Engineer

zurawski@es.net

March 29th, 2023

© 2022, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy). All rights reserved.

​

NOTICE. This material is owned by the U.S. Department of Energy. As such, the U.S. Government has been granted for itself and others acting on its behalf a paid-up, nonexclusive, irrevocable, worldwide license in the material to reproduce, prepare derivative works, and perform publicly and display publicly.