Not Just Fast Data: �ESnet Networking Services to Enable Science

Jon-Paul Herron

Dale W. Carder

Ed Balas

Bruce Mah

John MacAuley

Overview of ESnet Advanced Services

Jon-Paul Herron

Department Head, ESnet Network Services

ESnet’s Service Philosophy: What Really Matters

• Infrastructure
• Reach: We have to be in the right places
• Security, Availability, & Performance: Our services need to work
• Flexibility & Agility: We have to be able to change quickly
• Visibility: We should be proactively transparent
• Users
• Understand Science Needs: We anticipate where science is going
• Collaborate: We work with our users to build just what they need

3

ESnet’s Service Approach

World Class Network Infrastructure:

Like a Lego factory, our reliable, high performance network is the basis for our services.

Science-focused Capabilities: Like Lego pieces, we build flexible services and capabilities using this reliable infrastructure.

Tailored Solutions:

Like Lego models, we co-design with science projects to put the pieces together to create just what’s needed to accelerate their science.

4

ESnet Infrastructure By the Numbers

​

5

Science-focused Capabilities

6

Automation

ESnet Service Menu

Co-designed

Solutions

• Enabled by Engagement
• Goal:
• On-demand solutions
• Integrated into workflows

7

Putting It All Together: Our Future Vision

8

Auto

API/UI

ESnet Cloud Services

Dale W. Carder

ESnet Network Engineering

ESnet-Enabled Public Cloud Examples

10

Nationwide Cloud Peering

11

+ More

Connect From Sites into Private Clouds

​

• Proven L2VPN Solution built upon OSCARS
• Dedicated Bandwidth
• Flexible
• Resilient
• Secure

12

ESnet Private Cloud Connect

13

13

Cloud Solution Co-Design

• Each cloud service instance tends to have unique needs
• Consultation & co-design process
• Connection to Site
• Handoff to Cloud
• Resilience at either end & through ESnet
• Security

14

Request

Design

Implement

Operate

ESnet Measurement & Analytics

Ed Balas <ebalas@es.net>

Measurement and Analytics Group Lead

“Heart-Healthy” for Cyberinfrastructure?

• Congestion, blockages, failures happen here too

​

• Measurement and monitoring is preventative care

We Inject Packets Rather than Dye

When looking for blockage, perfSONAR is the go to

• Throughput
• Loss
• latency

17

Hardware Replaced

Combine Observations with Context

Need to understand the relationships between measurements

• Location, topology,
• Associated organizations

18

Vascular System Extends Past ESnet

19

We are Part of a Larger Community

20

Collaboration for End-to-end View

Aggregate Behaviors via Metadata

22

Measure the Whole Stack / Pipeline

23

20+ Years of Measurement Services and Software

24

IPerf

High Touch

Current Initiatives

• End-to-end R&E visibility
• Create composite view of infrastructure
• Employ federated identity / zero trust for controlled sharing
• Improved mapping

​

• Innovative ops using AI/ML
• employ intuitive interfaces to complicated data
• Look for insights and anomalies

​

25

Stardust as a Service Concept

• Dedicated resource for community
• Production 24/7 system
• Provide an authenticated end to end view
• Role based access control

​

• Standardized Offering:
• One stop shop: Net, Host, App
• Grafana, Jupyter, my.es.net for analysis
• Appropriate ingest rate and data retention

​

• Key Features
• End-to-end view of pipelines
• View of all layers in stack
• Federated Auth for controlled sharing
• multi-tenancy supporting virtual orgs.
• Relationships between data types

26

LBNL

ANL

BNL

ORNL

SLAC

JLAB

LSST

LHC

AI-/ML-Based Analytics

• Forecast future resource use
• Natural Language Interfaces
• Incident similarity
• Incident summary

Have Needs or Ideas?

Catch me after the talk

28

High Touch: �High-Precision, Real-Time Visibility into Network Traffic

Bruce A. Mah, PhD

Software Engineer, Planning and Architecture Group

ESnet High-Touch

• High-precision real-time network telemetry and measurements
• Programmable hardware and software
• Network visibility to deliver better services
• Flow- and packet-level
• Line-rate for multiple links
• Network engineering, troubleshooting, security, etc.

30

ESnet6 High-Touch Platform Deployment

Approximately 40 deployment locations

Focus: Network perimeter coverage

Router packet mirroring allows 100% packet inspection

31

High-Touch Server Hardware Deployment

ESnet6 Core Router

FPGA Servers �(2 per hub site)

AMD Xilinx Alveo U280 FPGA

(1 per server)

Use Case #1: LHCONE

• Flow-level measurements of LHCONE traffic
• Data transfers are much slower than commonly believed
• No signs of packet loss / congestion

32

Use Case #2: LSST Troubleshooting

High-Touch hardware inferring loss and retransmission

33

Use Case #2: LSST Troubleshooting

Unsampled packet capture shows out-of-order data

34

Measurement and Analytics Integration

• Shared metadata for networks, sites
• High-Touch flow data available to Stardust
• Cross-checking flow data sources (High-Touch vs. IPFIX)
• Enables joint visualization across data sources
• Enables joint analytics

35

High-Touch Synopsis

• High-Touch uses a combination of programmable hardware and software to collect high-fidelity data on network traffic.�
• ESnet has developed and deployed this internal service with the goal of providing a better network experience.

36

Build Your Own Network for Science

John MacAuley

CSA and PPG Group Lead (Acting)

Our Philosophy

• Understand Science Needs: Anticipate where science is going
• We listen to our users through requirement reviews, public forums, and daily interactions.

​

• Collaborate: Work with our users to build what they need
• We develop foundational network services, add-on software services, and domain specific solutions.
• With our partners we rapidly prototype new concepts.

• Co-design: Consult, design, test, and support
• Deliver capabilities within the network that can be programmatically consumed by your scientific workflows.
• Provide solutions to control resources not only within ESnet, but wherever resources are consumed.

38

State of the Industry

• Science workflows engage cyber infrastructure to accomplish a complex set of tasks
• Rely on access to heterogeneous resources across distributed facilities.
• Desire workload mobility between facilities for diversity and availability.
• The Integrated Research Infrastructure (IRI) will provide key building blocks.

• Managing these diverse workloads can be overwhelming
• Workflows must communicate directly with target facilities when consuming resources, dealing with both diverse access methods, and varying behaviors.

• Perhaps a network-aware middleware solution is warranted to abstract complexity?
• Provide tools for end-to-end resource orchestration, enabling scientific workflows to dynamically interface with the full infrastructure stack while minimizing operational overhead.

39

The Evolving Research Infrastructure

40

The API-Driven SENSE Architecture

41

But John — this looks really complicated!

42

Use Case: SENSE and Rucio/FTS/XRootD

43

Summary

• The IRI vision is important to the future of the DOE complex�
• Defines a normalized set of services and resources, exposing via consistent API and models.
• Supports extensions for unique capabilities when needed.
• Demonstrates a working collaboration across facilities.

• SENSE provides our end users the capability to build their own virtualized network for science�
• Provides value added tooling to simplify the creation of end-to-end workflows.
• Exposes a simplified intent-based API to the end user, hiding the complexity of the cyber infrastructure.

• Collaboration and co-design are key to our combined success�
• Individually we cannot provide a complete end-to-end solution.
• Co-designed solutions are built with a shared sense of ownership.
• We align on goals, can anticipate challenges earlier, and integrate diverse perspectives.

44

To learn more about SENSE

45

SENSE Tutorial

Panel