N-DISE: NDN for Data Intensive Science

Experiments

Edmund Yeh

Large-Scale Data-Intensive Science

Challenges of Data-Intensive Science�

• Data-intensive applications face similar set of problems:

- LHC high energy physics, genomics, LSST, SKA, LIGO, EHT

​

• System challenges:

- Indexing, security, storage, distribution, analysis, learning

- Coordinated use of computing, storage, network resources

​

• Today: domain experts are dealing with these systems problems

- Incremental solutions; developed in isolation; replicated efforts

​

• Gap between application needs and existing networks/systems:

- Current computer networks/systems focus on addresses,

processes, servers, connections

- Applications care about data

​

​

​

​

Closing the Gap: Data-Centric Networking�

• Data-centric networks/systems design: Named Data Networking (NDN)

​

• Launched by $7.9M NSF Future Internet Architecture grant (2010)

​

• Name data instead of endpoints: paradigm shift.

​

• Pull-based data distribution architecture with stateful forwarding.

​

• Native support for caching and multicast

​

• Direct authentication and securing of data; controlled data access

​

• Website: http://named-data.net

​

​

Data-Centric Networking�

• Data-centric approach to system and network design

​

• Providing system support through the whole data lifecycle:

- Data production: naming, authenticating, securing data directly

- Delivering data using names enables scalable data retrieval

- in-network caching

- automated joint caching and forwarding

- multicast delivery

​

• Enables network-wide data-intensive computation and learning

​

• Common framework to support different application domains

​

​

​

NSF N-DISE and SANDIE Projects

• NSF CC* N-DISE: NDN for Data Intensive Science Experiments (2020-)

​

• Builds on NSF CC* SANDIE project (2017-2021)

​

• Team: Northeastern (PI: E. Yeh), Caltech Physics (Harvey Newman), UCLA CS (Lixia Zhang and Jason Cong), Tennessee Tech CS (Susmit Shannigrahi)

​

• Challenges:
• LHC program in HEP is one of world’s largest data intensive applications: handling one Exabyte by at > 170 sites
• LHC data volume to grow 10x due to High Luminosity LHC (2027): increased data complexity
• Human genome data, Earth Biogenome (~ exabyte range)
• Need to use diverse computation, storage, networking resources

​

• Approach: build data-centric ecosystem to provide interoperable, scalable, high-performance, robust and trustworthy solutions for heterogeneous data-intensive domains

​

​

​

​

N-DISE/SANDIE Results and Demos

• Development of data-centric naming scheme for fast access and efficient communication in HEP and other fields
• Implementation and integration of jointly optimal caching and forwarding algorithms developed by Northeastern (Yeh et al. 2014)
• Integration with high-speed NDN-DPDK forwarder developed by NIST (Shi et al. 2019)
• NDN-DPDK-based consumer and producer developed by Caltech (Iordache, et al. 2019)
• Integration with CMS via XRootD plugin by Caltech
• FPGA acceleration of NDN-DPDK forwarder by UCLA
• Genomics application: Kubernetes using NDN
• Results appeared in Proc. ACM ICN 2022

​

​

N-DISE WAN testbed

Throughput Test

• Test Topology: 2 consumers (SC22 booth, Starlight) and 1 producer(Caltech)

​

​

​

​

​

• 100 Gbps tagged VLAN connections to both consumers.
• Each NDN-DPDK forwarder uses 6 forwarding threads
• Each consumer server runs 18 consumer applications to request 18 1GB files named by CMS .
• Requested data is cached in advance in the DRAM of the Caltech server

​

Starlight

Caltech

SC22 booth

Consumer

& Forwarder

Consumer & Forwarder

3610

1875

Forwarder & Producer

3611

Throughput Test: Local

• Local Test at Caltech SANDIE-7
• With all data cached in advance in DRAM
• 18 NDNc consumers with AIMD congestion control, 8192 initial window, lifetime 500 ms.

​

​

​

​

​

Throughput Test

Throughput Test

• 18 consumer applications running in parallel at each consumer node.
• Each NDNc consumer: fixed pipeline with 8192 packets pipeline-size, Packet lifetime 500 ms

​

​

​

​

​

​

​

VIP caching Test

• Service Topology: 5 nodes are used.
• 20 GB cache space for each forwarder nodes.
• Files are 4GB each
• Test has three scenarios: nocache, ARC and VIP.
• Each NDNc consumer node requests 30 4GB files randomly with the zipf(1) distribution for 2 hours.

​

VIP caching Test

VIP caching Test

• Multi-path Topology VIP caching test: We use 4 servers in the test.
• Each forwarder node has 20GB cache capacity. UCLA server acts as a producer node.
• Each NDNc consumer node requests 30 4GB files randomly with the zipf(1) distribution for 2 hours.

VIP caching Test

• The test has three scenarios: nocache, ARC and VIP. Each

XRootD and the Open Storage System Plugin

• XRootD …

- is a software framework used for file management at CERN

- enables developers to implement various types of plugins (e.g., filesystem, caching, security)

- is written in C++, though there is ongoing work for a Golang version�

• Open Storage System plugin …

- is a C++ dynamic library that offers support for all related POSIX file system calls (e.g., open, close, read, opendir, readdir)

- bridges IO to/from Ceph File System without using a kernel

NDN-based XRootD OSS Plugin

• Developed by Catalin Iordache
• Embeds an NDNc consumer into an XRootD plugin
• Translates each file system call into an NDN name

/ndnc/xrootd/store/mc/file1/32=metadata for open and fstat

/ndnc/xrootd/store/mc/file1/v1/seg=3 for read

• Sends interests over the NDN network
• Receives data packets, validates them and replies to XRootD worker threads
• For read requests, producer will associate a segment number with an offset from a file to read depending on the predefined payload size

​

File Transfer Using NDN-based XRootD Plugin

• File transfer between Docker containers at the Caltech site�- A consumer running the plugin and an NDN-DPDK fileserver�- Same parameters as earlier throughput test (fixed pipeline with size 8192, packet lifetime of 500ms)�- Result is roughly equivalent to the performance of a single NDNc file transfer client (4.8 Gb/s)

​

NDNc File Transfer Client vs. XRootD OSS Plugin

• NDNc File Transfer Client

- Uses two threads: one for encoding interests, other for reading and decoding data

- The two threads operate asynchronously and do not communicate

- This approach eliminates overhead and benefits from parallelization, which leads to improved throughput

​

• XRootD OSS Plugin

- CMS configuration only allows synchronous requests

- Each worker thread will read 2MB of data at an offset from a file and will wait for the response before continuing, which limits throughput

FPGA Acceleration of NDN-DPDK Forwarder

Naming Compute and Data

• Goal: create infrastructure to transparently place data and compute in the network�
• We built a data lake using K8s pods that can be deployed anywhere
• Data inside the data lake can be accessed by names�
• We named the computes and mapped them to K8s namespaces
• Allows us to deploy compute anywhere based on names�
• The ability to name both data and compute allows workflows to treat the network as a distributed compute device

NDN Data Lake

NDN-TR70 - Utilizing NDN DPDK for Kubernetes Genomics Data Lake

Sankalpa Timilsina, Justin Presley, David Reddick, Susmit Shannigrahi, Tennessee Tech,�Xusheng Ai, Coleman Mcknight, Alex Feltus�

Compute Placement based on Names

Compute Placement based on Names

N-DISE Demo at SC22

N-DISE team:

​

Northeastern: E. Yeh, Y. Wu, V. Mutlu, Y. Liu

Caltech: H. Newman, C. Iordache, R. Sirvinskas, J. Balcas

UCLA: L. Zhang, J. Cong, S. Song, M. Lo

Tennesee Tech: S. Shannigrahi, S. Timilsina

NIST: D. Pesavento, J. Shi, L. Benmohamed

​

Special thanks to:

​

ESnet: T. Lehman

CENIC: S. Bellamine

StarLight: J. Mambretti, F. Yeh, J. Chen, S. Yu

Internet2: M. Zekauskas

RNP: M. Schwarz

N-DISE Long-Term Goals

• Deploy, commission first prototype production-ready data-centric petascale data distribution, caching, access, computation system serving major science programs

​

• LHC high energy physics program as leading target use case. BioGenome, human genome projects, ATLAS, LSST, SKA as future use cases

​

• Leverage data-centric protocols, high throughput forwarding/caching methods, hierarchical cache systems, rate-based congestion control

​

• Incorporate data integrity and provenance

​

• Integrated with FPGA acceleration subsystems

​

• Deliver LHC data over wide area network at throughputs ~ 100 Gbps

​

• Dramatically decrease download times by using optimizing caching

​

​

​

Thank you!

​