1 of 26

Scitags: Packet Marking and Flow Labeling Update

Marian Babik / CERN

on behalf of the Research Networking Technical Working Group

HEPiX IPv6 Workshop

2 of 26

Technical Spec for Packet Marking/Flow Labeling

The detailed technical specifications are maintained on a Google doc

The spec covers both Flow Labeling via UDP Fireflies and Packet Marking via the use of the IPv6 Flow Label.

Fireflies are UDP packets in Syslog format with a defined, versioned JSON schema.

Packets are intended to be sent to the same destination (port 10514) as the flow they are labeling and these packets are intended to be world readable.
Packets can also be sent to specific regional or global collectors.
Use of syslog format makes it easy to send to Logstash or similar receivers.

Packet marking is intended to use the 20 bit flow label field in IPv6 packets.

To meet the spirit of RFC6437, we use 5 of the bits for entropy, 6 for activity and 9 for owner/experiment.

The document also covers methods for communicating owner/activity and other services and frameworks that may be needed for implementation.

HEPiX IPv6 Workshop

3 of 26

Last year in review

Flow Marking (UDP firefly) implementations

Xrootd 5.0+ supports UDP fireflies

Site admins can configure mapping of paths to experiments and user/roles to activities
Xroot protocol extension via scitag URL flag

dCache PoC now ready, supports UDP fireflies

Testing deployment at AGLT2 (backported to 7.2) with issues reported back to Tigran

Flow and Packet Marking

Flowd v1.0 released - packet and flow marking service

Independent service that can mark flows and packets for 3^rd party services

Collectors/Receivers

Initial receiver prototype developed by ESnet (available on scitags github)

Registry

Provides list of experiments and activities supported
Exposed via JSON at api.scitags.org

Flow id propagation

Work needed has been agreed with Rucio and FTS (tickets were submitted to follow up)

SC22 demonstration

HEPiX IPv6 Workshop

4 of 26

Packet and Flow Marking Demonstrations

During Supercomputing 22 in Dallas, we demonstrated a number of aspects of our packet and flow marking work.

We showed packet marking at 200 Gbps rates using flowd with both xrootd and iperf3.
Scinet and ESnet set up packet collectors via sflow and demonstrated real-time monitoring of packets by experiment and activity.
Demos were also run on LHCONE using equipment in the SC22 booth, KIT, UVic and CERN where packet marking for all transfers was monitored using a P4 programmable switch.

HEPiX IPv6 Workshop

5 of 26

Packet and Flow Marking Demonstrations

HEPiX IPv6 Workshop

6 of 26

Packet and Flow Marking Demonstrations

HEPiX IPv6 Workshop

7 of 26

Packet and Flow Marking Demonstrations

HEPiX IPv6 Workshop

8 of 26

Plans

We have a number of activities planned to get us from where we are to where we want to be for the Second WLCG Network Data Challenge (Feb/Mar 2024?):

RNTWG plans (https://indico.cern.ch/event/1244448/)

Storages - engage more storage technologies to adopt Scitags

dCache implementation - target SC for production demo
Engage with EOS, Echo, StoRM to understand their plans and challenges
Flowd in production on multiple XRootd, dCache systems

Propagation of the flow identifier in WLCG DDM

FTS and Rucio implementations
Engage with DIRAC and Alice O2

Collectors/Receivers

Establish production level network of receivers (ESnet, Jisc, GEANT ?)

Routing and forwarding using flow label in P4 testbed (MultiONE)

Draft plan for networking objectives and milestones

HEPiX IPv6 Workshop

9 of 26

Finding More Information: https://scitags.org

Code

Presentations

Technical Spec

Mailing List

HEPiX IPv6 Workshop

10 of 26

Acknowledgements

We would like to thank the RNTWG, WLCG, HEPiX, perfSONAR and OSG organizations for their work on the topics presented.

In addition we want to explicitly acknowledge the support of the National Science Foundation which supported this work via:

OSG: NSF MPS-1148698
IRIS-HEP: NSF OAC-1836650

HEPiX IPv6 Workshop

11 of 26

Conclusion

The RNTWG, driven by the needs and interests of the LHC, HEP and R&E networking communities, is implementing packet marking and flow labeling of network flows for all R&E network users

We have a well defined program of work and strong collaboration with storage and transfer application providers, WLCG experiments and sites.
Rucio will play a central role in the process.

Our goal is to have large scale packet marking and flow labeling in place by the time of the next WLCG Data Challenge

HEPiX IPv6 Workshop

12 of 26

Questions / Discussion

Questions, Comments, Suggestions?

HEPiX IPv6 Workshop

13 of 26

Flowd: News

Flow and Packet Marking service developed in Python

Aims to provide a framework to test/evaluate various flow/packet marking ideas
Architecture:

Plugins provide different ways get connections to mark (or interact with storage)

New plugins were added to support netlink readout and UDP firefly consumer

Backends are used to implement flow and/or packet marking

New backends were added to mark packets (via eBPF-TC) and expose monitored connection to Prometheus

HEPiX IPv6 Workshop

14 of 26

Flowd: eBPF-TC Backend

eBPF is a general-purpose RISC instruction set that runs on an in-kernel VM; programs can be written in restricted C and compiled into bytecode that is injected into the kernel (after verification)
Can sometimes replace kernel modules
eBPF-TC programs run whenever the kernel receives (ingress) or sends (egress) a packet

The flowd backend maintains a hash table of flows to mark. The plugin sends the backend (src address, dst address, src port, dst port); this is used as the key in the hash, and the flow label to put on the packets is the value
Each packet is inspected, and if the attributes match an entry in the hash, the corresponding flow label is put on the packet

TCP/IP stack

eBPF program

GSO

NIC

Egress path:

HEPiX IPv6 Workshop

15 of 26

Protocol Extensions

XRoot protocol extension uses flow.scitag cgi

//path/?scitag.flow=flow_id
flow_id = (exp_id << 6) | act_id

exp_id - experiment id; act_id - activity id (both as seen in registry)

HTTP-TPC extension

Adds additional HTTP request headers in the COPY request

COPY /path/to/destination HTTP/1.1�Host: destination.example.org�Source: https://source.example.org/path/to/source�TransferHeaderAuthorization: Bearer ABCD…�TransferFlowExperiment: exp_id�TransferHeaderFlowExperiment: exp_id�TransferFlowActivity: act_id�TransferHeaderFlowActivity: act_id

Both extensions to be implemented in GFAL2 libs

Xroot extension already implemented in xrootd clients v5.0+

HEPiX IPv6 Workshop

16 of 26

Introduction and Context

There have been a number of presentations about the packet marking, flow labeling and SciTags initiative given over the last year including recent presentations at the 3rd GRP, HEPiX and the LHCONE/LHCOPN meeting.

With only a short slot here, I don’t want to re-hash those presentations
The focus on this talk will be to cover some of the more relevant technical details surrounding how we are implementing the packet marking and flow labeling
I will have a quick introduction and then dive into some details.

HEPiX IPv6 Workshop

17 of 26

Packet Marking and Flow Labeling

To start the presentation I would like to motivate and define the goal the Research Networking Technical Working Group (RNTWG) is trying to achieve.

Motivation: The poor experience for WLCG trying to understand network flows, especially across the Atlantic, using just end site transfer and ESnet stats.

GOAL: To be able to identify the owner and purpose of any research and education network flow anywhere in the network.

WHY??: Many reasons:

Network links can become congested and it is vital to understand the sources of the traffic involved and work with users to better orchestrate.
R&E networks want to understand their users and associated flows and optimize how they are served.
Science collaborations are often unaware of the impact tuning or changes to their workflows have on the wide area network and the possible detrimental effects they are causing.

HEPiX IPv6 Workshop

18 of 26

The SciTags Initiative

To manage our packet marking and flow labeling efforts, we started the Scientific Network Tags (scitags): an initiative promoting identification of the science domains and their high-level activities at the network level.

The initiative is managed by the RNTWG and is working to:

Enable tracking and correlation of network transfers with Research and Education Network Providers (R&Es) network flow monitoring.
Supporting collaborations to better understand network use and impact

Improve visibility into how network flows perform (per activity) within R&E segments
Get insights into how experiment is using the networks, get additional data from R&Es on behaviour of our transfers (traffic, paths, etc.)

Allow sites and end users to get detailed visibility into how different network flows perform

Network monitoring per flow (with experiment/activity information)

E.g. RTT, retransmits, segment size, congestion window, etc. all per flow

HEPiX IPv6 Workshop

19 of 26

Rucio’s Role

Rucio will be an important part of the SciTags work for those VOs that use it.

Rucio critically has information about “activity” for its transfers, as well as knowing the VO making the request, both of which are need to mark packets or label flows.

The main challenge is enabling the right information to flow from the collaborations => Rucio => FTS => server moving the data (will show protocol extensions we are proposing in two slides)

Of course, a successful SciTags could also benefit Rucio storage monitoring, including augmenting the visibility of specific types of transfers anywhere in the network.

Future work may also allow intelligent network decisions based upon activity…

HEPiX IPv6 Workshop