2 of 24

HipNET'22: HPC on Linux

What it IS about

Adapting to Forwarding Data Path On Linux
An Approach To Rapid Development and Experimentation of Custom Stacks
Enabling High Precision Communication with Linux Networking

What this talk is NOT about

Hardware Offloading
Acceleration
Performance deep dive in HPC (perhaps next time)

3 of 24

Dependency on Industrial IoT Communications

HipNET'22: HPC on Linux

OT/IT Focus

Enabled by software + compute intensive technologies

Manufacturing and Factory operations

DATA DRIVEN

PROCESS CONTROL DRIVEN

Fast outcomes

Remote operation and automation lead to frequent process changes

Precise Outcomes

Human/Software to Software

(applications)

Machine to Machine

Operations

AI, ML, DLT

Programmable controllers to Field devices

HPC

Synchronized motion control tasks.

~ 1ms

Soft-real time tasks { 100 to 50} ms

Non real time tasks. ~ 100 ms

PLC to actuator tasks with responses

{ 50 ~ 10} ms

Used to be Weeks/months 🡪 now the want is hours/days

Precision required in order of milliseconds

non-real time tasks order of seconds

4 of 24

Navigating IIoT Networks

HipNET'22: HPC on Linux

Best Effort traffic. Applications will work sometime. No inherent real-time properties. Forget any degree of precision.

Path 1:

Path 2:

RT Ethernet, Isochronous Real-time.
Applications do not get right type of control (mapped to special ethernet frames).
No application to device integration.

Path 3:

Time Sensitive Networking (TSN) covers several aspects of high precision. Periodic flows, Asynchronous flows.

Process control channels

APPS..

..(process control apps)

TCP/UDP

Ethernet

iRT

TSN

OPC-UA

We are here

Exact delay guarantees??
Usability ??
Integration with higher layers??
Edge cloud continuum

5 of 24

Towards Converged IIoT Networks

HipNET'22: HPC on Linux

Integrated APPS

Automation

IIoT Transport

HPC in IP

Layer 2

Understand coexistence of traffic patterns
Provide means to express goals (e.g., closed control loop, periodicity, command-based E2E, etc.)

IIoT Centric Network

Reduce Abstraction Layers

Abstractions create layers and layers cause overheads.
Direct impact on the degree of precision you want to achieve

Unified Interface to HPC

Different latency guarantees
Support reliability, scheduling, shaping.

Get Here

6 of 24

HPC Design and Implementation

7 of 24

Contract Structure for HPC-Suite

HPC is not one thing. But a Set of directives.

Contract as a fundamental unit for ‘host to network’ and with ‘in network’ service-agreements.

HPC involves one or more contracts for different high-precision constraints and agreements. E.g.

A contract Instructs and drives the network – Latency Based Forwarding (LBF Contract)
Validates and verifies late signal notify- (HPC Transport)
Per-hop delays on a given path – (Path-latency)

HipNET'22: HPC on Linux

IP header

HPC

Contract

Payload

IIOT

Transport

Implement HPC-Suite as ‘contracts’

Latency Delivery Assurance

Late Signal Notify

Latency specific path specification

8 of 24

Latency Based Forwarding (LBF)

Algorithm designed by Toerless Eckert^LBF and Alexander Clemm^LBF
Delivers packets within bounded latency (both minimum and maximum)
Number of hops to determine how much time the packet should wait at a node
Experienced delay stored in the packet so no need of clock synchronization.

HipNET'22: HPC on Linux

Contract: LBF
Min, Max (delay)
Hop-count
Experienced-delay

User defined

runtime

Hop-count/minimum

Guided average

(Min – experienced-delay)/

remaining-hops

Incrementally determined time at each hop

Drop if it is late

(Max – experienced-delay) < 0

9 of 24

LBF Algorithm Example 1

4 nodes (0, 1, 2, 3); number of hops = 3 (0-1,1-2,2-3)

minimum delay = 300ms

no queueing delay

packet delayed by LBF = (Min – Ex)/hops

Helps queue estimate for how long packet dwells in a node.

Packet pacing – LBF Queue knows how packet is paced on an average.

HipNET'22: HPC on Linux

(300 - 0)/3 = 100ms

(300 - 100)/2 = 100ms

(300 - 200)/1 = 100ms

Queueing delay of est. 30ms, but 40ms incurred on node 1’s LBF queue

(300 - 30)/3 = 90ms

(300 – (90+40)/2 = 85ms

(300 – 90 -85)/1 = 125ms

10 of 24

Lightweight IIoT Transport

Authenticate endpoints mutually (not sessions)
Adopts a modular protocol header

‣ Both long and short protocol control headers are supported

‣ Closed-control loop (async message)

‣ Publish/Subscribe messages (topic based)

‣ Coordinate with Time-centric packet delivery (LBF).

‣ Do not wait for RTT Acks/Reply tell lateness.

Connectionless

Uses a ‘magic token’ instead of source/destination ports.
Only device authentication

Focus on time-based delivery, reliability, ordering.
No specific congestion control (AIMD or buffering/CWND are not useful)

HipNET'22: HPC on Linux

11 of 24

Challenges and Choice Implementing HPC

Limitations

Process level simulation did not give clean insights.
Discrete event simulation (OMNETs, NS3, etc.) have steep learning curve, and less insights into multi-hop behavior.
We needed end to end packet processing with delay guarantees
A reproducible and reconfigurable test and verification environment.

Platform choice - Linux

Isolation through Network name spaces provide built-in stable environment with a plethora of tools and features.
No deadline aware queue discipline
We could build the entire HPC infrastructure in a box

HipNET'22: HPC on Linux

“Start small, think big. Don't worry about too many things at once. Take a handful of simple things to begin with, and then progress to more complex ones. Think about not just tomorrow, but the future. Put a ding in the universe

” Steve Jobs

12 of 24

HPC packet processing pipeline on a router-node

HipNET'22: HPC on Linux

Queue Disciplines /LBF Scheduler

Ingress Packet

Lookup Tables

Packet

Classification

Forwarding Engine

Contract spec Engine

IPv4

IPv6

LBF

Path Latency

Notify

Transit/Runtime state

Egress processing

Packet out

state

Types of contract

params

state

Contract action and Metadata processing as XDP program

13 of 24

Main Components

Scapy - Host side API for packet generation
eXpress Data Path XDP – for contract metadata processing
Traffic Control-BPF (TC) - LBF queue discipline
Topology setup - Network Stack Tester (NeST)

HipNET'22: HPC on Linux

14 of 24

Host side API: Scapy and Socket Interface

Testing and Evaluation

Craft and send the packets from the virtual interface setup by NeST
Sniff and decode the packet at the receiver

HipNET'22: HPC on Linux

Application Interface

User side socket to insert and extract LBF contracts as part of IIoT transport

15 of 24

Designing with BPF for XDP+TC

BPF is the kernel built-in fast data path and Provides packet processing at the lowest point in the kernel
Custom designed network stack in two parts
XDP and TC utilize eBPF’s kernel support

HipNET'22: HPC on Linux

RxPath

IP forward: XDP

Forward ingress TC-BPF

qdisc: egress TC-BPF

TxPath (driver)

Packet

XDP Prog

TC Prog

TC QDISC

Ifindex, meta etc.

Programs

Kernel Hooks

16 of 24

Topology Setup using NeST

Assign IP addresses to the name-spaced node interfaces
Populate the routing tables (Quagga/RIP) – Kernel FIB is used to get egress interface
Format IP packet with contracts by providing necessary information like MAC addresses

HipNET'22: HPC on Linux

17 of 24

Guts of our Project - Network Stack Tester

NeST uses network namespaces for reproducible process of setting up networks

Build a virtual network topology
Support Linux supported routing protocols
Run experiments on the virtual network topology
Collect statistics
Plot results

NeST assigns random names to the network namespaces
Cleans up the environment (deletes network namespaces) on termination

HipNET'22: HPC on Linux

RoutingHelper(protocol="rip").populate_routing_tables()

18 of 24

What is supported today

HipNET'22: HPC on Linux

19 of 24

Wireshark dissector for HPC Contract in

HipNET'22: HPC on Linux

20 of 24

Test and Verification

FIFO queues packet in order of arrival
But based on different delays packets are rearranges in the Queue

HipNET'22: HPC on Linux

21 of 24

Extract several statistics (work in progress)

HipNET'22: HPC on Linux

Collect different types of statistics

Packet burst rate of different sizes
Effect of increasing traffic
Delay incurred by packets in qdisc
Backlog in bytes

To ask questions like

What happens with mix of traffic
Different LBF values
When do drops happen
Effect of queue build up

22 of 24

What is supported today (Contd.)

Create tests for

Different types of topologies, different queue disciplines
Both v6 and v4 streams, mixed or independently
Different packet rates and different sets of latencies
LBF Queue lengths are configurable
Per queue stats can be extracted using TC harness

Work in progress

Comparison/analysis of different sizes of packets, different load tests, queue lengths, variance.
Transport implementation

HipNET'22: HPC on Linux

We are here

23 of 24

References

HPC https://github.com/mohittahiliani/New-IP
XDP https://www.iovisor.org/technology/xdp
NeST https://nest.nitk.ac.in/docs/
Scapy https://scapy.readthedocs.io/en/latest/usage.html
Paper LBF High-Precision Latency Forwarding over Packet-Programmable Networks : https://ieeexplore.ieee.org/document/9110431

HipNET'22: HPC on Linux

help this project grow, please join us

24 of 24

Thank You

Please contact us for any questions, feedback, and comments

Mohit P. Tahiliani tahiliani@nitk.edu.in

Bhaskar Kataria bhaskar.181co213@nitk.edu.in

Deepta Devkota deeptadevkota.191cs117@nitk.edu.in

Shashank D shashankd.181co248@nitk.edu.in

Kiran M kiranm@futurewei.com

Acknowledgements: @ameyanrd, @rohit-mp, @lesliemonis

HipNET'22: HPC on Linux

Mohit Tahiliani

(NITK, India)

Bhaskar Kataria

(NITK India)