1 of 30

5G MANTRA: A Multi-Access Network Testbed

for Research on ATSSS

Matan Broner, Sangwoo Lee, Liuyi Jin, Radu Stoleru

Department of Computer Science and Engineering

Texas A&M University

2 of 30

Problem Statement

Smartphones continue to power high-bandwidth and low latency applications such as HD-video streaming, mobile online gaming, and video calls.�
5G-RAN delivers improvements to throughput and latency.�
Both MNO’s and end-users benefit from aggregating traffic to all available access networks (ie. WiFi).

Performance improvements
Cost savings
Security�

Existing 3GPP (4G-LTE) deployments do not achieve complete network aggregation. �
Researchers lack the necessary tools to experiment with network aggregation in a 5G context.

2

Smartphones are an ever increasing part of the traffic we generate over the internet, and they power high bandwidth low latency applications including video streaming, online gaming, and video calls

On the right we see the Average BW per user across multiple MNOs over a decade, with a continuing upward trend year to year

To support this increasing demand for performance, the 5G Radio Access Network promises improvements to throughput and latency

BUT at the same time, we know that users generally have access to multiple access networks, including WiFi

Agrregating traffic across all AN’s benefits both the MNO’s and end users for reasons including:

Improved performance (ie. more bandwidth available at a time)
Cost savings (ie. using less of a mobile data plan)
Security (ie. compromised AN can stop being used without disruption)

As we will discuss, 4G-LTE provides limited offloading functionality that does not achieve network aggregation

Also, researchers do not have the tooling needed to experiment with the features provided by 5G to improve on 4G offloading.

Speaking of offloading, let’s understand what it is and how it works.

3 of 30

4G - Carrier WiFi Offloading

4G-LTE deployments give users the ability to offload traffic from a 4G-RAN (MNO) to a WiFi access point.

Session traffic can flow over a single interface.
Traffic steering policies are static.
Performance is hindered due to unused bandwidth and an inability to react to degradation.

3

[1] 3GPP TR 22.934, "Feasibility study on 3GPP system to Wireless Local Area Network (WLAN) interworking," V15.1.0, June 2018.

[2] 3GPP TS 23.261, "IP flow mobility and seamless Wireless - Local Area Network (WLAN) offload; Stage 2," V17.1.0, June 2021.

Untrusted

WiFi Access

4G Core Network

ePDG

P-GW

SWu (802.11)

SWn

DN

SWm

SWx

AAA

HSS

S6b

SGi

4 of 30

5G - Core Network Background

4

PCF

DN

WiFi AP

N1

N2

N3

N2

N11

N7

N4

N6

UE

Core Network

UPF

Transport

Convertor

SMF

RAN (eNB/gNB)

3GPP Interface

Non-3GPP Interface

AMF

N1

N3IWF

NWDAF

User Plane Function (UPF)

Responsible for packet forwarding, traffic routing, and applying quality of service (QoS) policies in the user plane, ensuring efficient and reliable data delivery between the user equipment (UE) and the network.

5 of 30

5G - Core Network Background

5

PCF

DN

WiFi AP

N1

N2

N3

N2

N11

N7

N4

N6

UE

Core Network

UPF

Transport

Convertor

SMF

RAN (eNB/gNB)

3GPP Interface

Non-3GPP Interface

AMF

N1

N3IWF

NWDAF

Non-3GPP Interworking Function (N3IWF)

Enables interworking and mobility management between 5G and non-3GPP technologies, such as Wi-Fi, allowing seamless connectivity and handover between different network types.

6 of 30

5G - Core Network Background

6

PCF

DN

WiFi AP

N1

N2

N3

N2

N11

N7

N4

N6

UE

Core Network

UPF

Transport

Convertor

SMF

RAN (eNB/gNB)

3GPP Interface

Non-3GPP Interface

AMF

N1

N3IWF

NWDAF

Network Data Analytics Function (NWDAF)

Responsible for collecting and analyzing network data to provide insights and intelligence for various network functions, such as network optimization, traffic steering, and resource allocation.

7 of 30

5G - ATSSS

Access Traffic

Steering
Switching
Splitting�

Multi-Access PDU (MA-PDU) Sessions introduced in 5G.�
ATSSS rules used by UE and UPF for traffic management across multiple links.�
Access traffic has fine grained control, including:

Percentage of traffic sent on 3GPP and non-3GPP links
Prioritized links
Allowed loss on links (QoS and SLA enforcement)

7

3GPP Interface

Non-3GPP Interface

UPF

ATSSS Rules

MA-PDU

[1] 3rd Generation Partnership Project (3GPP). 2019.

Access to the 3GPP 5G Core Network (5GCN) via non-3GPP access networks. TS 24.502 (2019)

5G introduces ATSSS, aiming to provide true network aggregation for the first time in 3GPP deployments

ATSSS stands for Access Traffic

Steering, meaning choosing the best access network at any given time
Switching, meaning seamlessly handing over between access networks while maintaining performance
And Splitting, meaning true network aggregation where traffic is sent over multiple access networks in tandem\

To enable ATSSS, 5G introduces the Multi Access PDU session

ATSSS rules are control plane policies enforced by the UE and UPF when managing incoming and outgoing traffic across multiple available interfaces

On the right we see a high level view of an MA-PDU session, with ATSSS rules facilitating traffic transmission

These rules provide fine grained control for the first time, with the ability to control

The amount of traffic sent on individual links
Prioritizing specific links in different conditions
And QoS enforcement through allowed packet loss per link

As we can see, ATSSS provides the answer to a number of missing features in 4G’s WiFi offloading

Let’s now explore existing works in the areas of 5G core networks and ATSSS

8 of 30

State of the Art

Core Network Solutions

Open5GS

Continuation project of NextEPC: supports 4G-LTE and 5G NSA/SA (Release 16) but does not support MA-PDU sessions, as it does not implement an N3IWF.

Free5GC

Supports 5G SA (Release 15), as it implements an N3IWF. It does not support 4G-LTE or 5G NSA.

Multi-Access

[1] implements an end-to-end MPTCP proxy in user-space.

Codebase is based on MPTCP version 0 and is not maintained.

ATSSS

[2] provides design considerations for ATSSS, including how to structure child PDU sessions within a parent session.

Does not provide a reference implementation of ATSSS.

[3] documents experimentation with various packet schedulers within MA-PDU sessions.

Does not implement ATSSS and operates in Mininet as opposed to real-world scenario.

8

[1] Georg Hampel, Anil Rana, and Thierry Klein. 2013. Seamless TCP Mobility Using Lightweight MPTCP Proxy

[2] Jeounglak Ha and Young-Il Choi. 2019. Support of a Multi-access Session in 5G Mobile Network

[3] Hongjia Wu, Giuseppe Caso, Simone Ferlin, Özgü Alay, and Anna Brunstrom. 2021. Multipath Scheduling for 5G Networks: Evaluation and Outlook.

When it comes to core network solutions, there are two projects that are most widely used in 5G research

Open5GS: the successor to the widely popular NextEPC project, Open5GS maintains support for 4G, while also supporting 5G-NSA and 5G-SA.

With that said, Open5GS does not implement an N3iWF, and as such it does not support multi-access PDU sessions

On the other hand exists Free5GC, a 5G SA exclusive core network that does implement an N3IWF, meaning it supports non-3GPP sessions.

As you can see, these two core networks are both missing features that the other has.

Regarding existing multi-access technologies, reference 1 implements an MPTCP proxy in user space.

It suffers from a lack of maintenance and is outdated given it is built for MPTCP version 0

Finally, in the area of ATSSS research

Reference 2 details some conbsiderations in how child PDU sessions can be structured within a multi-access PDU session

The work is limited to design with no actual implementation

Furthermore, reference 3 experiments with a number of packet schedulers in order to understand which fits best with Multi-Access PDU sessions

ATSSS is not implemented and the experiments are conducted in Mininet, making them not applicable in real world network settings

9 of 30

State of the Art (Core Networks)

5G-MANTRA fills the gaps in existing core network solutions by supporting MA-PDU sessions while maintaining 3GPP backwards compatibility.

9

10 of 30

MPTCP

Multi-Path TCP�
Common approach for bandwidth aggregation at the Transport Layer, and selected by 3GPP as the de-facto enabling technology for ATSSS.�
MPTCP provides:

TCP “subflows” within a “parent” MPTCP connection → Child PDU sessions
QoS management and built-in congestion control per child connection. → ATSSS rules�

MPTCP is included as of Linux kernel version 5.6, and is actively updated with new kernel releases.

10

MPTCP Flow

TCP Subflow

9 Mbps.

2 Mbps.

7 Mbps.

[1] Alan Ford, Costin Raiciu, Mark J. Handley, and Olivier Bonaventure. 2013. TCP Extensions for Multipath Operation with Multiple Addresses. RFC 6824. 6824 (Jan. 2013)

[2] Jonathan Corbet. 2019. Upstreaming multipath TCP.

LWN.net (26 92019).

11 of 30

Transport Converter

MPTCP has not been widely adopted by global internet servers.

High costs of upgrading servers’ Linux kernels
Middleboxes tamper with connections �

RFC 8803 defines a method for UE’s to utilize MPTCP despite these current flaws.

A “Transport Converter” proxies MPTCP connections as TCP connections.�

The 5G MA-PDU standards state that a UPF must support a Transport Converter present to support multi-access sessions.

11

[1] Olivier Bonaventure, Mohamed Boucadair, Sri Gundavelli, SungHoon Seo, and Benjamin Hesmans. 2020. 0-RTT TCP Convert Protocol. RFC 8803. 8803 (July 2020)

[2] 3rd Generation Partnership Project (3GPP). 2019.

Access to the 3GPP 5G Core Network (5GCN) via non-3GPP access networks. TS 24.502 (2019)

MPTCP UE

Transport Converter

TCP Server

SYN[MP-CAPABLE]

SYN

SYN-ACK

SYN-ACK[MP-CAPABLE]

ACK

SYN[MP-JOIN]

SYN-ACK[MP-JOIN]

ACK

LTE

WiFi

A major problem with MPTCP is that it has not been widely deployed across internet servers for the over 10 years it has been around

This is due to:

High cost of upgrading servers to new versions of the Linux kernel
Middleboxes tampering with MPTCP connections

Being aware of this, RFC 8803 defines a method for a UE to utilize MPTCP while communicating with a server that does not support it

A Transport Converter serves as proxy for MPTCP to TCP connections

On the right we see the procedure for an MPTCP to TCP session establishment. The transport converter handles all MPTCP enabled traffic from both of the UE’s intergrfaces and handles the protocol conversion for both ends of the connection.

In order to support Multi Access PDU sessions, a UPF in the 5G core network must support a Transport Converter.

Let’s observe the complete network stack for Multi Access PDU sessions proxied over a Transport Converter

12 of 30

5G MANTRA - Network Stack (UE to DN)

12

13 of 30

Integrated Core Network

We combined the advantages of Open5GS and Free5GC.

All Open5GS NF’s
Free5GC’s N3IWF�

We modified Open5GS’s code to support:

MA-PDU sessions within the AMF
N3IWF registration as a gNodeB �

5G-MANTRA’s core network supports 5G-SA, 5G-NSA, and 4G-LTE.�
We utilized an NWu protocol dialer in user-space to enable the connection establishment functionality present in a UE’s kernel.

Two IPSec tunnels used for control plane and user plane traffic.

13

[1] Fasferraz. [n. d.].

GitHub - fasferraz/NWu-Non3GPP-5GC: NWu IKEv2/IPSec Dialer for 5GC / N3IWF — github.com. https://github.com/fasferraz/NWu-Non3GPP-5GC. ([n. d.]).

Our core network in takes the advantages of both Open5GS and Free5GC and combines them into an integrated core.

We use all of the network functions in Open5GS, and import the N3IWF from Free5GC

To do so, we made modifications to Open5GS’s source code to allow Multi Access PDU sessions to be established and for the N3IWF to be able to register itself

The result is a core network that supports 5G SA, 5G NSA, and 4G LTE, all while supporting multi-access PDU sessions.

On the right you can see a diagram of a session being established over WiFi. We utilized a protocol dialer running in user space to handle this connection establishment. This process would generally happen within the UE’s kernel.

Connecting initially to the N3iWF, the UE establishes two IPSec tunnels by establishing a secure session with the AMF. These tunnels are used to transport control and user plane data.

Let’s gain a better understanding of how our UE works overall…

14 of 30

Transport Converter - UE

We modified and used the first and only 5G MA-PDU compliant client, libconvert.

Uses LD_PRELOAD environment variable to intercept and redefine system calls: socket, connect, read, write.�
Facilitates the connection between a UE and the Transport Converter for proxying MPTCP traffic.�
Sockets using IPPROTO_MPTCP are redirected to our Transport Converter without the user’s intervention.�

14

libconvert

socket()

If IPPROTO_MPTCP, store socket for proxying

libconvert

connect()

Redirect socket to Transport Converter. Store original (IP, PORT) in SYN body.

15 of 30

Transport Converter - Server

15

Transport Converter

16 of 30

Transport Converter

Transport Converter - Server

16

Connection Proxy

Incoming Data

select()

New Connection

accept()

Forward Data

write()

5G CN

UPF

Web Server

LTE

WiFi

TCP

Transport Converter

MPTCP Subflow

TCP Flow

Convert Protocol Utility

17 of 30

Transport Converter

Transport Converter - Server

Connection Proxy

We implemented a split-connection proxy in order to maintain Linux kernel’s advanced MPTCP tracing capabilities (mptcp_info).

An end-to-end proxy must operate on the IP layer, requiring raw sockets or manually queuing and modifying packets.

17

18 of 30

Transport Converter

Transport Converter - Server

18

Connection Proxy

Incoming Data

select()

New Connection

accept()

Forward Data

write()

Performance Logger

MPTCP Info Utility

getsockopt(MPTCP_TCPINFO)

DB Client

5G CN

UPF

Web Server

LTE

WiFi

TCP

Transport Converter

Generate thread per-session

MPTCP Subflow

TCP Flow

Convert Protocol Utility

SQLite

19 of 30

Transport Converter

Transport Converter - Server

Connection Proxy

We implemented a split-connection proxy (ie. not standards compliant) in order to maintain Linux kernel’s advanced MPTCP tracing capabilities (mptcp_info).

An end-to-end proxy must operate on the IP layer, requiring raw sockets or manually queuing and modifying packets.

Maintains a collection of active connections in individual threads.�

Performance Logger

Stores tcp_info per active-subflow on a configurable interval.

Uses SQLite client as storage backend.

Linux kernel’s 6.x releases expose mptcp_subflow_data and mptcp_subflow_addrs objects, which provide MPTCP specific connection information.
Serves as a future data source for the NWDAF.

19

20 of 30

Transport Converter

Transport Converter - Server

20

Connection Proxy

Incoming Data

select()

New Connection

accept()

Forward Data

write()

Performance Logger

MPTCP Info Utility

getsockopt(MPTCP_TCPINFO)

DB Client

Web Interface

DB Client

Flask Server

5G CN

UPF

Web Server

LTE

WiFi

TCP

Transport Converter

Generate thread per-session

Session ID

MPTCP Info Plots

MPTCP Subflow

TCP Flow

Convert Protocol Utility

SQLite

21 of 30

Transport Converter

Transport Converter - Server

Connection Proxy

We implemented a split-connection proxy (ie. not standards compliant) in order to maintain Linux kernel’s advanced MPTCP tracing capabilities (mptcp_info).

An end-to-end proxy must operate on the IP layer, requiring raw sockets or manually queuing and modifying packets.

Maintains a collection of active connections in individual threads.�

Performance Logger

Stores tcp_info per active-subflow on a configurable interval.

Uses SQLite client as storage backend.

Linux kernel’s 6.x releases expose mptcp_subflow_data and mptcp_subflow_addrs objects, which provide MPTCP specific connection information.
Serves as a future data source for the NWDAF.�

Web Interface

Proof of concept for MPTCP subflow data utilization.
Provides live charts of configurable tcp_info attributes, enabling faster tracking of results for experiments.
Also uses of SQLite client.

21

22 of 30

Congestion

ATSSS Scenario: Steering

As the WiFi access network that the user is connected to becomes more congested, the LTE access network becomes optimal and more traffic is diverted away from WiFi.

22

23 of 30

Mobility

ATSSS Scenario: Switching

As the user distances themselves from a WiFi AP, their connection is uninterrupted due to an LTE access network that begins to handle increased traffic.

23

24 of 30

Offloading

ATSSS Scenario: Splitting

When the user connects to a WiFi access network during an active session, traffic is split across WiFi and LTE while maintaining required transfer rate.

All three scenarios demonstrate splitting.

24

25 of 30

Real World Scenario

Connect a laptop UE through commercial WiFi and LTE access networks in order to measure the performance impact of our Transport Converter.�

Transport Converter is deployed in AWS local zone in Houston in order to minimize hop to hop distance to Operator N’s EPC.
Web server deployed on AWS in us-east region.

25

LTE Interface

WiFi

Interface

Client

libconvert

RAN (eNB)

WiFi AP

WiFi DN

LTE EPC

MME

PGW

SGW

Transport Converter

Performance Logger

Proxy Server

Web Server

WiFi Subflow

LTE Subflow

TCP Flow

Operator N

26 of 30

Real World Scenario

Performance of Transport Converter compared against a direct connection an MPTCP enabled web server.

Upload completion time of varying file sizes
Download completion time of varying file sizes
Time to First Byte (TTFB) during file downloading.

26

27 of 30

Conclusions

5G-MANTRA is the first publicly available implementation of an ATSSS testbed that operates in real-world environments.

A Transport Converter design is proposed and implemented.

3GPP specification compliant.
Performance metric tracking capabilities across individual subflows.
Real time visualization provides experimental visibility.�

We show that 5G-MANTRA can accurately measure all ATSSS scenarios while not incurring performance degradation through its Transport Converter.

27

28 of 30

Q&A

28

29 of 30

Extra Slides

29

30 of 30

Network Stack (Access Networks)

30