Mark Stout

T-Mobile

Using wireshark in LTE, and 5G networks…

#sf25us

About me?

• In wireless technology CDMA/LTE/5G since 1998
• Work on various OEM’s
• Cisco
• Ericsson
• Nokia
• Nortel

#sf25us

UE

eNodeB

eNodeB

MME

S-GW

P-GW

HSS

PCRF

Uu

Uu

X2

S1-U

Enhanced Packet Core (EPC)

S11

S6a

S5/S8

Gx

SGi

Internet

E-UTRAN

We will concentrate on packets entering, exiting, and within the highlighted portion.

S1-MME

UE User Equipment, Users phone, or device

eNodeB Radio Access side of the network

MME Local Mobility Management Entity. Only handles signaling for call setup, and certain hand offs

S-GW Serving Gateway. Mobility anchor point. Handles signaling, and data plane

P-GW PDN-GW. Packet Data Network Gateway. Provides IP, and session connectivity(UE Anchor).

HSS Home Subscriber Server. Houses subscriber related information, and provides Authentication, and Authorization.

PCRF Policy and Charging Function.

**Certain nodes are not represented for simplicity

#sf25us

Protocols we’re covering

MME

S-GW

P-GW

HSS

PCRF

S1-U

S11

S6a

S5/S8

Gx

SGi

Internet

S1-MME

S1ap

GTP-C

GTP-C

GTP-U

GTP-U

Diameter

Diameter

• SCTP – Session Control Transmission Protocol
• Diameter
• GTP – GPRS Tunneling Protocol
• S1ap – S1 Application Protocol
• Then 5G

SCTP

SCTP

UDP

UDP

UDP

TCP

#sf25us

SCTP �Stream Control Transmission Protocol RFC4960

• Not datagrams, nor Segments, SCTP uses Chunks.
• Chunk: Unit of information within an SCTP packet. Chunks contain either user data, or SCTP control data.
• Multiple chunks can be bundled into one SCTP packet up to the MTU size, except for the INIT, INIT ACK, and SHUTDOWN COMPLETE chunks.
• Endpoint: The logical sender/receiver of SCTP packets. In the context of a multi-homed host, an SCTP endpoint is represented to its peers as an association of multiple transport addresses, consisting of a set of eligible destination addresses for outbound SCTP packet transmission and a set of eligible source addresses for inbound reception
• SCTP uses stream as the logical channel for transporting in-order application messages. A stream is defined as unidirectional

​

#sf25us

Standard header

• Start of all SCTP headers

• Different Chunk types are defined starting at the 13^th Byte (96 bit)

#sf25us

SCTP �Stream Control Transmission Protocol RFC4960

• Association establishment and shutdown. SCTP uses a cookie mechanism in a four-way handshake to establish an association.
• The shutdown process is a three-way handshake.

​

#sf25us

Redundancy built in

Multihoming can be established at time of init ack.

​

This is how we dynamically build redundant route from Cell Site, to MME.

​

Redundant route start heart beat.

#sf25us

Initiation (INIT)

• Initiate Tag
• This value MUST be placed into the Verification Tag field of every SCTP packet.

9

#sf25us

Initiation Acknowledgement (INITACK)

• INIT ACK uses two extra variable parameters: The State Cookie and the Unrecognized Parameter:

10

#sf25us

Cookie Echo (COOKIE ECHO)

• Cookie is echo’d from cookie in INIT ACK

11

• Cookie Acknowledgement (COOKIE ACK)

#sf25us

SCTP �Stream Control Transmission Protocol RFC4960

• Selective Ack’s
• The acknowledgements carry all Transmission Sequence Number (TSN) numbers that have been received by one side with them.
• That is, there is a Cumulative TSN Ack value, that indicates all the data that has successfully been reassembled at the receivers side.
• there are Gap Blocks that indicate which segments of data chunks have arrived, with some data chunks missing in between.
• Path Monitoring
• HEARTBEAT chunks are sent over all paths. Each HEARTBEAT chunk has to be acknowledged by a HEARTBEAT-ACK chunk.
• Each path is assigned a state: active or inactive.
• The number of events in which heartbeats were not acknowledged within a certain time or the number of retransmission events exceeds a certain configurable limit, the peer endpoint is considered unreachable and the association will be terminated via an ABORT chunk.

​

#sf25us

Selective Acknowledgement (SACK)

13

#sf25us

SCTP �Stream Control Transmission Protocol RFC4960

• S1 peering setup

#sf25us

Selective Acknowledgement (SACK)

15

#sf25us

SCTP profile

• Create a button to remove heartbeats/acks
• Create columns to show the stream, gap count, and TSN
• Create a color rule that shows you when gaps are present
• Create a color rule that shows sctp layer fragments
• I always create a color rule that shows the start of a peer, or session.

#sf25us

Diameter

• Diameter is the protocol used within the EPC architecture for AAA (Authentication, Authorization, and Accounting), PCRF (Policy and Charging Rules Function), and HSS (Home Subscriber Server).
• Diameter is specified in RFC 3588 and then by the RFC 6733.
• The name is a pun on the name of the predecessor protocol, RADIUS(Remote Authentication Dial In User Service) - a diameter is twice the radius.

#sf25us

Diameter

• All data delivered by diameter is in the form of AVPs (Attribute Value Pairs). Some of these AVP values are used by the Diameter protocol itself, while others deliver data associated with particular applications that employ Diameter.

​

• Ability to exchange messages and deliver AVPs ( Attribute Value Pairs)

​

• Capabilities negotiation

​

• Error notification

​

• Extensibility, required in [RFC2989], through addition of new applications, commands, and AVPs

​

• Basic services necessary for applications, such as the handling of user sessions or accounting

#sf25us

Diameter

• I-Send-Conn-Req = SYN
• Wait-Conn-Ack = SYN, ACK
• I-Rcv-Conn-Ack = ACK
• I-Send-CER = send Capabilities Exchange Request
• I-Wait-I-CEA = ACK
• I-Rcv-CEA = receive Capabilities Exchange Answer

Peering State Machine

#sf25us

Diameter

• Capabilities-Exchange-Request CER 257
• Capabilities-Exchange-Answer CEA 257
• Device-Watchdog-Request DWR 280
• Device-Watchdog-Answer DWA 280
• Disconnect-Peer-Request DPR 282
• Disconnect-Peer-Answer DPA 282
• Abort-Session-Request ASR 274
• Abort-Session-Answer ASA 274
• Re-Auth-Request RAR 258
• Re-Auth-Answer RAA 258
• Session-Termination-Request STR 275
• Session-Termination-Answer STA 275
• Accounting-Request ACR 271
• Accounting-Answer ACA 271

​

All Diameter headers start with the same 20 bytes. After that AVP’s are added

Diameter Header information

#sf25us

Diameter

• Diameter AVP’s carry a lot of data is asci format. Use it to your advantage using the “frame contains xxx” as a filter
• You can also just click on the ascii portion of the “Packet Bytes” pane, and the AVP will open directly in the “Packet Details” view.

#sf25us

Diameter

• The AVP’s are grouped in a parent, child relationship. This relationship is setup in readable xml files in the <wireshark home>/diameter directory.
• This info is helpful when trying to find what AVP’s could be grouped under one parent, or to find documentation associated with that avp.

#sf25us

Diameter profile

• Diameter has multiple request, and answer types. These can be in different AVP’s. Don’t build multiple columns for that. Use the custom column, and use || to define different AVP’s.
• Create custom color rule for known bad issues, then show color rule in column.

#sf25us

Diameter profile

Dictionary.xml is the main file, and is responsible for calling all the other specific files

<avp name="Vendor-Id" code="266" mandatory="must" may-encrypt="no" protected="may" vendor-bit="mustnot">

<type type-name="VendorId"/>

</avp>

<avp name="Firmware-Revision" code="267" mandatory="mustnot" protected="mustnot" may-encrypt="no" vendor-bit="mustnot">

<type type-name="Unsigned32"/>

</avp>

<avp name="Result-Code" code="268" mandatory="must" may-encrypt="no" protected="mustnot" vendor-bit="mustnot">

<type type-name="Enumerated"/>

<enum name="DIAMETER_MULTI_ROUND_AUTH" code="1001"/>

<enum name="DIAMETER_SUCCESS" code="2001"/>

#sf25us

Diameter profile

<avp name="Charging-Rule-Name" code="1005" mandatory="must" may-encrypt="yes" protected="may" vendor-bit="must" vendor-id="TGPP">

<type type-name="OctetString"/>

</avp>

<avp name="Event-Trigger" code="1006" mandatory="must" may-encrypt="yes" protected="may" vendor-bit="must" vendor-id="TGPP">

<type type-name="Enumerated"/>

<enum name="SGSN_CHANGE" code="0"/>

<enum name="QOS_CHANGE" code="1"/>

<enum name="RAT_CHANGE" code="2"/>

<enum name="TFT_CHANGE" code="3"/>

<enum name="PLMN_CHANGE" code="4"/>

<enum name="LOSS_OF_BEARER" code="5"/>

</avp>

In this example <avp starts the a new description

name= defines the name that will show in wireshark for the code= defined

type-name=“Enumerated = a number value that will later get a name assigned to the value

OctetString = A string of octets ie. 0a ff cd bb

UTF8String = text

if “Enumerated” is defined then <enum name=“what ever” code=“0”/> associates a string with whatever number value is seen.

​

​

Mandatory, May-encrypt, Protected, Vendor-bit = Not used in wireshark, but refers to the flags in the diameter message

#sf25us

Diameter profile

<avp name="Charging-Rule-Install" code="1001" mandatory="must" may-encrypt="yes" protected="may" vendor-bit="must" vendor-id="TGPP">

<grouped>

<gavp name="Charging-Rule-Definition"/>

<gavp name="Charging-Rule-Name"/>

<gavp name="Charging-Rule-Base-Name"/>

<gavp name="Bearer-Identifier"/>

<gavp name="Rule-Activation-Time"/>

<gavp name="Rule-Deactivation-Time"/>

<gavp name="Resource-Allocation-Notification"/>

</grouped>

</avp>

​

<avp name="Charging-Rule-Base-Name" code="1004" mandatory="must" may-encrypt="yes" protected="may" vendor-bit="must" vendor-id="TGPP">

<type type-name="UTF8String"/>

</avp>

#sf25us

GTP

• GPRS Tunneling protocol is a IP/UDP based protocol used in GSM, UMTS and LTE core networks. It is used to encapsulate user data when passing through core network and also signaling traffic between various core network entities.
• In LTE, version 2 is used for GTP-C and version 1 is used for GTP-U
• In our shown LTE network implementation, GTP-v2 is used on S5 and S11 interfaces and GTPv1 is used on S1-U, S5.

MME

S-GW

P-GW

HSS

PCRF

S1-U

S11

S6a

S5/S8

Gx

SGi

Internet

S1-MME

GTP-C

GTP-C

GTP-U

GTP-U

#sf25us

GTPv2

• Different GTP-C messages are used to manage a Tunnel or to set up the tunnel for GTPv1.
• GTP-C does not necessarily tunnel messages, as no payload is carried above GTP-C.
• The GTP-C messages are Create Session, Create Bearer, Modify Bearer, Update Bearer, Delete Session, Delete Bearer.�

#sf25us

GTPv1

• The transport bearer is identified by the GTP-U TEID and the IP address (source TEID, destination TEID, source IP address, destination IP address)
• As Bearer is received over a TEID on S1-U at the SGW, the SGW then forwards on a different TEID towards the PGW.

#sf25us

GTP

• While GTP is stateless the upper level protocols may not be.
• GTPv2 should be diagnosed from request to response (or lack there of).
• Looking at TCP over GTPv1 holds the best advantages for me. But there are things to look out for!

• When you take a capture at the SGW, and PGW. You will see that wireshark shows 3 packets per 1. Then Wireshark will do TCP analysis on 3 packets, and mark all packets (except the first) with some error (retran, ooo)
• You’ll also notice that WS will only show the upper most layer IP in the source/Destination field

#sf25us

GTP Tunnel Viewing Version1

Using the previous example, the 1^st thing I want to do is look at my GTP endpoints and not just the upper layer IP

1. Create a new column name “tunnel_source”
2. Choose type “custom”, and fields as ip.src
3. Make occurrences “1” (this is for single tunnel, you can do 2 when you have a tunnel in a tunnel
4. You now see the tunnel_source address, and the upper layer address. (only “Source” will resolve name)

#sf25us

GTP Tunnel Viewing Version2

Using the previous example, the 1^st thing I want to do is look at my GTP endpoints and not just the upper layer IP

1. Create a new column name “tunnel_source”
2. Choose type “custom”, and fields as ip.src_host#1
3. Change “Source” column to “Inner Source” and make the fields as ip.src_host#2
4. You now see the tunnel_source address, and the upper layer address.

(both addresses will be resolved using the _host field)

#sf25us

GTP

2^nd, we need to de’dup the packets so that we can use the TCP analysis feature in wireshark (assuming we’re looking for things like packet loss).

Putting a display filter does not help in the cases where you capture S1-u, S5, and Sgi in the same capture. Wireshark analysis feature is based on all packets in the capture regardless of whether you are view them or not.

1. Find something unique in each of the interfaces. I usually use the MAC address or tunnel IP, since S1, S5, and Sgi will typically use different interfaces. Then create a display filter for that.
2. This will now show you just show just a single flow of the TCP packets (still showing error). So now “export specified packets”, and save “displayed”
3. Open the capture you just created.

​

AFTER

BEFORE

Original File

#sf25us

Let’s build a profile

#sf25us

Sharing tunneled payload

#sf25us

Editcap to remove true

.\editcap.exe -w –I .0001 C:\temp\roamingIssue.pcap C:\temp\roamDedup.pcap

153 packets seen, 12 packets skipped with duplicate time window equal to or less than 0.000100000 seconds.

duplicate packets. Most useful on S5 interface that leaves SGW, and arrives at PGW.

​

​

For large files use capinfos to determine size, and calculate the size of the chunks that the pcap can be broken up into.

​

Mergecap…..

Tools, Tools, Tools, they are there for you.

#sf25us

Slicing packets in large traces file to reduce the file size

editcap -s 128 superbowl.pcap superbowl_128.pcapng

​

​

Take a larger trace and slice the packets after byte 128 and save to a new file. This means I won’t get the complete payload, but that is ok in the cases where your troubleshooting packet loss using TCP analysis. You can always go back to the original trace file if you need the payload. This will dramatically reduce the size of a trace. 

​

Split a large file into many smaller ones.

​

editcap -c <number of packets per file> superbowl.pcap smallSuperbowl.pcap

​

Becomes smallSuperbowl_0000.pacp _0001.pcap,

smallSuperbowl_0000.pacp _0002.pcap,

etc.

#sf25us

​

Recommend doing this before deduping packets.

​

tshark.exe -r 4perToBeDeduped.pcap -T fields -e tcp.seq_raw | sort | uniq -c

4 1000461189

4 1218696134

4 1218697032

4 1218698412

4 1218699792

4 1218701172

1 2689484057

1 2689485425

1 2689486793

1 2689488161

4 2710474120

4 2710476617

4 2746774803

​

​

-T <def> = defines output type (fields, text, ps). When using –e this MUST be <fields>

-e <field> = which field to print. Must be repeated print multiple fields

Verify each packet has hit every tap

#sf25us

Use a common host file, and distribute to all my profiles using a batch file

#sf25us

IMS / Volte

Volte is really nothing more that SIP/RTP (VoIP) over LTE with a guaranteed bearer and more favorable DSCP markers on the packet.

​

Most tools for SIP will work with trouble shooting Volte

​

#sf25us

A quick view into the nature, and timing of RTP packets.

​

Knowing your RTP timers help when using these tools.

​

Example; I know in this case active conversation packets are 20ms, and silent packets are 160ms

RTP Stream Analysis

#sf25us

IO Graph for jitter/packet arrival

1. Find all RTP streams in the capture.
2. Pick one side of the stream you are looking for, then click “Find Reverse”.
3. Click “Prepare Filter”.
4. Go back to Wireshark main screen, and click the arrow in the display filter portion, to execute the filtering of the streams.

​

1

4

2

3

#sf25us

Note the left and right sides of the || operator

#sf25us

CTRL-V or

Left side of conversation filter

To get display filter expression

• Open IO Graph

​

• We know RTP active packet time is 20ms, so we’ll set interval to 1ms.

​

• Our Y field will be udp.time_delta which will give us time of arrival delta between each packet that matches our display filter

​

#sf25us

Left graph depicts what we expect to see. That is 20ms time on active packets, and 160ms on silent packets. Using this graph you can tell who was speaking during this conversation.

Right graph is showing the Caller with an active call time at 40ms instead of 20ms, with spikes down to 0ms. Hmm.

​

#sf25us

Set time format to “seconds since previous displayed packet”

Adjust display filter to just show caller.

We can now see that the caller sends 2 packets every 40ms. That is one at the 40ms mark, then another one within the same sub-millisecond.

​

??

​

This was found to be uplink packet time scheduling on the eNodeB. Packets could only be requested on the 40ms timeslot. So UE would send 2 at a time.

#sf25us

4G to 5G

What was combined functions on 4G have now been separated, and segregated into their own network functions

​

Functions are now treated like services, and can be called upon with API like queries.

#sf25us

• Authentication Server Function (AUSF) acts as an authentication server. (part of HSS from EPC world)
• Unified Data Management (UDM) supports: generation of Authentication and Key Agreement (AKA) credentials, user identification handling, access authorization, subscription management. (part of HSS functionality from EPC world)
• Application Function (AF) supports: application influence on traffic routing, accessing NEF, interaction with policy framework for policy control. (same as AF in EPC world)
• Network Exposure function (NEF) supports: exposure of capabilities and events, secure provision of information from external application to 3GPP network, translation of internal/external information. (not present in EPC world)
• NF Repository function (NRF) supports: service discovery function, maintains NF profile and available NF instances. (not present in EPC world)
• Network Slice Selection Function (NSSF) supports: selecting of the Network Slice instances to serve the UE, determining the allowed NSSAI, determining the AMF set to be used to serve the UE. (not present in EPC world)

• Access and Mobility Management function (AMF) supports: Termination of NAS signaling, NAS ciphering & integrity protection, registration management, connection management, mobility management, access authentication and authorization, security context management. (AMF has part of the MME functionality from EPC world)
• Session Management function (SMF) supports: session management (session establishment, modification, release), UE IP address allocation & management, DHCP functions, termination of NAS signaling related to session management, DL data notification, traffic steering configuration for UPF for proper traffic routing. (SMF has part of the MME and PGW functionality from EPC world)
• User plane function (UPF) supports: packet routing & forwarding, packet inspection, QoS handling, acts as external PDU session point of interconnect to Data Network (DN), and is an anchor point for intra- & inter-RAT mobility. (UPF has part of the SGW & PGW functionality from EPC world)
• Policy Control Function (PCF) supports: unified policy framework, providing policy rules to CP functions, access subscription information for policy decisions in UDR. (PCF has part of the PCRF functionality from EPC world)

​

NETWORK FUNCTIONS

#sf25us

User plane protocols unchanged

The highlighted part still uses GTP, like the previous 4G slides

#sf25us

RAN Signaling

S1AP has changed to NGAP, but is the same type of informational elements, just more

​

Also is still on top of SCTP

#sf25us

Core signaling over http2

• 4G diameter interfaces have now become JSON over HTTP2 (API like queries)

#sf25us

HTTP2 information

• Must have wireshark 3.4 or greater
• Decode As: HTTP2 content type in stream, enter value, and set as JSON
• Currently must be done

for each stream ID

#sf25us

JSON over HTTP2

Current WS versions require you to filter like this

This is not great, because it doesn’t match a key value pair.

So any key with a value of “ACTIVATING” could be

matched, as long as the key upCnxState exist.

Development version allows for absolute filters

While this allows for more exact filtering, there is still work to be done.

#sf25us

LUA dissector pt1

• Provided to you:
• tvb: Protocol data access
• pinfo: Packet meta-info
• tree: detail tree
• You provide:
• Header info

​

​

#sf25us

LUA dissector pt2

-- Append "NAT64" address to the Info column with a post-dissector, and create it's own tree.

�-- create a new protocol so we can register a post-dissector

local myproto = Proto("nat64","NAT 64 encapsulated IPv4")

-- create a new field so we can assign that to the Proto

myproto_source = ProtoField.string("nat64.ipv4", "IPv4 Address")

-- Assign field to the proto object.

myproto.fields = { myproto_source }

-- register our new dummy protocol for post-dissection

register_postdissector(myproto) 

�--local nat64_field = Field

�-- the dissector function callback

function myproto.dissector(tvb,pinfo,tree)

--

end    

​

#sf25us

LUA dissector pt3

function myproto.dissector(tvb,pinfo,tree)

local  dest = tostring(pinfo.dst)

    local source = tostring(pinfo.src)

    if string.find(dest, "7700") then

        first, second = string.match(dest, "%d+:%d+:%d+:%d+:%d+:%d+:(.*):(.*)$")

        firstOct, secondOct = string.match(first, "(..)(.+)")

        thirdOct, fourthOct = string.match(second, "(..)(.+)")

        ipv4Address = tonumber(firstOct, 16) .. "." .. tonumber(secondOct,16) .. "." .. tonumber(thirdOct,16) .. "." .. tonumber(fourthOct,16)

        pinfo.cols.info:append( " NAT64 " .. tonumber(firstOct, 16) .. "." .. tonumber(secondOct,16) .. "." .. tonumber(thirdOct,16) .. "." .. tonumber(fourthOct,16))

local subtree = tree:add(myproto,tvb(), "NAT64 v4")

        subtree:add(myproto_source, ipv4Address)

--repeat for source

     else

 end

​

#sf25us

-- Append "NAT64" address to the Info column with a post-dissector, and create it's own tree.

�-- create a new protocol so we can register a post-dissector

local myproto = Proto("nat64Test","NAT1 64 encapsulated IPv4")

myproto_source = ProtoField.string("nat64.ipv4", "IPv4 Address")

myproto.fields = { myproto_source }

�-- the dissector function callback

function myproto.dissector(tvb,pinfo,tree)

    local  dest = tostring(pinfo.dst)

    local source = tostring(pinfo.src)

    if string.find(dest, "7700") then

        first, second = string.match(dest, "%d+:%d+:%d+:%d+:%d+:%d+:(.*):(.*)$")

        firstOct, secondOct = string.match(first, "(..)(.+)")

        thirdOct, fourthOct = string.match(second, "(..)(.+)")

        ipv4Address = tonumber(firstOct, 16) .. "." .. tonumber(secondOct,16) .. "." .. tonumber(thirdOct,16) .. "." .. tonumber(fourthOct,16)

        pinfo.cols.info:append( " NAT64 " .. tonumber(firstOct, 16) .. "." .. tonumber(secondOct,16) .. "." .. tonumber(thirdOct,16) .. "." .. tonumber(fourthOct,16))

        local subtree = tree:add(myproto,tvb(), "NAT64 v4")

        subtree:add(myproto_source, ipv4Address)

    elseif string.find(source, "7700") then

        first, second = string.match(source, "%d+:%d+:%d+:%d+:%d+:%d+:(.*):(.*)$")

        firstOct, secondOct = string.match(first, "(..)(.+)")

        thirdOct, fourthOct = string.match(second, "(..)(.+)")

        ipv4Address = tonumber(firstOct, 16) .. "." .. tonumber(secondOct,16) .. "." .. tonumber(thirdOct,16) .. "." .. tonumber(fourthOct,16)

        pinfo.cols.info:append( " NAT64 " .. tonumber(firstOct, 16) .. "." .. tonumber(secondOct,16) .. "." .. tonumber(thirdOct,16) .. "." .. tonumber(fourthOct,16))

        local subtree = tree:add(myproto,tvb(), "NAT64 v4")

        subtree:add(myproto_source, ipv4Address)

    else

    end

 end    

               �-- register our new dummy protocol for post-dissection

register_postdissector(myproto)   

#sf25us

#sf25us

https://conference.wireshark.org/sharkfest-25-us-2024/talk/WDQNXL/feedback/

#sf25us