RDMA over Commodity Ethernet at Scale
Chuanxiong Guo, Haitao Wu, Zhong Deng, Gaurav Soni, Jianxi Ye, Jitendra Padhye, Marina Lipshteyn
Presented by Matthew Sickler
The Problem
TCP/IP, the prevalent transport/network stack in data center networks, was increasingly unable to keep up with DC workload demands:
�To address these shortcomings… RDMA over Commodity Ethernet (RoCEv2)
Background: RDMA (Remote Direct Memory Access)
RDMA is “a method of accessing memory on a remote system without interrupting the processing of the CPU(s) on that system.”
RDMA-accessible virtual memory must first be registered with the NIC, which involves pinning the corresponding physical pages in memory and providing the NIC with the virtual address translations.
RoCEv2 was designed with a lossless network requirement, meaning switches would not overflow and drop packets.
Background: PFC
RoCEv2 uses PFC (Priority-based Flow Control) to provide this lossless network.��PFC works hop-by-hop between Ethernet nodes:
Because these pauses must propagate hop-by-hop to the source of the congestion, other flows may be unfairly blocked.
Background: DCQCN
PFC pauses can lead to unfairness.
To reduce their frequency, RoCEv2 also uses per-flow DCQCN (Data Center Quantized Congestion Notification).
DCQCN does not guarantee a lossless network.
Discussion Questions
Does application-side logic stay the same, or do newer issues arise with using RDMA?
Would small requests and large transfers benefit equally from PFC, or could the delays from pausing traffic sometimes be worse than retransmitting lost packets?
VLAN-based PFC
Traditional VLAN-based PFC carries the packet priority in the VLAN tag, but:
DSCP-based PFC
…so they moved the packet priority to the IP header DSCP field.
Challenge: RDMA Transport Livelock
RDMA transport protocol assumes packets are not dropped due to congestion, but occasional drops can still occur, for instance due to bugs in switch SW and HW.
�Initially, the NICs used a retransmission scheme of go-back-0, restarting at the first packet, because it was stateless and easy to implement.
Their solution was to use a go-back-N retransmission scheme, which instead restarts at the first dropped packet, as this was still simple enough to implement in the NICs.
Challenge: PFC Deadlock
Deadlock was initially assumed impossible in a Clos network topology with up-down routing. Packets first travel up to a common ancestor then go down the path to the destination, so there should be no cyclic buffer dependency.
Recall that for a ToR switch to send an IP packet to a server, it must check two tables:
�ARP table timeouts are more expensive, as they are handled by the switch CPU, so the timeout value for ARP table entries is much greater than MAC table entries. This can lead to “incomplete” ARP entries, which lack an associated MAC table entry.
Challenge: PFC Deadlock
Upon an “incomplete” ARP entry (without an associated MAC table entry), the standard behavior is to flood the packet to all ports, but this can create a cyclic buffer dependency…
Possible solutions for handling “incomplete” ARP entry packets, instead of flooding:
Challenge: NIC PFC Pause Frame Storms
PFC can cause collateral damage to innocent flows due to head-of-line blocking.
In the worst-case scenario, a single malfunctioning NIC continually sends pause frames to its ToR switch.
This sets off a chain reaction, blocking the whole network.
Challenge: NIC PFC Pause Frame Storms
Their solution was two complementary and individually sufficient watchdogs:
Challenge: Slow Receiver Symptom
The authors thought the server NIC would not generate pause frames because the PCIe bandwidth to memory was greater than the NIC throughput. However, they still found receiving bottlenecks forming at the NIC.
Their solution in the NIC was to increase the page size, so the translation cache miss rate would lessen.��Moreover, they implemented dynamic buffer sharing in the switch, allow its queues to absorb more traffic before propagating the pause frames further in the network.
Discussion Question
Could a centralized, SDN-like architecture better solve these safety challenges? (For instance, could an SDN controller inspecting and managing PFC frames improve PFC storm handling? Could an SDN controller instead maintain ARP and MAC table entries to avoid packet drops?)
Management and Monitoring Capabilities
Configuration management and monitoring
PFC pause frame and traffic monitoring
RDMA Pingmesh
Latency Reduction
RoCEv2 shows a significant end-to-end tail latency reduction over TCP in production environments. In one highly-reliable, latency-sensitive service, RDMA’s 99th and 99.9th percentiles were 90us and 200us, while TCP’s 99th was 700us.
High Throughput
RoCEv2 was also shown to utilize the available network bandwidth without becoming a bottleneck.
Aggregate throughput reached 60% utilization of network capacity, a limitation consistent with ECMP routing hash collisions rather than one from RoCEv2.
High Throughput Degrades Latency
RoCEv2's low tail latency benefits worsen under high throughput. During another throughput experiment, RDMA tail latency jumped significantly, as queues became congested and pause frames were created.
Discussion Questions
How much of the latency benefit comes from kernel bypass, loss prevention, or congestion control?
What do modern RDMA protocols use instead of ECMP in order to maximize throughput?
Deployment and Incidents
Deployment was completed incrementally, beginning with a small lab network followed by test clusters. It was then introduced to production networks in stages: first at ToR level, then at the Podset level, and finally throughout the Spine switches.
A few incidents did hit their production networks:
Prior Work
There had been attempts to improve TCP latency within its existing framework, for example by tuning the retransmission timer. RoCEv2 instead tries a different approach, bypassing the OS kernel and avoiding packet drops entirely.�
Two other RDMA technologies:
Future Work: Lossless Network Assumption
The reliance on a lossless network (and the use of PFC) led to many safety challenges and several incidents on production networks. Future work that weakens this assumption, using programmable hardware for instance, will likely be able to avoid many of these issues.
PFC is limited by hop distance and the associated propagation delay, making it infeasible for inter-DC communications. Other solutions are needed for more generic use cases.
Discussion Questions
How could RoCEv2 be extended for inter-DCN communications? Has this been implemented in the past few years?
What other strategies could be used to ensure a lossless network besides PFC? What might their tradeoffs look like?
Given the complexity and risks of maintaining a lossless network with mechanisms like PFC, are the performance benefits of lossless RDMA worth it? Has recent progress in programmable hardware made it feasible to weaken or remove that requirement?