13.2. Transferring Data Using RoCE
13.2. Transferring Data Using RoCE RDMA over Converged Ethernet (RoCE) is a network protocol that enables remote direct memory access (RDMA) over an Ethernet network. There are two RoCE versions, RoCE v1 and RoCE v2, depending on the network adapter used. RoCE v1 The RoCE v1 protocol is an Ethernet link layer protocol with ethertype 0x8915 that enables communication between any two hosts in the same Ethernet broadcast domain. RoCE v1 is the default version for RDMA Connection Manager (RDMA_CM) when using the ConnectX-3 network adapter. RoCE v2 The RoCE v2 protocol exists on top of either the UDP over IPv4 or the UDP over IPv6 protocol. The UDP destination port number 4791 has been reserved for RoCE v2. Since Red Hat Enterprise Linux 7.5, RoCE v2 is the default version for RDMA_CM when using the ConnectX-3 Pro, ConnectX-4, ConnectX-4 Lx and ConnectX-5 network adapters. Hardware supports both RoCE v1 and RoCE v2 . RDMA Connection Manager (RDMA_CM) is used to set up a reliable connection between a client and a server for transferring data. RDMA_CM provides an RDMA transport-neutral interface for establishing connections. The communication is over a specific RDMA device, and data transfers are message-based. Prerequisites An RDMA_CM session requires one of the following: Both client and server support the same RoCE mode. A client supports RoCE v1 and a server RoCE v2. Since a client determines the mode of the connection, the following cases are possible: A successful connection: If a client is in RoCE v1 or in RoCE v2 mode depending on the network card and the driver used, the corresponding server must have the same version to create a connection. Also, the connection is successful if a client is in RoCE v1 and a server in RoCE v2 mode. A failed connection: If a client is in RoCE v2 and the corresponding server is in RoCE v1, no connection can be established. In this case, update the driver or the network adapter of the corresponding server, see Section 13.2, "Transferring Data Using RoCE" Table 13.1. RoCE Version Defaults Using RDMA_CM Client Server Default setting RoCE v1 RoCE v1 Connection RoCE v1 RoCE v2 Connection RoCE v2 RoCE v2 Connection RoCE v2 RoCE v1 No connection That RoCE v2 on the client and RoCE v1 on the server are not compatible. To resolve this issue, force both the server and client-side environment to communicate over RoCE v1. This means to force hardware that supports RoCE v2 to use RoCE v1: Procedure 13.1. Changing the Default RoCE Mode When the Hardware Is Already Running in Roce v2 Change into the /sys/kernel/config/rdma_cm directory to et the RoCE mode: Enter the ibstat command with an Ethernet network device to display the status. For example, for mlx5_0 : Create a directory for the mlx5_0 device: Display the RoCE mode in the default_roce_mode file in the tree format: Change the default RoCE mode: View the changes:
~]# cd /sys/kernel/config/rdma_cm ~]USD ibstat mlx5_0 CA 'mlx5_0' CA type: MT4115 Number of ports: 1 Firmware version: 12.17.1010 Hardware version: 0 Node GUID: 0x248a0703004bf0a4 System image GUID: 0x248a0703004bf0a4 Port 1: State: Active Physical state: LinkUp Rate: 40 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x04010000 Port GUID: 0x268a07fffe4bf0a4 Link layer: Ethernet ~]# mkdir mlx5_0 ~]# cd mlx5_0 ~]USD tree └── ports └── 1 ├── default_roce_mode └── default_roce_tos ~]USD cat /sys/kernel/config/rdma_cm/mlx5_0/ports/1/default_roce_mode RoCE v2 ~]# echo "RoCE v1" > /sys/kernel/config/rdma_cm/mlx5_0/ports/1/default_roce_mode ~]USD cat /sys/kernel/config/rdma_cm/mlx5_0/ports/1/default_roce_mode RoCE v1