Chapter 29. Overview of NVMe over fabric devices
Chapter 29. Overview of NVMe over fabric devices Non-volatile Memory Express (NVMe) is an interface that allows host software utility to communicate with solid state drives. Use the following types of fabric transport to configure NVMe over fabric devices: NVMe over fabrics using Remote Direct Memory Access (RDMA). For information on how to configure NVMe/RDMA, see Section 29.1, "NVMe over fabrics using RDMA" . NVMe over fabrics using Fibre Channel (FC). For information on how to configure FC-NVMe, see Section 29.2, "NVMe over fabrics using FC" . When using FC and RDMA, the solid-state drive does not have to be local to your system; it can be configured remotely through a FC or RDMA controller. 29.1. NVMe over fabrics using RDMA The following sections describe how to deploy an NVMe over RDMA (NVMe/RDMA) initiator configuration. 29.1.1. Configuring an NVMe over RDMA client Use this procedure to configure an NVMe/RDMA client using the NVMe management command line interface ( nvme-cli ). Install the nvme-cli package: Load the nvme-rdma module if it is not loaded: Discover available subsystems on the NVMe target: Connect to the discovered subsystems: Replace testnqn with the NVMe subsystem name. Replace 172.31.0.202 with the target IP address. Replace 4420 with the port number. List the NVMe devices that are currently connected: Optional: Disconnect from the target: Additional resources For more information, see the nvme man page and the NVMe-cli Github repository .
yum install nvme-cli modprobe nvme-rdma nvme discover -t rdma -a 172.31.0.202 -s 4420 Discovery Log Number of Records 1, Generation counter 2 =====Discovery Log Entry 0====== trtype: rdma adrfam: ipv4 subtype: nvme subsystem treq: not specified, sq flow control disable supported portid: 1 trsvcid: 4420 subnqn: testnqn traddr: 172.31.0.202 rdma_prtype: not specified rdma_qptype: connected rdma_cms: rdma-cm rdma_pkey: 0x0000 nvme connect -t rdma -n testnqn -a 172.31.0.202 -s 4420 # lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 465.8G 0 disk ├─sda1 8:1 0 1G 0 part /boot └─sda2 8:2 0 464.8G 0 part ├─rhel_rdma--virt--03-root 253:0 0 50G 0 lvm / ├─rhel_rdma--virt--03-swap 253:1 0 4G 0 lvm [SWAP] └─rhel_rdma--virt--03-home 253:2 0 410.8G 0 lvm /home nvme0n1 # cat /sys/class/nvme/nvme0/transport rdma nvme list nvme disconnect -n testnqn NQN:testnqn disconnected 1 controller(s) # lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 465.8G 0 disk ├─sda1 8:1 0 1G 0 part /boot └─sda2 8:2 0 464.8G 0 part ├─rhel_rdma--virt--03-root 253:0 0 50G 0 lvm / ├─rhel_rdma--virt--03-swap 253:1 0 4G 0 lvm [SWAP] └─rhel_rdma--virt--03-home 253:2 0 410.8G 0 lvm /home