Chapter 12. Configuring NVMe over fabrics using NVMe/TCP
Chapter 12. Configuring NVMe over fabrics using NVMe/TCP In an Non-volatile Memory ExpressTM (NVMeTM) over TCP (NVMe/TCP) setup, the host mode is fully supported and the controller setup is not supported. Note In RHEL 9, the native NVMe multipathing is enabled by default. Enabling DM multipathing is not supported with NVMe/TCP. 12.1. Configuring an NVMe/TCP host You can configure a Non-volatile Memory ExpressTM (NVMeTM) over TCP (NVMe/TCP) host by using the NVMe management command-line interface ( nvme-cli ) tool. Procedure Install the nvme-cli tool: This tool creates the hostnqn file in the /etc/nvme/ directory, which identifies the NVMe host. Find the nvme hostid and hostnqn : Use the hostid and hostnqn values to configure the NVMe/TCP controller. Check the status of the controller: Configure the host network for a newly installed Ethernet controller with a static IP address: Here, replace 192.168.101.154 with the host IP address. Since a new network is created to connect the NVMe/TCP host to the NVMe/TCP controller, execute this step on the controller too. Verification Verify if the newly created host network works correctly: Additional resources nvme(1) man page on your system 12.2. Connecting the NVMe/TCP host to the NVMe/TCP controller Connect the NVMeTM over TCP (NVMe/TCP) host to the NVMe/TCP controller system to verify that the NVMe/TCP host can now access the namespace. Note The NVMe/TCP controller ( nvmet-tcp ) module is not supported. Prerequisites You have configured an NVMe/TCP host. For more information, see Configuring an NVMe/TCP host . You have configured an NVMe/TCP controller using external storage software and the network is configured on the controller. In this procedure, 192.168.101.55 is the IP address of NVMe/TCP controller. Procedure Load the nvme-tcp module if not already: Discover the available subsystems on the NVMe controller: Here, 192.168.101.55 is the NVMe/TCP controller IP address and 192.168.101.154 is the NVMe/TCP host IP address. Configure the /etc/nvme/discovery.conf file to add the parameters used in the nvme discover command : Connect the NVMe/TCP host to the controller system: Verification Verify that the NVMe/TCP host can access the namespace: Additional resources nvme(1) man page on your system
dnf install nvme-cli cat /etc/nvme/hostnqn nqn.2014-08.org.nvmexpress:uuid:8ae2b12c-3d28-4458-83e3-658e571ed4b8 cat /etc/nvme/hostid 09e2ce17-ccc9-412d-8dcf-2b0a1d581ee3 nmcli device show ens6 GENERAL.DEVICE: ens6 GENERAL.TYPE: ethernet GENERAL.HWADDR: 52:57:02:12:02:02 GENERAL.MTU: 1500 GENERAL.STATE: 30 (disconnected) GENERAL.CONNECTION: -- GENERAL.CON-PATH: -- WIRED-PROPERTIES.CARRIER: on nmcli connection add con-name ens6 ifname ens6 type ethernet ip4 192.168.101.154/24 gw4 192.168.101.1 nmcli connection mod ens6 ipv4.method manual nmcli connection up ens6 nmcli device show ens6 GENERAL.DEVICE: ens6 GENERAL.TYPE: ethernet GENERAL.HWADDR: 52:57:02:12:02:02 GENERAL.MTU: 1500 GENERAL.STATE: 100 (connected) GENERAL.CONNECTION: ens6 GENERAL.CON-PATH: /org/freedesktop/NetworkManager/ActiveConnection/5 WIRED-PROPERTIES.CARRIER: on IP4.ADDRESS[1]: 192.168.101.154/24 IP4.GATEWAY: 192.168.101.1 IP4.ROUTE[1]: dst = 192.168.101.0/24, nh = 0.0.0.0, mt = 101 IP4.ROUTE[2]: dst = 192.168.1.1/32, nh = 0.0.0.0, mt = 101 IP4.ROUTE[3]: dst = 0.0.0.0/0, nh = 192.168.1.1, mt = 101 IP6.ADDRESS[1]: fe80::27ce:dde1:620:996c/64 IP6.GATEWAY: -- IP6.ROUTE[1]: dst = fe80::/64, nh = ::, mt = 101 modprobe nvme-tcp nvme discover --transport=tcp --traddr= 192.168.101.55 --host-traddr= 192.168.101.154 --trsvcid=8009 Discovery Log Number of Records 2, Generation counter 7 =====Discovery Log Entry 0====== trtype: tcp adrfam: ipv4 subtype: current discovery subsystem treq: not specified, sq flow control disable supported portid: 2 trsvcid: 8009 subnqn: nqn.2014-08.org.nvmexpress.discovery traddr: 192.168.101.55 eflags: not specified sectype: none =====Discovery Log Entry 1====== trtype: tcp adrfam: ipv4 subtype: nvme subsystem treq: not specified, sq flow control disable supported portid: 2 trsvcid: 8009 subnqn: nqn.2014-08.org.nvmexpress:uuid:0c468c4d-a385-47e0-8299-6e95051277db traddr: 192.168.101.55 eflags: not specified sectype: none echo "--transport=tcp --traddr= 192.168.101.55 --host-traddr= 192.168.101.154 --trsvcid=8009" >> /etc/nvme/discovery.conf nvme connect-all nvme list-subsys nvme-subsys3 - NQN= nqn.2014-08.org.nvmexpress:uuid:0c468c4d-a385-47e0-8299-6e95051277db +- nvme3 tcp traddr= 192.168.101.55 ,trsvcid=8009,host_traddr= 192.168.101.154 live optimized nvme list Node Generic SN Model Namespace Usage Format FW Rev --------------------- --------------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- /dev/nvme3n1 /dev/ng3n1 d93a63d394d043ab4b74 Linux 1 21.47 GB / 21.47 GB 512 B + 0 B 5.18.5-2