Chapter 17. Enabling multipathing on NVMe devices
Chapter 17. Enabling multipathing on NVMe devices You can multipath Non-volatile Memory ExpressTM (NVMeTM) devices that are connected to your system over a fabric transport, such as Fibre Channel (FC). You can select between multiple multipathing solutions. 17.1. Native NVMe multipathing and DM Multipath Non-volatile Memory ExpressTM (NVMeTM) devices support a native multipathing functionality. When configuring multipathing on NVMe, you can select between the standard DM Multipath framework and the native NVMe multipathing. Both DM Multipath and native NVMe multipathing support the Asymmetric Namespace Access (ANA) multipathing scheme of NVMe devices. ANA identifies optimized paths between the controller and the host, and improves performance. When native NVMe multipathing is enabled, it applies globally to all NVMe devices. It can provide higher performance, but does not contain all of the functionality that DM Multipath provides. For example, native NVMe multipathing supports only the numa and round-robin path selection methods. Red Hat recommends that you use DM Multipath in Red Hat Enterprise Linux 8 as your default multipathing solution. 17.2. Enabling native NVMe multipathing The default kernel setting for the nvme_core.multipath option is set to N , which means that the native Non-volatile Memory ExpressTM (NVMeTM) multipathing is disabled. You can enable native NVMe multipathing using the native NVMe multipathing solution. Prerequisites The NVMe devices are connected to your system. For more information, see Overview of NVMe over fabric devices . Procedure Check if native NVMe multipathing is enabled in the kernel: The command displays one of the following: N Native NVMe multipathing is disabled. Y Native NVMe multipathing is enabled. If native NVMe multipathing is disabled, enable it by using one of the following methods: Using a kernel option: Add the nvme_core.multipath=Y option to the command line: On the 64-bit IBM Z architecture, update the boot menu: Reboot the system. Using a kernel module configuration file: Create the /etc/modprobe.d/nvme_core.conf configuration file with the following content: Back up the initramfs file: Rebuild the initramfs : Reboot the system. Optional: On the running system, change the I/O policy on NVMe devices to distribute the I/O on all available paths: Optional: Set the I/O policy persistently using udev rules. Create the /etc/udev/rules.d/71-nvme-io-policy.rules file with the following content: Verification Verify if your system recognizes the NVMe devices. The following example assumes you have a connected NVMe over fabrics storage subsystem with two NVMe namespaces: List all connected NVMe subsystems: Check the active transport type. For example, nvme0 fc indicates that the device is connected over the Fibre Channel transport, and nvme tcp indicates that the device is connected over TCP. If you edited the kernel options, verify if native NVMe multipathing is enabled on the kernel command line: If you changed the I/O policy, verify if round-robin is the active I/O policy on NVMe devices: Additional resources Configuring kernel command-line parameters 17.3. Enabling DM Multipath on NVMe devices You can enable DM Multipath on connected NVMe devices by disabling native NVMe multipathing. Prerequisites The NVMe devices are connected to your system. For more information, see Overview of NVMe over fabric devices . Procedure Check if the native NVMe multipathing is disabled: The command displays one of the following: N Native NVMe multipathing is disabled. Y Native NVMe multipathing is enabled. If the native NVMe multipathing is enabled, disable it by using one of the following methods: Using a kernel option: Remove the nvme_core.multipath=Y option from the kernel command line: On the 64-bit IBM Z architecture, update the boot menu: Reboot the system. Using a kernel module configuration file: Remove the nvme_core multipath=Y option line from the /etc/modprobe.d/nvme_core.conf file, if it is present. Back up the initramfs file: Rebuild the initramfs : Reboot the system. Enable DM Multipath: Distribute I/O on all available paths. Add the following content in the /etc/multipath.conf file: Note The /sys/class/nvme-subsystem/nvme-subsys0/iopolicy configuration file has no effect on the I/O distribution when DM Multipath manages the NVMe devices. Reload the multipathd service to apply the configuration changes: Verification Verify if the native NVMe multipathing is disabled: Verify if DM multipath recognizes the nvme devices: Additional resources Configuring kernel command-line parameters Configuring DM Multipath
cat /sys/module/nvme_core/parameters/multipath grubby --update-kernel=ALL --args="nvme_core.multipath=Y" zipl options nvme_core multipath=Y cp /boot/initramfs-USD(uname -r).img /boot/initramfs-USD(uname -r).bak.USD(date +%m-%d-%H%M%S).img dracut --force --verbose echo "round-robin" > /sys/class/nvme-subsystem/nvme-subsys0/iopolicy ACTION=="add|change", SUBSYSTEM=="nvme-subsystem", ATTR{iopolicy}="round-robin" nvme list Node SN Model Namespace Usage Format FW Rev ---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- /dev/nvme0n1 a34c4f3a0d6f5cec Linux 1 250.06 GB / 250.06 GB 512 B + 0 B 4.18.0-2 /dev/nvme0n2 a34c4f3a0d6f5cec Linux 2 250.06 GB / 250.06 GB 512 B + 0 B 4.18.0-2 nvme list-subsys nvme-subsys0 - NQN=testnqn +- nvme0 fc traddr=nn-0x20000090fadd597a:pn-0x10000090fadd597a host_traddr=nn-0x20000090fac7e1dd:pn-0x10000090fac7e1dd live +- nvme1 fc traddr=nn-0x20000090fadd5979:pn-0x10000090fadd5979 host_traddr=nn-0x20000090fac7e1dd:pn-0x10000090fac7e1dd live +- nvme2 fc traddr=nn-0x20000090fadd5979:pn-0x10000090fadd5979 host_traddr=nn-0x20000090fac7e1de:pn-0x10000090fac7e1de live +- nvme3 fc traddr=nn-0x20000090fadd597a:pn-0x10000090fadd597a host_traddr=nn-0x20000090fac7e1de:pn-0x10000090fac7e1de live cat /proc/cmdline BOOT_IMAGE=[...] nvme_core.multipath=Y cat /sys/class/nvme-subsystem/nvme-subsys0/iopolicy round-robin cat /sys/module/nvme_core/parameters/multipath grubby --update-kernel=ALL --remove-args="nvme_core.multipath=Y" zipl cp /boot/initramfs-USD(uname -r).img /boot/initramfs-USD(uname -r).bak.USD(date +%m%d-%H%M%S).img cp /boot/initramfs-USD(uname -r).img /boot/initramfs-USD(uname -r).bak.USD(date +%m-%d-%H%M%S).img dracut --force --verbose systemctl enable --now multipathd.service devices { device { vendor "NVME" product ".*" path_grouping_policy group_by_prio } } multipath -r cat /sys/module/nvme_core/parameters/multipath N multipath -l eui.00007a8962ab241100a0980000d851c8 dm-6 NVME,NetApp E-Series size=20G features='0' hwhandler='0' wp=rw `-+- policy='service-time 0' prio=0 status=active |- 0:10:2:2 nvme0n2 259:3 active undef running `-+- policy='service-time 0' prio=0 status=enabled |- 4:11:2:2 nvme4n2 259:28 active undef running `-+- policy='service-time 0' prio=0 status=enabled |- 5:32778:2:2 nvme5n2 259:38 active undef running `-+- policy='service-time 0' prio=0 status=enabled |- 6:32779:2:2 nvme6n2 259:44 active undef running