Chapter 28. Configuring quorum devices
Chapter 28. Configuring quorum devices You can allow a cluster to sustain more node failures than standard quorum rules allows by configuring a separate quorum device which acts as a third-party arbitration device for the cluster. A quorum device is recommended for clusters with an even number of nodes. With two-node clusters, the use of a quorum device can better determine which node survives in a split-brain situation. You must take the following into account when configuring a quorum device. It is recommended that a quorum device be run on a different physical network at the same site as the cluster that uses the quorum device. Ideally, the quorum device host should be in a separate rack than the main cluster, or at least on a separate PSU and not on the same network segment as the corosync ring or rings. You cannot use more than one quorum device in a cluster at the same time. Although you cannot use more than one quorum device in a cluster at the same time, a single quorum device may be used by several clusters at the same time. Each cluster using that quorum device can use different algorithms and quorum options, as those are stored on the cluster nodes themselves. For example, a single quorum device can be used by one cluster with an ffsplit (fifty/fifty split) algorithm and by a second cluster with an lms (last man standing) algorithm. A quorum device should not be run on an existing cluster node. 28.1. Installing quorum device packages Configuring a quorum device for a cluster requires that you install the following packages: Install corosync-qdevice on the nodes of an existing cluster. Install pcs and corosync-qnetd on the quorum device host. Start the pcsd service and enable pcsd at system start on the quorum device host. 28.2. Configuring a quorum device Configure a quorum device and add it to the cluster with the following procedure. In this example: The node used for a quorum device is qdevice . The quorum device model is net , which is currently the only supported model. The net model supports the following algorithms: ffsplit : fifty-fifty split. This provides exactly one vote to the partition with the highest number of active nodes. lms : last-man-standing. If the node is the only one left in the cluster that can see the qnetd server, then it returns a vote. Warning The LMS algorithm allows the cluster to remain quorate even with only one remaining node, but it also means that the voting power of the quorum device is great since it is the same as number_of_nodes - 1. Losing connection with the quorum device means losing number_of_nodes - 1 votes, which means that only a cluster with all nodes active can remain quorate (by overvoting the quorum device); any other cluster becomes inquorate. For more detailed information about the implementation of these algorithms, see the corosync-qdevice (8) man page. The cluster nodes are node1 and node2 . Procedure On the node that you will use to host your quorum device, configure the quorum device with the following command. This command configures and starts the quorum device model net and configures the device to start on boot. After configuring the quorum device, you can check its status. This should show that the corosync-qnetd daemon is running and, at this point, there are no clients connected to it. The --full command option provides detailed output. Enable the ports on the firewall needed by the pcsd daemon and the net quorum device by enabling the high-availability service on firewalld with following commands. From one of the nodes in the existing cluster, authenticate user hacluster on the node that is hosting the quorum device. This allows pcs on the cluster to connect to pcs on the qdevice host, but does not allow pcs on the qdevice host to connect to pcs on the cluster. Add the quorum device to the cluster. Before adding the quorum device, you can check the current configuration and status for the quorum device for later comparison. The output for these commands indicates that the cluster is not yet using a quorum device, and the Qdevice membership status for each node is NR (Not Registered). The following command adds the quorum device that you have previously created to the cluster. You cannot use more than one quorum device in a cluster at the same time. However, one quorum device can be used by several clusters at the same time. This example command configures the quorum device to use the ffsplit algorithm. For information about the configuration options for the quorum device, see the corosync-qdevice (8) man page. Check the configuration status of the quorum device. From the cluster side, you can execute the following commands to see how the configuration has changed. The pcs quorum config shows the quorum device that has been configured. The pcs quorum status command shows the quorum runtime status, indicating that the quorum device is in use. The meanings of of the Qdevice membership information status values for each cluster node are as follows: A/NA - The quorum device is alive or not alive, indicating whether there is a heartbeat between qdevice and corosync . This should always indicate that the quorum device is alive. V/NV - V is set when the quorum device has given a vote to a node. In this example, both nodes are set to V since they can communicate with each other. If the cluster were to split into two single-node clusters, one of the nodes would be set to V and the other node would be set to NV . MW/NMW - The internal quorum device flag is set ( MW ) or not set ( NMW ). By default the flag is not set and the value is NMW . The pcs quorum device status shows the quorum device runtime status. From the quorum device side, you can execute the following status command, which shows the status of the corosync-qnetd daemon. 28.3. Managing the quorum device service PCS provides the ability to manage the quorum device service on the local host ( corosync-qnetd ), as shown in the following example commands. Note that these commands affect only the corosync-qnetd service. 28.4. Managing a quorum device in a cluster There are a variety of pcs commands that you can use to change the quorum device settings in a cluster, disable a quorum device, and remove a quorum device. 28.4.1. Changing quorum device settings You can change the setting of a quorum device with the pcs quorum device update command. Warning To change the host option of quorum device model net , use the pcs quorum device remove and the pcs quorum device add commands to set up the configuration properly, unless the old and the new host are the same machine. The following command changes the quorum device algorithm to lms . 28.4.2. Removing a quorum device The following command removes a quorum device configured on a cluster node. After you have removed a quorum device, you should see the following error message when displaying the quorum device status. 28.4.3. Destroying a quorum device The following command disables and stops a quorum device on the quorum device host and deletes all of its configuration files.
dnf install corosync-qdevice dnf install corosync-qdevice dnf install pcs corosync-qnetd systemctl start pcsd.service systemctl enable pcsd.service pcs qdevice setup model net --enable --start Quorum device 'net' initialized quorum device enabled Starting quorum device quorum device started pcs qdevice status net --full QNetd address: *:5403 TLS: Supported (client certificate required) Connected clients: 0 Connected clusters: 0 Maximum send/receive size: 32768/32768 bytes firewall-cmd --permanent --add-service=high-availability firewall-cmd --add-service=high-availability pcs host auth qdevice Username: hacluster Password: qdevice: Authorized pcs quorum config Options: pcs quorum status Quorum information ------------------ Date: Wed Jun 29 13:15:36 2016 Quorum provider: corosync_votequorum Nodes: 2 Node ID: 1 Ring ID: 1/8272 Quorate: Yes Votequorum information ---------------------- Expected votes: 2 Highest expected: 2 Total votes: 2 Quorum: 1 Flags: 2Node Quorate Membership information ---------------------- Nodeid Votes Qdevice Name 1 1 NR node1 (local) 2 1 NR node2 pcs quorum device add model net host=qdevice algorithm=ffsplit Setting up qdevice certificates on nodes node2: Succeeded node1: Succeeded Enabling corosync-qdevice node1: corosync-qdevice enabled node2: corosync-qdevice enabled Sending updated corosync.conf to nodes node1: Succeeded node2: Succeeded Corosync configuration reloaded Starting corosync-qdevice node1: corosync-qdevice started node2: corosync-qdevice started pcs quorum config Options: Device: Model: net algorithm: ffsplit host: qdevice pcs quorum status Quorum information ------------------ Date: Wed Jun 29 13:17:02 2016 Quorum provider: corosync_votequorum Nodes: 2 Node ID: 1 Ring ID: 1/8272 Quorate: Yes Votequorum information ---------------------- Expected votes: 3 Highest expected: 3 Total votes: 3 Quorum: 2 Flags: Quorate Qdevice Membership information ---------------------- Nodeid Votes Qdevice Name 1 1 A,V,NMW node1 (local) 2 1 A,V,NMW node2 0 1 Qdevice pcs quorum device status Qdevice information ------------------- Model: Net Node ID: 1 Configured node list: 0 Node ID = 1 1 Node ID = 2 Membership node list: 1, 2 Qdevice-net information ---------------------- Cluster name: mycluster QNetd host: qdevice:5403 Algorithm: ffsplit Tie-breaker: Node with lowest node ID State: Connected pcs qdevice status net --full QNetd address: *:5403 TLS: Supported (client certificate required) Connected clients: 2 Connected clusters: 1 Maximum send/receive size: 32768/32768 bytes Cluster "mycluster": Algorithm: ffsplit Tie-breaker: Node with lowest node ID Node ID 2: Client address: ::ffff:192.168.122.122:50028 HB interval: 8000ms Configured node list: 1, 2 Ring ID: 1.2050 Membership node list: 1, 2 TLS active: Yes (client certificate verified) Vote: ACK (ACK) Node ID 1: Client address: ::ffff:192.168.122.121:48786 HB interval: 8000ms Configured node list: 1, 2 Ring ID: 1.2050 Membership node list: 1, 2 TLS active: Yes (client certificate verified) Vote: ACK (ACK) pcs qdevice start net pcs qdevice stop net pcs qdevice enable net pcs qdevice disable net pcs qdevice kill net pcs quorum device update model algorithm=lms Sending updated corosync.conf to nodes node1: Succeeded node2: Succeeded Corosync configuration reloaded Reloading qdevice configuration on nodes node1: corosync-qdevice stopped node2: corosync-qdevice stopped node1: corosync-qdevice started node2: corosync-qdevice started pcs quorum device remove Sending updated corosync.conf to nodes node1: Succeeded node2: Succeeded Corosync configuration reloaded Disabling corosync-qdevice node1: corosync-qdevice disabled node2: corosync-qdevice disabled Stopping corosync-qdevice node1: corosync-qdevice stopped node2: corosync-qdevice stopped Removing qdevice certificates from nodes node1: Succeeded node2: Succeeded pcs quorum device status Error: Unable to get quorum status: corosync-qdevice-tool: Can't connect to QDevice socket (is QDevice running?): No such file or directory pcs qdevice destroy net Stopping quorum device quorum device stopped quorum device disabled Quorum device 'net' configuration files removed