Configuring and managing high availability clusters
Configuring and managing high availability clusters Red Hat Enterprise Linux 9 Using the Red Hat High Availability Add-On to create and maintain Pacemaker clusters Red Hat Customer Content Services
dnf install pcs pacemaker fence-agents-all systemctl start pcsd.service systemctl enable pcsd.service firewall-cmd --permanent --add-service=high-availability firewall-cmd --reload passwd hacluster pcs host auth z1.example.com pcs cluster setup my_cluster --start z1.example.com pcs cluster status Cluster Status: Stack: corosync Current DC: z1.example.com (version 2.0.0-10.el8-b67d8d0de9) - partition with quorum Last updated: Thu Oct 11 16:11:18 2018 Last change: Thu Oct 11 16:11:00 2018 by hacluster via crmd on z1.example.com 1 node configured 0 resources configured PCSD Status: z1.example.com: Online pcs property set stonith-enabled=false dnf install -y httpd wget firewall-cmd --permanent --add-service=http firewall-cmd --reload cat <<-END >/var/www/html/index.html <html> <body>My Test Site - USD(hostname)</body> </html> END cat <<-END > /etc/httpd/conf.d/status.conf <Location /server-status> SetHandler server-status Order deny,allow Deny from all Allow from 127.0.0.1 Allow from ::1 </Location> END pcs resource describe apache pcs resource create ClusterIP ocf:heartbeat:IPaddr2 ip=192.168.122.120 --group apachegroup pcs resource create WebSite ocf:heartbeat:apache configfile=/etc/httpd/conf/httpd.conf statusurl="http://localhost/server-status" --group apachegroup pcs status Cluster name: my_cluster Stack: corosync Current DC: z1.example.com (version 2.0.0-10.el8-b67d8d0de9) - partition with quorum Last updated: Fri Oct 12 09:54:33 2018 Last change: Fri Oct 12 09:54:30 2018 by root via cibadmin on z1.example.com 1 node configured 2 resources configured Online: [ z1.example.com ] Full list of resources: Resource Group: apachegroup ClusterIP (ocf::heartbeat:IPaddr2): Started z1.example.com WebSite (ocf::heartbeat:apache): Started z1.example.com PCSD Status: z1.example.com: Online pcs resource config WebSite Resource: WebSite (class=ocf provider=heartbeat type=apache) Attributes: configfile=/etc/httpd/conf/httpd.conf statusurl=http://localhost/server-status Operations: start interval=0s timeout=40s (WebSite-start-interval-0s) stop interval=0s timeout=60s (WebSite-stop-interval-0s) monitor interval=1min (WebSite-monitor-interval-1min) killall -9 httpd pcs status Cluster name: my_cluster Current DC: z1.example.com (version 1.1.13-10.el7-44eb2dd) - partition with quorum 1 node and 2 resources configured Online: [ z1.example.com ] Full list of resources: Resource Group: apachegroup ClusterIP (ocf::heartbeat:IPaddr2): Started z1.example.com WebSite (ocf::heartbeat:apache): Started z1.example.com Failed Resource Actions: * WebSite_monitor_60000 on z1.example.com 'not running' (7): call=13, status=complete, exitreason='none', last-rc-change='Thu Oct 11 23:45:50 2016', queued=0ms, exec=0ms PCSD Status: z1.example.com: Online pcs resource cleanup WebSite pcs cluster stop --all dnf install pcs pacemaker fence-agents-all systemctl start pcsd.service systemctl enable pcsd.service firewall-cmd --permanent --add-service=high-availability firewall-cmd --reload passwd hacluster pcs host auth z1.example.com z2.example.com pcs cluster setup my_cluster --start z1.example.com z2.example.com pcs property set stonith-enabled=false pcs cluster status Cluster Status: Stack: corosync Current DC: z1.example.com (version 2.0.0-10.el8-b67d8d0de9) - partition with quorum Last updated: Thu Oct 11 16:11:18 2018 Last change: Thu Oct 11 16:11:00 2018 by hacluster via crmd on z1.example.com 2 nodes configured 0 resources configured PCSD Status: z1.example.com: Online z2.example.com: Online dnf install -y httpd wget firewall-cmd --permanent --add-service=http firewall-cmd --reload cat <<-END >/var/www/html/index.html <html> <body>My Test Site - USD(hostname)</body> </html> END cat <<-END > /etc/httpd/conf.d/status.conf <Location /server-status> SetHandler server-status Order deny,allow Deny from all Allow from 127.0.0.1 Allow from ::1 </Location> END pcs resource describe apache pcs resource create ClusterIP ocf:heartbeat:IPaddr2 ip=192.168.122.120 --group apachegroup pcs resource create WebSite ocf:heartbeat:apache configfile=/etc/httpd/conf/httpd.conf statusurl="http://localhost/server-status" --group apachegroup pcs status Cluster name: my_cluster Stack: corosync Current DC: z1.example.com (version 2.0.0-10.el8-b67d8d0de9) - partition with quorum Last updated: Fri Oct 12 09:54:33 2018 Last change: Fri Oct 12 09:54:30 2018 by root via cibadmin on z1.example.com 2 nodes configured 2 resources configured Online: [ z1.example.com z2.example.com ] Full list of resources: Resource Group: apachegroup ClusterIP (ocf::heartbeat:IPaddr2): Started z1.example.com WebSite (ocf::heartbeat:apache): Started z1.example.com PCSD Status: z1.example.com: Online z2.example.com: Online killall -9 httpd pcs status Cluster name: my_cluster Stack: corosync Current DC: z1.example.com (version 2.0.0-10.el8-b67d8d0de9) - partition with quorum Last updated: Fri Oct 12 09:54:33 2018 Last change: Fri Oct 12 09:54:30 2018 by root via cibadmin on z1.example.com 2 nodes configured 2 resources configured Online: [ z1.example.com z2.example.com ] Full list of resources: Resource Group: apachegroup ClusterIP (ocf::heartbeat:IPaddr2): Started z1.example.com WebSite (ocf::heartbeat:apache): Started z1.example.com Failed Resource Actions: * WebSite_monitor_60000 on z1.example.com 'not running' (7): call=31, status=complete, exitreason='none', last-rc-change='Fri Feb 5 21:01:41 2016', queued=0ms, exec=0ms pcs resource cleanup WebSite pcs node standby z1.example.com pcs status Cluster name: my_cluster Stack: corosync Current DC: z1.example.com (version 2.0.0-10.el8-b67d8d0de9) - partition with quorum Last updated: Fri Oct 12 09:54:33 2018 Last change: Fri Oct 12 09:54:30 2018 by root via cibadmin on z1.example.com 2 nodes configured 2 resources configured Node z1.example.com: standby Online: [ z2.example.com ] Full list of resources: Resource Group: apachegroup ClusterIP (ocf::heartbeat:IPaddr2): Started z2.example.com WebSite (ocf::heartbeat:apache): Started z2.example.com pcs node unstandby z1.example.com pcs cluster stop --all pcs resource -h pcs cluster cib filename pcs cluster cib testfile pcs cluster cib original.xml cp original.xml updated.xml pcs -f updated.xml resource create VirtualIP ocf:heartbeat:IPaddr2 ip=192.168.0.120 op monitor interval=30s pcs cluster cib-push updated.xml diff-against=original.xml pcs cluster cib-push filename pcs cluster cib-push --config filename pcs status pcs status commands pcs status resources pcs cluster status pcs status wait Waiting for the cluster to apply configuration changes pcs status wait 1min Waiting for the cluster to apply configuration changes (timeout: 60 seconds) pcs config pcs status query resource RESOURCE_ID is-state started pcs status query resource resource1 is-state started True pcs cluster config update [transport pass:quotes[ transport options ]] [compression pass:quotes[ compression options ]] [crypto pass:quotes[ crypto options ]] [totem pass:quotes[ totem options ]] [--corosync_conf pass:quotes[ path ]] pcs cluster config update transport knet_pmtud_interval=35 totem token=10000 join=100 pcs cluster corosync pcs cluster config Cluster Name: HACluster Cluster UUID: ad4ae07dcafe4066b01f1cc9391f54f5 Transport: knet Nodes: r8-node-01: Link 0 address: r8-node-01 Link 1 address: 192.168.122.121 nodeid: 1 r8-node-02: Link 0 address: r8-node-02 Link 1 address: 192.168.122.122 nodeid: 2 Links: Link 1: linknumber: 1 ping_interval: 1000 ping_timeout: 2000 pong_count: 5 Compression Options: level: 9 model: zlib threshold: 150 Crypto Options: cipher: aes256 hash: sha256 Totem Options: downcheck: 2000 join: 50 token: 10000 Quorum Device: net Options: sync_timeout: 2000 timeout: 3000 Model Options: algorithm: lms host: r8-node-03 Heuristics: exec_ping: ping -c 1 127.0.0.1 pcs cluster config show --output-format=cmd pcs cluster setup HACluster r8-node-01 addr=r8-node-01 addr=192.168.122.121 r8-node-02 addr=r8-node-02 addr=192.168.122.122 transport knet link linknumber=1 ping_interval=1000 ping_timeout=2000 pong_count=5 compression level=9 model=zlib threshold=150 crypto cipher=aes256 hash=sha256 totem downcheck=2000 join=50 token=10000 subscription-manager repos --enable=rhel-9-for-x86_64-highavailability-rpms dnf install pcs pacemaker fence-agents-all dnf install pcs pacemaker fence-agents- model rpm -q -a | grep fence fence-agents-rhevm-4.0.2-3.el7.x86_64 fence-agents-ilo-mp-4.0.2-3.el7.x86_64 fence-agents-ipmilan-4.0.2-3.el7.x86_64 firewall-cmd --permanent --add-service=high-availability firewall-cmd --add-service=high-availability passwd hacluster Changing password for user hacluster. New password: Retype new password: passwd: all authentication tokens updated successfully. systemctl start pcsd.service systemctl enable pcsd.service dnf install pcp-zeroconf pcs host auth z1.example.com z2.example.com Username: hacluster Password: z1.example.com: Authorized z2.example.com: Authorized pcs cluster setup my_cluster --start z1.example.com z2.example.com pcs cluster enable --all pcs cluster status Cluster Status: Stack: corosync Current DC: z2.example.com (version 2.0.0-10.el8-b67d8d0de9) - partition with quorum Last updated: Thu Oct 11 16:11:18 2018 Last change: Thu Oct 11 16:11:00 2018 by hacluster via crmd on z2.example.com 2 Nodes configured 0 Resources configured pcs cluster setup pass:quotes[ cluster_name ] pass:quotes[ node1_name ] addr=pass:quotes[ node1_link0_address ] addr=pass:quotes[ node1_link1_address ] pass:quotes[ node2_name ] addr=pass:quotes[ node2_link0_address ] addr=pass:quotes[ node2_link1_address ] pcs cluster setup my_twolink_cluster rh80-node1 addr=192.168.122.201 addr=192.168.123.201 rh80-node2 addr=192.168.122.202 addr=192.168.123.202 pcs cluster setup my_twolink_cluster rh80-node1 addr=192.168.122.201 addr=192.168.123.201 rh80-node2 addr=192.168.122.202 addr=192.168.123.202 transport knet link link_priority=1 link link_priority=0 pcs cluster setup my_twolink_cluster rh80-node1 addr=192.168.122.201 addr=192.168.123.201 rh80-node2 addr=192.168.122.202 addr=192.168.123.202 transport knet link linknumber=1 link_priority=0 link link_priority=1 pcs cluster setup my_twolink_cluster rh80-node1 addr=192.168.122.201 addr=192.168.123.201 rh80-node2 addr=192.168.122.202 addr=192.168.123.202 transport knet link_mode=active pcs cluster setup my_twolink_cluster rh80-node1 addr=192.168.122.201 addr=192.168.123.201 rh80-node2 addr=192.168.122.202 addr=192.168.123.202 transport knet link_mode=active link link_priority=1 link link_priority=0 pcs stonith create myapc fence_apc_snmp ipaddr="zapc.example.com" pcmk_host_map="z1.example.com:1;z2.example.com:2" login="apc" passwd="apc" pcs stonith config myapc Resource: myapc (class=stonith type=fence_apc_snmp) Attributes: ipaddr=zapc.example.com pcmk_host_map=z1.example.com:1;z2.example.com:2 login=apc passwd=apc Operations: monitor interval=60s (myapc-monitor-interval-60s) pcs config backup filename pcs config restore [--local] [ filename ] firewall-cmd --permanent --add-service=high-availability firewall-cmd --add-service=high-availability Configuration option global/system_id_source. system_id_source = "uname" lvm systemid system ID: z1.example.com uname -n z1.example.com pvcreate /dev/sdb1 Physical volume "/dev/sdb1" successfully created vgcreate --setautoactivation n my_vg /dev/sdb1 Volume group "my_vg" successfully created vgs -o+systemid VG #PV #LV #SN Attr VSize VFree System ID my_vg 1 0 0 wz--n- <1.82t <1.82t z1.example.com lvcreate -L450 -n my_lv my_vg Rounding up size to full physical extent 452.00 MiB Logical volume "my_lv" created lvs LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert my_lv my_vg -wi-a---- 452.00m mkfs.xfs /dev/my_vg/my_lv meta-data=/dev/my_vg/my_lv isize=512 agcount=4, agsize=28928 blks = sectsz=512 attr=2, projid32bit=1 lvmdevices --adddev /dev/sdb1 dnf install -y httpd wget firewall-cmd --permanent --add-service=http firewall-cmd --permanent --zone=public --add-service=http firewall-cmd --reload cat <<-END > /etc/httpd/conf.d/status.conf <Location /server-status> SetHandler server-status Require local </Location> END lvchange -ay my_vg/my_lv mount /dev/my_vg/my_lv /var/www/ mkdir /var/www/html mkdir /var/www/cgi-bin mkdir /var/www/error restorecon -R /var/www cat <<-END >/var/www/html/index.html <html> <body>Hello</body> </html> END umount /var/www pcs resource create my_lvm ocf:heartbeat:LVM-activate vgname=my_vg vg_access_mode=system_id --group apachegroup pcs resource status Resource Group: apachegroup my_lvm (ocf::heartbeat:LVM-activate): Started pcs resource create my_fs Filesystem device="/dev/my_vg/my_lv" directory="/var/www" fstype="xfs" --group apachegroup pcs resource create VirtualIP IPaddr2 ip=198.51.100.3 cidr_netmask=24 --group apachegroup pcs resource create Website apache configfile="/etc/httpd/conf/httpd.conf" statusurl="http://127.0.0.1/server-status" --group apachegroup pcs status Cluster name: my_cluster Last updated: Wed Jul 31 16:38:51 2013 Last change: Wed Jul 31 16:42:14 2013 via crm_attribute on z1.example.com Stack: corosync Current DC: z2.example.com (2) - partition with quorum Version: 1.1.10-5.el7-9abe687 2 Nodes configured 6 Resources configured Online: [ z1.example.com z2.example.com ] Full list of resources: myapc (stonith:fence_apc_snmp): Started z1.example.com Resource Group: apachegroup my_lvm (ocf::heartbeat:LVM-activate): Started z1.example.com my_fs (ocf::heartbeat:Filesystem): Started z1.example.com VirtualIP (ocf::heartbeat:IPaddr2): Started z1.example.com Website (ocf::heartbeat:apache): Started z1.example.com Hello /bin/systemctl reload httpd.service > /dev/null 2>/dev/null || true /usr/bin/test -f /var/run/httpd-Website.pid >/dev/null 2>/dev/null && /usr/bin/ps -q USD(/usr/bin/cat /var/run/httpd-Website.pid) >/dev/null 2>/dev/null && /usr/sbin/httpd -f /etc/httpd/conf/httpd.conf -c "PidFile /var/run/httpd-Website.pid" -k graceful > /dev/null 2>/dev/null || true pcs node standby z1.example.com pcs status Cluster name: my_cluster Last updated: Wed Jul 31 17:16:17 2013 Last change: Wed Jul 31 17:18:34 2013 via crm_attribute on z1.example.com Stack: corosync Current DC: z2.example.com (2) - partition with quorum Version: 1.1.10-5.el7-9abe687 2 Nodes configured 6 Resources configured Node z1.example.com (1): standby Online: [ z2.example.com ] Full list of resources: myapc (stonith:fence_apc_snmp): Started z1.example.com Resource Group: apachegroup my_lvm (ocf::heartbeat:LVM-activate): Started z2.example.com my_fs (ocf::heartbeat:Filesystem): Started z2.example.com VirtualIP (ocf::heartbeat:IPaddr2): Started z2.example.com Website (ocf::heartbeat:apache): Started z2.example.com pcs node unstandby z1.example.com Configuration option global/system_id_source. system_id_source = "uname" lvm systemid system ID: z1.example.com uname -n z1.example.com pvcreate /dev/sdb1 Physical volume "/dev/sdb1" successfully created vgcreate --setautoactivation n my_vg /dev/sdb1 Volume group "my_vg" successfully created vgs -o+systemid VG #PV #LV #SN Attr VSize VFree System ID my_vg 1 0 0 wz--n- <1.82t <1.82t z1.example.com lvcreate -L450 -n my_lv my_vg Rounding up size to full physical extent 452.00 MiB Logical volume "my_lv" created lvs LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert my_lv my_vg -wi-a---- 452.00m mkfs.xfs /dev/my_vg/my_lv meta-data=/dev/my_vg/my_lv isize=512 agcount=4, agsize=28928 blks = sectsz=512 attr=2, projid32bit=1 lvmdevices --adddev /dev/sdb1 mkdir /nfsshare lvchange -ay my_vg/my_lv mount /dev/my_vg/my_lv /nfsshare mkdir -p /nfsshare/exports mkdir -p /nfsshare/exports/export1 mkdir -p /nfsshare/exports/export2 touch /nfsshare/exports/export1/clientdatafile1 touch /nfsshare/exports/export2/clientdatafile2 umount /dev/my_vg/my_lv vgchange -an my_vg pcs resource create my_lvm ocf:heartbeat:LVM-activate vgname=my_vg vg_access_mode=system_id --group nfsgroup root@z1 ~]# pcs status Cluster name: my_cluster Last updated: Thu Jan 8 11:13:17 2015 Last change: Thu Jan 8 11:13:08 2015 Stack: corosync Current DC: z2.example.com (2) - partition with quorum Version: 1.1.12-a14efad 2 Nodes configured 3 Resources configured Online: [ z1.example.com z2.example.com ] Full list of resources: myapc (stonith:fence_apc_snmp): Started z1.example.com Resource Group: nfsgroup my_lvm (ocf::heartbeat:LVM-activate): Started z1.example.com PCSD Status: z1.example.com: Online z2.example.com: Online Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled pcs resource create nfsshare Filesystem device=/dev/my_vg/my_lv directory=/nfsshare fstype=xfs --group nfsgroup pcs status Full list of resources: myapc (stonith:fence_apc_snmp): Started z1.example.com Resource Group: nfsgroup my_lvm (ocf::heartbeat:LVM-activate): Started z1.example.com nfsshare (ocf::heartbeat:Filesystem): Started z1.example.com pcs resource create nfs-daemon nfsserver nfs_shared_infodir=/nfsshare/nfsinfo nfs_no_notify=true --group nfsgroup pcs status pcs resource create nfs-root exportfs clientspec=192.168.122.0/255.255.255.0 options=rw,sync,no_root_squash directory=/nfsshare/exports fsid=0 --group nfsgroup pcs resource create nfs-export1 exportfs clientspec=192.168.122.0/255.255.255.0 options=rw,sync,no_root_squash directory=/nfsshare/exports/export1 fsid=1 --group nfsgroup pcs resource create nfs-export2 exportfs clientspec=192.168.122.0/255.255.255.0 options=rw,sync,no_root_squash directory=/nfsshare/exports/export2 fsid=2 --group nfsgroup pcs resource create nfs_ip IPaddr2 ip=192.168.122.200 cidr_netmask=24 --group nfsgroup pcs resource create nfs-notify nfsnotify source_host=192.168.122.200 --group nfsgroup pcs status Full list of resources: myapc (stonith:fence_apc_snmp): Started z1.example.com Resource Group: nfsgroup my_lvm (ocf::heartbeat:LVM-activate): Started z1.example.com nfsshare (ocf::heartbeat:Filesystem): Started z1.example.com nfs-daemon (ocf::heartbeat:nfsserver): Started z1.example.com nfs-root (ocf::heartbeat:exportfs): Started z1.example.com nfs-export1 (ocf::heartbeat:exportfs): Started z1.example.com nfs-export2 (ocf::heartbeat:exportfs): Started z1.example.com nfs_ip (ocf::heartbeat:IPaddr2): Started z1.example.com nfs-notify (ocf::heartbeat:nfsnotify): Started z1.example.com showmount -e 192.168.122.200 Export list for 192.168.122.200: /nfsshare/exports/export1 192.168.122.0/255.255.255.0 /nfsshare/exports 192.168.122.0/255.255.255.0 /nfsshare/exports/export2 192.168.122.0/255.255.255.0 mkdir nfsshare mount -o "vers=4" 192.168.122.200:export1 nfsshare ls nfsshare clientdatafile1 umount nfsshare mkdir nfsshare mount -o "vers=3" 192.168.122.200:/nfsshare/exports/export2 nfsshare ls nfsshare clientdatafile2 umount nfsshare mkdir nfsshare mount -o "vers=4" 192.168.122.200:export1 nfsshare ls nfsshare clientdatafile1 pcs status Full list of resources: myapc (stonith:fence_apc_snmp): Started z1.example.com Resource Group: nfsgroup my_lvm (ocf::heartbeat:LVM-activate): Started z1.example.com nfsshare (ocf::heartbeat:Filesystem): Started z1.example.com nfs-daemon (ocf::heartbeat:nfsserver): Started z1.example.com nfs-root (ocf::heartbeat:exportfs): Started z1.example.com nfs-export1 (ocf::heartbeat:exportfs): Started z1.example.com nfs-export2 (ocf::heartbeat:exportfs): Started z1.example.com nfs_ip (ocf::heartbeat:IPaddr2): Started z1.example.com nfs-notify (ocf::heartbeat:nfsnotify): Started z1.example.com pcs node standby z1.example.com pcs status Full list of resources: Resource Group: nfsgroup my_lvm (ocf::heartbeat:LVM-activate): Started z2.example.com nfsshare (ocf::heartbeat:Filesystem): Started z2.example.com nfs-daemon (ocf::heartbeat:nfsserver): Started z2.example.com nfs-root (ocf::heartbeat:exportfs): Started z2.example.com nfs-export1 (ocf::heartbeat:exportfs): Started z2.example.com nfs-export2 (ocf::heartbeat:exportfs): Started z2.example.com nfs_ip (ocf::heartbeat:IPaddr2): Started z2.example.com nfs-notify (ocf::heartbeat:nfsnotify): Started z2.example.com ls nfsshare clientdatafile1 pcs node unstandby z1.example.com subscription-manager repos --enable=rhel-9-for-x86_64-resilientstorage-rpms dnf install lvm2-lockd gfs2-utils dlm use_lvmlockd = 1 pcs property set no-quorum-policy=freeze pcs resource create dlm --group locking ocf:pacemaker:controld op monitor interval=30s on-fail=fence pcs resource clone locking interleave=true pcs resource create lvmlockd --group locking ocf:heartbeat:lvmlockd op monitor interval=30s on-fail=fence pcs status --full Cluster name: my_cluster [...] Online: [ z1.example.com (1) z2.example.com (2) ] Full list of resources: smoke-apc (stonith:fence_apc): Started z1.example.com Clone Set: locking-clone [locking] Resource Group: locking:0 dlm (ocf::pacemaker:controld): Started z1.example.com lvmlockd (ocf::heartbeat:lvmlockd): Started z1.example.com Resource Group: locking:1 dlm (ocf::pacemaker:controld): Started z2.example.com lvmlockd (ocf::heartbeat:lvmlockd): Started z2.example.com Started: [ z1.example.com z2.example.com ] vgcreate --shared shared_vg1 /dev/vdb Physical volume "/dev/vdb" successfully created. Volume group "shared_vg1" successfully created VG shared_vg1 starting dlm lockspace Starting locking. Waiting until locks are ready vgcreate --shared shared_vg2 /dev/vdc Physical volume "/dev/vdc" successfully created. Volume group "shared_vg2" successfully created VG shared_vg2 starting dlm lockspace Starting locking. Waiting until locks are ready lvmdevices --adddev /dev/vdb lvmdevices --adddev /dev/vdc vgchange --lockstart shared_vg1 VG shared_vg1 starting dlm lockspace Starting locking. Waiting until locks are ready vgchange --lockstart shared_vg2 VG shared_vg2 starting dlm lockspace Starting locking. Waiting until locks are ready lvcreate --activate sy -L5G -n shared_lv1 shared_vg1 Logical volume "shared_lv1" created. lvcreate --activate sy -L5G -n shared_lv2 shared_vg1 Logical volume "shared_lv2" created. lvcreate --activate sy -L5G -n shared_lv1 shared_vg2 Logical volume "shared_lv1" created. mkfs.gfs2 -j2 -p lock_dlm -t my_cluster:gfs2-demo1 /dev/shared_vg1/shared_lv1 mkfs.gfs2 -j2 -p lock_dlm -t my_cluster:gfs2-demo2 /dev/shared_vg1/shared_lv2 mkfs.gfs2 -j2 -p lock_dlm -t my_cluster:gfs2-demo3 /dev/shared_vg2/shared_lv1 pcs resource create sharedlv1 --group shared_vg1 ocf:heartbeat:LVM-activate lvname=shared_lv1 vgname=shared_vg1 activation_mode=shared vg_access_mode=lvmlockd pcs resource create sharedlv2 --group shared_vg1 ocf:heartbeat:LVM-activate lvname=shared_lv2 vgname=shared_vg1 activation_mode=shared vg_access_mode=lvmlockd pcs resource create sharedlv3 --group shared_vg2 ocf:heartbeat:LVM-activate lvname=shared_lv1 vgname=shared_vg2 activation_mode=shared vg_access_mode=lvmlockd pcs resource clone shared_vg1 interleave=true pcs resource clone shared_vg2 interleave=true pcs constraint order start locking-clone then shared_vg1-clone Adding locking-clone shared_vg1-clone (kind: Mandatory) (Options: first-action=start then-action=start) pcs constraint order start locking-clone then shared_vg2-clone Adding locking-clone shared_vg2-clone (kind: Mandatory) (Options: first-action=start then-action=start) pcs constraint colocation add shared_vg1-clone with locking-clone pcs constraint colocation add shared_vg2-clone with locking-clone lvs LV VG Attr LSize shared_lv1 shared_vg1 -wi-a----- 5.00g shared_lv2 shared_vg1 -wi-a----- 5.00g shared_lv1 shared_vg2 -wi-a----- 5.00g lvs LV VG Attr LSize shared_lv1 shared_vg1 -wi-a----- 5.00g shared_lv2 shared_vg1 -wi-a----- 5.00g shared_lv1 shared_vg2 -wi-a----- 5.00g pcs resource create sharedfs1 --group shared_vg1 ocf:heartbeat:Filesystem device="/dev/shared_vg1/shared_lv1" directory="/mnt/gfs1" fstype="gfs2" options=noatime op monitor interval=10s on-fail=fence pcs resource create sharedfs2 --group shared_vg1 ocf:heartbeat:Filesystem device="/dev/shared_vg1/shared_lv2" directory="/mnt/gfs2" fstype="gfs2" options=noatime op monitor interval=10s on-fail=fence pcs resource create sharedfs3 --group shared_vg2 ocf:heartbeat:Filesystem device="/dev/shared_vg2/shared_lv1" directory="/mnt/gfs3" fstype="gfs2" options=noatime op monitor interval=10s on-fail=fence mount | grep gfs2 /dev/mapper/shared_vg1-shared_lv1 on /mnt/gfs1 type gfs2 (rw,noatime,seclabel) /dev/mapper/shared_vg1-shared_lv2 on /mnt/gfs2 type gfs2 (rw,noatime,seclabel) /dev/mapper/shared_vg2-shared_lv1 on /mnt/gfs3 type gfs2 (rw,noatime,seclabel) mount | grep gfs2 /dev/mapper/shared_vg1-shared_lv1 on /mnt/gfs1 type gfs2 (rw,noatime,seclabel) /dev/mapper/shared_vg1-shared_lv2 on /mnt/gfs2 type gfs2 (rw,noatime,seclabel) /dev/mapper/shared_vg2-shared_lv1 on /mnt/gfs3 type gfs2 (rw,noatime,seclabel) pcs status --full Cluster name: my_cluster [...] Full list of resources: smoke-apc (stonith:fence_apc): Started z1.example.com Clone Set: locking-clone [locking] Resource Group: locking:0 dlm (ocf::pacemaker:controld): Started z2.example.com lvmlockd (ocf::heartbeat:lvmlockd): Started z2.example.com Resource Group: locking:1 dlm (ocf::pacemaker:controld): Started z1.example.com lvmlockd (ocf::heartbeat:lvmlockd): Started z1.example.com Started: [ z1.example.com z2.example.com ] Clone Set: shared_vg1-clone [shared_vg1] Resource Group: shared_vg1:0 sharedlv1 (ocf::heartbeat:LVM-activate): Started z2.example.com sharedlv2 (ocf::heartbeat:LVM-activate): Started z2.example.com sharedfs1 (ocf::heartbeat:Filesystem): Started z2.example.com sharedfs2 (ocf::heartbeat:Filesystem): Started z2.example.com Resource Group: shared_vg1:1 sharedlv1 (ocf::heartbeat:LVM-activate): Started z1.example.com sharedlv2 (ocf::heartbeat:LVM-activate): Started z1.example.com sharedfs1 (ocf::heartbeat:Filesystem): Started z1.example.com sharedfs2 (ocf::heartbeat:Filesystem): Started z1.example.com Started: [ z1.example.com z2.example.com ] Clone Set: shared_vg2-clone [shared_vg2] Resource Group: shared_vg2:0 sharedlv3 (ocf::heartbeat:LVM-activate): Started z2.example.com sharedfs3 (ocf::heartbeat:Filesystem): Started z2.example.com Resource Group: shared_vg2:1 sharedlv3 (ocf::heartbeat:LVM-activate): Started z1.example.com sharedfs3 (ocf::heartbeat:Filesystem): Started z1.example.com Started: [ z1.example.com z2.example.com ] subscription-manager repos --enable=rhel-9-for-x86_64-resilientstorage-rpms dnf install lvm2-lockd gfs2-utils dlm use_lvmlockd = 1 pcs property set no-quorum-policy=freeze pcs resource create dlm --group locking ocf:pacemaker:controld op monitor interval=30s on-fail=fence pcs resource clone locking interleave=true pcs resource create lvmlockd --group locking ocf:heartbeat:lvmlockd op monitor interval=30s on-fail=fence pcs status --full Cluster name: my_cluster [...] Online: [ z1.example.com (1) z2.example.com (2) ] Full list of resources: smoke-apc (stonith:fence_apc): Started z1.example.com Clone Set: locking-clone [locking] Resource Group: locking:0 dlm (ocf::pacemaker:controld): Started z1.example.com lvmlockd (ocf::heartbeat:lvmlockd): Started z1.example.com Resource Group: locking:1 dlm (ocf::pacemaker:controld): Started z2.example.com lvmlockd (ocf::heartbeat:lvmlockd): Started z2.example.com Started: [ z1.example.com z2.example.com ] vgcreate --shared shared_vg1 /dev/sda1 Physical volume "/dev/sda1" successfully created. Volume group "shared_vg1" successfully created VG shared_vg1 starting dlm lockspace Starting locking. Waiting until locks are ready lvmdevices --adddev /dev/sda1 vgchange --lockstart shared_vg1 VG shared_vg1 starting dlm lockspace Starting locking. Waiting until locks are ready lvcreate --activate sy -L5G -n shared_lv1 shared_vg1 Logical volume "shared_lv1" created. pcs resource create sharedlv1 --group shared_vg1 ocf:heartbeat:LVM-activate lvname=shared_lv1 vgname=shared_vg1 activation_mode=shared vg_access_mode=lvmlockd pcs resource clone shared_vg1 interleave=true pcs constraint order start locking-clone then shared_vg1-clone Adding locking-clone shared_vg1-clone (kind: Mandatory) (Options: first-action=start then-action=start) pcs constraint colocation add shared_vg1-clone with locking-clone lvs LV VG Attr LSize shared_lv1 shared_vg1 -wi-a----- 5.00g lvs LV VG Attr LSize shared_lv1 shared_vg1 -wi-a----- 5.00g touch /etc/crypt_keyfile chmod 600 /etc/crypt_keyfile dd if=/dev/urandom bs=4K count=1 of=/etc/crypt_keyfile 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000306202 s, 13.4 MB/s scp /etc/crypt_keyfile root@z2.example.com:/etc/ scp -p /etc/crypt_keyfile root@z2.example.com:/etc/ cryptsetup luksFormat /dev/shared_vg1/shared_lv1 --type luks2 --key-file=/etc/crypt_keyfile WARNING! ======== This will overwrite data on /dev/shared_vg1/shared_lv1 irrevocably. Are you sure? (Type 'yes' in capital letters): YES pcs resource create crypt --group shared_vg1 ocf:heartbeat:crypt crypt_dev="luks_lv1" crypt_type=luks2 key_file=/etc/crypt_keyfile encrypted_dev="/dev/shared_vg1/shared_lv1" ls -l /dev/mapper/ lrwxrwxrwx 1 root root 7 Mar 4 09:52 luks_lv1 -> ../dm-3 mkfs.gfs2 -j3 -p lock_dlm -t my_cluster:gfs2-demo1 /dev/mapper/luks_lv1 /dev/mapper/luks_lv1 is a symbolic link to /dev/dm-3 This will destroy any data on /dev/dm-3 Are you sure you want to proceed? [y/n] y Discarding device contents (may take a while on large devices): Done Adding journals: Done Building resource groups: Done Creating quota file: Done Writing superblock and syncing: Done Device: /dev/mapper/luks_lv1 Block size: 4096 Device size: 4.98 GB (1306624 blocks) Filesystem size: 4.98 GB (1306622 blocks) Journals: 3 Journal size: 16MB Resource groups: 23 Locking protocol: "lock_dlm" Lock table: "my_cluster:gfs2-demo1" UUID: de263f7b-0f12-4d02-bbb2-56642fade293 pcs resource create sharedfs1 --group shared_vg1 ocf:heartbeat:Filesystem device="/dev/mapper/luks_lv1" directory="/mnt/gfs1" fstype="gfs2" options=noatime op monitor interval=10s on-fail=fence mount | grep gfs2 /dev/mapper/luks_lv1 on /mnt/gfs1 type gfs2 (rw,noatime,seclabel) mount | grep gfs2 /dev/mapper/luks_lv1 on /mnt/gfs1 type gfs2 (rw,noatime,seclabel) pcs status --full Cluster name: my_cluster [...] Full list of resources: smoke-apc (stonith:fence_apc): Started z1.example.com Clone Set: locking-clone [locking] Resource Group: locking:0 dlm (ocf::pacemaker:controld): Started z2.example.com lvmlockd (ocf::heartbeat:lvmlockd): Started z2.example.com Resource Group: locking:1 dlm (ocf::pacemaker:controld): Started z1.example.com lvmlockd (ocf::heartbeat:lvmlockd): Started z1.example.com Started: [ z1.example.com z2.example.com ] Clone Set: shared_vg1-clone [shared_vg1] Resource Group: shared_vg1:0 sharedlv1 (ocf::heartbeat:LVM-activate): Started z2.example.com crypt (ocf::heartbeat:crypt) Started z2.example.com sharedfs1 (ocf::heartbeat:Filesystem): Started z2.example.com Resource Group: shared_vg1:1 sharedlv1 (ocf::heartbeat:LVM-activate): Started z1.example.com crypt (ocf::heartbeat:crypt) Started z1.example.com sharedfs1 (ocf::heartbeat:Filesystem): Started z1.example.com Started: [z1.example.com z2.example.com ] subscription-manager repos --enable=rhel-9-for-x86_64-resilientstorage-rpms yum install lvm2-lockd gfs2-utils dlm use_lvmlockd = 1 pcs property set no-quorum-policy=freeze pcs resource create dlm --group locking ocf:pacemaker:controld op monitor interval=30s on-fail=fence pcs resource clone locking interleave=true pcs resource create lvmlockd --group locking ocf:heartbeat:lvmlockd op monitor interval=30s on-fail=fence pvcreate /dev/vdb vgcreate -Ay --shared csmb_vg /dev/vdb Volume group "csmb_vg" successfully created VG csmb_vg starting dlm lockspace Starting locking. Waiting until locks are ready lvmdevices --adddev /dev/vdb vgchange --lockstart csmb_vg VG csmb_vg starting dlm lockspace Starting locking. Waiting until locks are ready lvcreate -L1G -n ctdb_lv csmb_vg mkfs.gfs2 -j3 -p lock_dlm -t my_cluster:ctdb /dev/csmb_vg/ctdb_lv lvcreate -L50G -n csmb_lv1 csmb_vg mkfs.gfs2 -j3 -p lock_dlm -t my_cluster:csmb1 /dev/csmb_vg/csmb_lv1 pcs resource create --disabled --group shared_vg ctdb_lv ocf:heartbeat:LVM-activate lvname=ctdb_lv vgname=csmb_vg activation_mode=shared vg_access_mode=lvmlockd pcs resource create --disabled --group shared_vg csmb_lv1 ocf:heartbeat:LVM-activate lvname=csmb_lv1 vgname=csmb_vg activation_mode=shared vg_access_mode=lvmlockd pcs resource clone shared_vg interleave=true pcs constraint order start locking-clone then shared_vg-clone Adding locking-clone shared_vg-clone (kind: Mandatory) (Options: first-action=start then-action=start) pcs resource enable ctdb_lv csmb_lv1 pcs resource create ctdb_fs Filesystem device="/dev/csmb_vg/ctdb_lv" directory="/mnt/ctdb" fstype="gfs2" op monitor interval=10s on-fail=fence clone interleave=true pcs resource create csmb_fs1 Filesystem device="/dev/csmb_vg/csmb_lv1" directory="/srv/samba/share1" fstype="gfs2" op monitor interval=10s on-fail=fence clone interleave=true pcs constraint order start shared_vg-clone then ctdb_fs-clone Adding shared_vg-clone ctdb_fs-clone (kind: Mandatory) (Options: first-action=start then-action=start) pcs constraint order start shared_vg-clone then csmb_fs1-clone Adding shared_vg-clone csmb_fs1-clone (kind: Mandatory) (Options: first-action=start then-action=start) dnf -y install samba ctdb cifs-utils samba-winbind systemctl disable --now ctdb smb nmb winbind [global] netbios name = linuxserver workgroup = WORKGROUP security = user clustering = yes [share1] path = /srv/samba/share1 read only = no testparm 192.0.2.11 192.0.2.12 192.0.2.201/24 enp1s0 192.0.2.202/24 enp1s0 firewall-cmd --add-service=ctdb --add-service=samba --permanent firewall-cmd --reload semanage fcontext -at ctdbd_var_run_t -s system_u "/mnt/ctdb(/. )?" restorecon -Rv /mnt/ctdb semanage fcontext -at samba_share_t -s system_u "/srv/samba/share1(/. )?" restorecon -Rv /srv/samba/share1 pcs resource create --disabled ctdb --group samba-group ocf:heartbeat:CTDB ctdb_recovery_lock=/mnt/ctdb/ctdb.lock ctdb_dbdir=/var/lib/ctdb ctdb_logfile=/var/log/ctdb.log op monitor interval=10 timeout=30 op start timeout=90 op stop timeout=100 pcs resource clone samba-group pcs constraint order start ctdb_fs-clone then samba-group-clone pcs constraint order start csmb_fs1-clone then samba-group-clone pcs resource create samba --group samba-group systemd:smb pcs resource enable ctdb samba pcs status Full List of Resources: * fence-z1 (stonith:fence_xvm): Started z1.example.com * fence-z2 (stonith:fence_xvm): Started z2.example.com * Clone Set: locking-clone [locking]: * Started: [ z1.example.com z2.example.com ] * Clone Set: shared_vg-clone [shared_vg]: * Started: [ z1.example.com z2.example.com ] * Clone Set: ctdb_fs-clone [ctdb_fs]: * Started: [ z1.example.com z2.example.com ] * Clone Set: csmb_fs1-clone [csmb_fs1]: * Started: [ z1.example.com z2.example.com ] * Clone Set: samba-group-clone [samba-group]: * Started: [ z1.example.com z2.example.com ] useradd -M -s /sbin/nologin example_user passwd example_user smbpasswd -a example_user New SMB password: Retype new SMB password: Added user example_user smbpasswd -e example_user chown example_user:users /srv/samba/share1/ chmod 755 /srv/samba/share1/ mkdir /mnt/sambashare mount -t cifs -o user=example_user //192.0.2.201/share1 /mnt/sambashare Password for example_user@//192.0.2.201/public: XXXXXXX mount | grep /mnt/sambashare //192.0.2.201/public on /mnt/sambashare type cifs (rw,relatime,vers=1.0,cache=strict,username=example_user,domain=LINUXSERVER,uid=0,noforceuid,gid=0,noforcegid,addr=192.0.2.201,unix,posixpaths,serverino,mapposix,acl,rsize=1048576,wsize=65536,echo_interval=60,actimeo=1,user=example_user) touch /mnt/sambashare/testfile1 ls /mnt/sambashare testfile1 ip -4 addr show enp1s0 | grep inet inet 192.0.2.11/24 brd 192.0.2.255 scope global dynamic noprefixroute enp1s0 inet 192.0.2.201/24 brd 192.0.2.255 scope global secondary enp1s0 ip -4 addr show enp1s0 | grep inet inet 192.0.2.12/24 brd 192.0.2.255 scope global dynamic noprefixroute enp1s0 inet 192.0.2.202/24 brd 192.0.2.255 scope global secondary enp1s0 pcs node standby z1.example.com touch /mnt/sambashare/testfile2 ls /mnt/sambashare testfile1 testfile2 rm /mnt/sambashare/testfile1 /mnt/sambashare/testfile2 rm: remove regular empty file '/mnt/sambashare/testfile1'? y rm: remove regular empty file '/mnt/sambashare/testfile1'? y umount /mnt/sambashare pcs node unstandby z1.example.com https:// nodename :2224 pcs stonith list [ filter ] pcs stonith describe [ stonith_agent ] pcs stonith describe fence_apc Stonith options for: fence_apc ipaddr (required): IP Address or Hostname login (required): Login Name passwd: Login password or passphrase passwd_script: Script to retrieve password cmd_prompt: Force command prompt secure: SSH connection port (required): Physical plug number or name of virtual machine identity_file: Identity file for ssh switch: Physical switch number on device inet4_only: Forces agent to use IPv4 addresses only inet6_only: Forces agent to use IPv6 addresses only ipport: TCP port to use for connection with device action (required): Fencing Action verbose: Verbose mode debug: Write debug information to given file version: Display version information and exit help: Display help and exit separator: Separator for CSV created by operation list power_timeout: Test X seconds for status change after ON/OFF shell_timeout: Wait X seconds for cmd prompt after issuing command login_timeout: Wait X seconds for cmd prompt after login power_wait: Wait X seconds after issuing ON/OFF delay: Wait X seconds before fencing is started retry_on: Count of attempts to retry power on pcs stonith create stonith_id stonith_device_type [ stonith_device_options ] [op operation_action operation_options ] pcs stonith create MyStonith fence_virt pcmk_host_list=f1 op monitor interval=30s fence_ipmilan -a ipaddress -l username -p password -o status fence_ipmilan -a ipaddress -l username -p password -o reboot fence_ipmilan -a ipaddress -l username -p password -o status -D /tmp/USD(hostname)-fence_agent.debug pcs stonith fence node_name firewall-cmd --direct --add-rule ipv4 filter OUTPUT 2 -p udp --dport=5405 -j DROP firewall-cmd --add-rich-rule='rule family="ipv4" port port="5405" protocol="udp" drop echo c > /proc/sysrq-trigger pcs stonith level add level node devices pcs stonith level pcs stonith level add 1 rh7-2 my_ilo pcs stonith level add 2 rh7-2 my_apc pcs stonith level Node: rh7-2 Level 1 - my_ilo Level 2 - my_apc pcs stonith level remove level [ node_id ] [ stonith_id ] ... [ stonith_id ] pcs stonith level clear [ node ]| stonith_id (s)] pcs stonith level clear dev_a,dev_b pcs stonith level verify pcs stonith level add 1 "regexp%node[1-3]" apc1,apc2 pcs stonith level add 1 "regexp%node[4-6]" apc3,apc4 pcs node attribute node1 rack=1 pcs node attribute node2 rack=1 pcs node attribute node3 rack=1 pcs node attribute node4 rack=2 pcs node attribute node5 rack=2 pcs node attribute node6 rack=2 pcs stonith level add 1 attrib%rack=1 apc1,apc2 pcs stonith level add 1 attrib%rack=2 apc3,apc4 pcs stonith create apc1 fence_apc_snmp ipaddr=apc1.example.com login=user passwd='7a4D#1j!pz864' pcmk_host_map="node1.example.com:1;node2.example.com:2" pcs stonith create apc2 fence_apc_snmp ipaddr=apc2.example.com login=user passwd='7a4D#1j!pz864' pcmk_host_map="node1.example.com:1;node2.example.com:2" pcs stonith level add 1 node1.example.com apc1,apc2 pcs stonith level add 1 node2.example.com apc1,apc2 pcs stonith config [ stonith_id ] [--full] pcs stonith create myapc fence_apc_snmp ip="zapc.example.com" pcmk_host_map="z1.example.com:1;z2.example.com:2" username="apc" password="apc" pcs stonith config --output-format=cmd Warning: Only 'text' output format is supported for stonith levels pcs stonith create --no-default-ops --force -- myapc fence_apc_snmp ip=zapc.example.com password=apc 'pcmk_host_map=z1.example.com:1;z2.example.com:2' username=apc op monitor interval=60s id=myapc-monitor-interval-60s pcs stonith update stonith_id [ stonith_device_options ] pcs stonith update-scsi-devices stonith_id set device-path1 device-path2 pcs stonith update-scsi-devices stonith_id add device-path1 remove device-path2 pcs stonith delete stonith_id pcs stonith fence node [--off] pcs stonith confirm node pcs stonith disable myapc pcs constraint location node1-ipmi avoids node1 `Soft-Off by PWR-BTTN` set to `Instant-Off` +---------------------------------------------|-------------------+ | ACPI Function [Enabled] | Item Help | | ACPI Suspend Type [S1(POS)] |-------------------| | x Run VGABIOS if S3 Resume Auto | Menu Level * | | Suspend Mode [Disabled] | | | HDD Power Down [Disabled] | | | Soft-Off by PWR-BTTN [Instant-Off | | | CPU THRM-Throttling [50.0%] | | | Wake-Up by PCI card [Enabled] | | | Power On by Ring [Enabled] | | | Wake Up On LAN [Enabled] | | | x USB KB Wake-Up From S3 Disabled | | | Resume by Alarm [Disabled] | | | x Date(of Month) Alarm 0 | | | x Time(hh:mm:ss) Alarm 0 : 0 : | | | POWER ON Function [BUTTON ONLY | | | x KB Power ON Password Enter | | | x Hot Key Power ON Ctrl-F1 | | | | | | | | +---------------------------------------------|-------------------+ HandlePowerKey=ignore systemctl restart systemd-logind.service grubby --args=acpi=off --update-kernel=ALL pcs resource create resource_id [ standard :[ provider :]] type [ resource_options ] [op operation_action operation_options [ operation_action operation options ]...] [meta meta_options ...] [clone [ clone_id ] [ clone_options ] | promotable [ clone_id ] [ clone_options ] [--wait[= n ]] pcs resource create VirtualIP ocf:heartbeat:IPaddr2 ip=192.168.0.120 cidr_netmask=24 op monitor interval=30s pcs resource create VirtualIP IPaddr2 ip=192.168.0.120 cidr_netmask=24 op monitor interval=30s pcs resource delete resource_id pcs resource delete VirtualIP pcs resource describe [ standard :[ provider :]] type pcs resource describe ocf:heartbeat:apache This is the resource agent for the Apache Web server. This resource agent operates both version 1.x and version 2.x Apache servers. pcs resource defaults update resource-stickiness=100 pcs resource defaults set create id=pgsql-stickiness meta resource-stickiness=100 rule resource ::pgsql pcs resource defaults set create id=set-1 score=100 meta resource-stickiness=10 rule resource ocf:pacemaker:pgsql pcs resource defaults config Meta Attrs: build-resource-defaults resource-stickiness=1 Meta Attrs: set-1 score=100 resource-stickiness=10 Rule: boolean-op=and score=INFINITY Expression: resource ocf:pacemaker:pgsql pcs resource defaults config --output-format=cmd pcs -- resource defaults set create id=build-resource-defaults meta resource-stickiness=1; pcs -- resource defaults set create id=set-1 score=100 meta resource-stickiness=10 rule 'resource ocf:pacemaker:pgsql' pcs resource defaults config --output-format=json {"instance_attributes": [], "meta_attributes": [{"id": "build-resource-defaults", "options": {}, "rule": null, "nvpairs": [{"id": "build-resource-stickiness", "name": "resource-stickiness", "value": "1"}]}, {"id": "set-1", "options": {"score": "100"}, "rule": {"id": "set-1-rule", "type": "RULE", "in_effect": "UNKNOWN", "options": {"boolean-op": "and", "score": "INFINITY"}, "date_spec": null, "duration": null, "expressions": [{"id": "set-1-rule-rsc-ocf-pacemaker-pgsql", "type": "RSC_EXPRESSION", "in_effect": "UNKNOWN", "options": {"class": "ocf", "provider": "pacemaker", "type": "pgsql"}, "date_spec": null, "duration": null, "expressions": [], "as_string": "resource ocf:pacemaker:pgsql"}], "as_string": "resource ocf:pacemaker:pgsql"}, "nvpairs": [{"id": "set-1-resource-stickiness", "name": "resource-stickiness", "value": "10"}]}]} pcs resource create resource_id [ standard :[ provider :]] type [ resource options ] [meta meta_options ...] pcs resource create VirtualIP ocf:heartbeat:IPaddr2 ip=192.168.0.120 meta resource-stickiness=50 pcs resource meta resource_id | group_id | clone_id meta_options pcs resource meta dummy_resource failure-timeout=20s pcs resource config dummy_resource Resource: dummy_resource (class=ocf provider=heartbeat type=Dummy) Meta Attrs: failure-timeout=20s pcs resource group add group_name resource_id [ resource_id ] ... [ resource_id ] [--before resource_id | --after resource_id ] pcs resource create resource_id [ standard :[ provider :]] type [resource_options] [op operation_action operation_options ] --group group_name pcs resource group add shortcut IPaddr Email pcs resource group remove group_name resource_id pcs resource group list pcs constraint location rsc prefers node [= score ] [ node [= score ]] pcs constraint location rsc avoids node [= score ] [ node [= score ]] pcs constraint location Webserver prefers node1 pcs constraint location 'regexp%dummy[0-9]' prefers node1 pcs constraint location 'regexp%dummy[[:digit:]]' prefers node1 pcs constraint location add id rsc node score [resource-discovery= option ] pcs property set symmetric-cluster=false pcs constraint location Webserver prefers example-1=200 pcs constraint location Webserver prefers example-3=0 pcs constraint location Database prefers example-2=200 pcs constraint location Database prefers example-3=0 pcs property set symmetric-cluster=true pcs constraint location Webserver prefers example-1=200 pcs constraint location Webserver avoids example-2=INFINITY pcs constraint location Database avoids example-1=INFINITY pcs constraint location Database prefers example-2=200 pcs resource defaults update resource-stickiness=0 pcs resource create VirtualIP IPaddr2 ip=198.51.100.3 cidr_netmask=24 Assumed agent name 'ocf:heartbeat:IPaddr2' (deduced from 'IPaddr2') pcs resource create Website apache configfile="/etc/httpd/conf/httpd.conf" statusurl="http://127.0.0.1/server-status" Assumed agent name 'ocf:heartbeat:apache' (deduced from 'apache') pcs constraint location Website avoids node1 pcs constraint colocation add Website with VirtualIP pcs constraint order VirtualIP then Website Adding VirtualIP Website (kind: Mandatory) (Options: first-action=start then-action=start) pcs constraint --output-format=cmd pcs -- constraint location add location-Website-node1--INFINITY resource%Website node1 -INFINITY; pcs -- constraint colocation add Website with VirtualIP INFINITY id=colocation-Website-VirtualIP-INFINITY; pcs -- constraint order start VirtualIP then start Website id=order-VirtualIP-Website-mandatory pcs constraint order [ action ] resource_id then [ action ] resource_id [ options ] pcs constraint order remove resource1 [ resourceN ] pcs constraint order VirtualIP then dummy_resource kind=Optional pcs constraint order set resource1 resource2 [ resourceN ]... [ options ] [set resourceX resourceY ... [ options ]] [setoptions [ constraint_options ]] pcs constraint order set D1 D2 D3 pcs constraint order set A B sequential=false require-all=false set C D set E F sequential=false setoptions symmetrical=false [Unit] Requires=foo.service After=foo.service [Unit] Requires=srv.mount After=srv.mount [Unit] Requires=blk-availability.service After=blk-availability.service pcs constraint colocation add [promoted|unpromoted] source_resource with [promoted|unpromoted] target_resource [ score ] [ options ] pcs constraint colocation add myresource1 with myresource2 score=INFINITY pcs constraint colocation add myresource1 with myresource2 score=-INFINITY pcs constraint colocation set resource1 resource2 ] [ resourceN ]... [ options ] [set resourceX resourceY ] ... [ options ]] [setoptions [ constraint_options ]] pcs constraint colocation remove source_resource target_resource pcs constraint [list|show] [--full] pcs constraint location [show [resources [ resource ...]] | [nodes [ node ...]]] [--full] pcs constraint order [show] pcs constraint colocation [show] pcs constraint ref resource pcs resource relations resource [--full] pcs constraint order start C then start D Adding C D (kind: Mandatory) (Options: first-action=start then-action=start) pcs constraint order start D then start E Adding D E (kind: Mandatory) (Options: first-action=start then-action=start) pcs resource relations C C `- order | start C then start D `- D `- order | start D then start E `- E pcs resource relations D D |- order | | start C then start D | `- C `- order | start D then start E `- E pcs resource relations E E `- order | start D then start E `- D `- order | start C then start D `- C pcs resource relations A A `- outer resource `- G `- inner resource(s) | members: A B `- B pcs resource relations B B `- outer resource `- G `- inner resource(s) | members: A B `- A pcs resource relations G G `- inner resource(s) | members: A B |- A `- B pcs node attribute node1 rack=1 pcs node attribute node2 rack=2 pcs constraint location rsc rule [resource-discovery= option ] [role=promoted|unpromoted] [score= score | score-attribute= attribute ] expression pcs constraint location Webserver rule score=INFINITY date-spec years=2018 pcs constraint location Webserver rule score=INFINITY date-spec hours="9-16" weekdays="1-5" pcs constraint location Webserver rule date-spec weekdays=5 monthdays=13 moon=4 pcs constraint rule remove rule_id pcs resource status pcs resource status VirtualIP (ocf::heartbeat:IPaddr2): Started WebSite (ocf::heartbeat:apache): Started pcs resource config resource_id pcs resource config VirtualIP Resource: VirtualIP (type=IPaddr2 class=ocf provider=heartbeat) Attributes: ip=192.168.0.120 cidr_netmask=24 Operations: monitor interval=30s pcs resource status resource_id pcs resource status VirtualIP VirtualIP (ocf::heartbeat:IPaddr2): Started pcs resource status node= node_id pcs resource status node=node-01 VirtualIP (ocf::heartbeat:IPaddr2): Started WebSite (ocf::heartbeat:apache): Started pcs resource create my_lvm ocf:heartbeat:LVM-activate vgname=my_vg vg_access_mode=system_id --group apachegroup pcs resource create my_fs Filesystem device="/dev/my_vg/my_lv" directory="/var/www" fstype="xfs" --group apachegroup pcs resource create VirtualIP IPaddr2 ip=198.51.100.3 cidr_netmask=24 --group apachegroup pcs resource create Website apache configfile="/etc/httpd/conf/httpd.conf" statusurl="http://127.0.0.1/server-status" --group apachegroup pcs resource config --output-format=cmd pcs resource create --no-default-ops --force -- my_lvm ocf:heartbeat:LVM-activate vg_access_mode=system_id vgname=my_vg op monitor interval=30s id=my_lvm-monitor-interval-30s timeout=90s start interval=0s id=my_lvm-start-interval-0s timeout=90s stop interval=0s id=my_lvm-stop-interval-0s timeout=90s; pcs resource create --no-default-ops --force -- my_fs ocf:heartbeat:Filesystem device=/dev/my_vg/my_lv directory=/var/www fstype=xfs op monitor interval=20s id=my_fs-monitor-interval-20s timeout=40s start interval=0s id=my_fs-start-interval-0s timeout=60s stop interval=0s id=my_fs-stop-interval-0s timeout=60s; pcs resource create --no-default-ops --force -- VirtualIP ocf:heartbeat:IPaddr2 cidr_netmask=24 ip=198.51.100.3 op monitor interval=10s id=VirtualIP-monitor-interval-10s timeout=20s start interval=0s id=VirtualIP-start-interval-0s timeout=20s stop interval=0s id=VirtualIP-stop-interval-0s timeout=20s; pcs resource create --no-default-ops --force -- Website ocf:heartbeat:apache configfile=/etc/httpd/conf/httpd.conf statusurl=http://127.0.0.1/server-status op monitor interval=10s id=Website-monitor-interval-10s timeout=20s start interval=0s id=Website-start-interval-0s timeout=40s stop interval=0s id=Website-stop-interval-0s timeout=60s; pcs resource group add apachegroup my_lvm my_fs VirtualIP Website pcs resource config VirtualIP --output-format=cmd pcs resource create --no-default-ops --force -- VirtualIP ocf:heartbeat:IPaddr2 cidr_netmask=24 ip=198.51.100.3 op monitor interval=10s id=VirtualIP-monitor-interval-10s timeout=20s start interval=0s id=VirtualIP-start-interval-0s timeout=20s stop interval=0s id=VirtualIP-stop-interval-0s timeout=20s pcs resource update resource_id [ resource_options ] pcs resource config VirtualIP Resource: VirtualIP (type=IPaddr2 class=ocf provider=heartbeat) Attributes: ip=192.168.0.120 cidr_netmask=24 Operations: monitor interval=30s pcs resource update VirtualIP ip=192.169.0.120 pcs resource config VirtualIP Resource: VirtualIP (type=IPaddr2 class=ocf provider=heartbeat) Attributes: ip=192.169.0.120 cidr_netmask=24 Operations: monitor interval=30s pcs resource cleanup resource_id pcs resource meta dummy_resource migration-threshold=10 pcs resource defaults update migration-threshold=10 pcs resource create ping ocf:pacemaker:ping dampen=5s multiplier=1000 host_list=gateway.example.com clone pcs constraint location Webserver rule score=-INFINITY pingd lt 1 or not_defined pingd pcs resource update resourceXZY op monitor enabled=false pcs resource update resourceXZY op monitor enabled=true pcs resource update resourceXZY op monitor timeout=600 enabled=true pcs tag create special-resources d-01 d-02 pcs tag config special-resources d-01 d-02 pcs resource disable special-resources pcs resource * d-01 (ocf::pacemaker:Dummy): Stopped (disabled) * d-02 (ocf::pacemaker:Dummy): Stopped (disabled) pcs tag remove special-resources pcs tag No tags defined pcs tag update special-resources remove d-01 pcs resource delete d-01 Attempting to stop: d-01... Stopped pcs resource create resource_id [ standard :[ provider :]] type [ resource options ] [meta resource meta options ] clone [ clone_id ] [ clone options ] pcs resource clone resource_id | group_id [ clone_id ][ clone options ] pcs resource create webfarm apache clone pcs resource unclone resource_id | clone_id | group_name pcs constraint location webfarm-clone prefers node1 pcs constraint order start webfarm-clone then webfarm-stats pcs constraint colocation add webfarm-stats with webfarm-clone pcs resource create resource_id [ standard :[ provider :]] type [ resource options ] promotable [ clone_id ] [ clone options ] pcs resource promotable resource_id [ clone_id ] [ clone options ] pcs constraint colocation add [promoted|unpromoted] source_resource with [promoted|unpromoted] target_resource [ score ] [ options ] pcs constraint order [ action ] resource_id then [ action ] resource_id [ options ] pcs resource op add my-rsc promote on-fail="demote" pcs resource op add my-rsc monitor interval="10s" on-fail="demote" role="Promoted" pcs cluster stop [--all | node ] [...] pcs cluster kill pcs cluster enable [--all | node ] [...] pcs cluster disable [--all | node ] [...] dnf install -y pcs fence-agents-all firewall-cmd --permanent --add-service=high-availability firewall-cmd --add-service=high-availability passwd hacluster Changing password for user hacluster. New password: Retype new password: passwd: all authentication tokens updated successfully. systemctl start pcsd.service systemctl enable pcsd.service pcs host auth newnode.example.com Username: hacluster Password: newnode.example.com: Authorized pcs cluster node add newnode.example.com pcs cluster start Starting Cluster pcs cluster enable pcs cluster node remove node pcs cluster node add rh80-node3 addr=192.168.122.203 addr=192.168.123.203 pcs cluster link add node1=10.0.5.11 node2=10.0.5.12 node3=10.0.5.31 options linknumber=5 pcs cluster link delete 5 pcs cluster link remove 5 pcs cluster link remove 2 pcs cluster link add node1=10.0.5.11 node2=10.0.5.12 node3=10.0.5.31 options linknumber=2 pcs cluster link add node1=10.0.5.11 node2=10.0.5.12 node3=10.0.5.31 options linknumber=2 pcs cluster link remove 1 pcs cluster link add node1=10.0.5.13 node2=10.0.5.14 options linknumber=2 pcs cluster link remove 1 pcs cluster link add node1=10.0.5.11 node2=10.0.5.31 options linknumber=1 pcs cluster link remove 2 pcs cluster link add node1=10.0.5.13 node2=10.0.5.14 options linknumber=2 pcs cluster link remove 1 pcs cluster link add node1=10.0.5.11 node2=10.0.5.12 options linknumber=1 link_priority=11 pcs cluster link remove 2 pcs cluster stop --all pcs cluster link update 1 node1=10.0.5.11 node3=10.0.5.31 options link_priority=11 pcs cluster start --all pcs property set node-health-strategy=migrate-on-red pcs resource create io-monitor ocf:pacemaker:HealthIOWait red_limit=15 op monitor interval=10s meta allow-unhealthy-nodes=true clone pcs property set cluster-ipc-limit=2000 Compressed message exceeds X % of configured IPC limit ( X bytes); consider setting PCMK_ipc_buffer to X or higher PCMK_ipc_buffer=13396332 systemctl restart pacemaker adduser rouser usermod -a -G haclient rouser pcs acl enable pcs acl role create read-only description="Read access to cluster" read xpath /cib pcs acl user create rouser read-only pcs acl User: rouser Roles: read-only Role: read-only Description: Read access to cluster Permission: read xpath /cib (read-only-read) [rouser ~]USD pcs client local-auth pcs resource create resource_id standard:provider:type|type [ resource_options ] [op operation_action operation_options [ operation_type operation_options ]...] pcs resource create VirtualIP ocf:heartbeat:IPaddr2 ip=192.168.0.99 cidr_netmask=24 nic=eth2 op monitor interval=30s pcs resource op add resource_id operation_action [ operation_properties ] pcs resource op remove resource_id operation_name operation_properties pcs resource create VirtualIP ocf:heartbeat:IPaddr2 ip=192.168.0.99 cidr_netmask=24 nic=eth2 Operations: start interval=0s timeout=20s (VirtualIP-start-timeout-20s) stop interval=0s timeout=20s (VirtualIP-stop-timeout-20s) monitor interval=10s timeout=20s (VirtualIP-monitor-interval-10s) pcs resource update VirtualIP op stop interval=0s timeout=40s pcs resource config VirtualIP Resource: VirtualIP (class=ocf provider=heartbeat type=IPaddr2) Attributes: ip=192.168.0.99 cidr_netmask=24 nic=eth2 Operations: start interval=0s timeout=20s (VirtualIP-start-timeout-20s) monitor interval=10s timeout=20s (VirtualIP-monitor-interval-10s) stop interval=0s timeout=40s (VirtualIP-name-stop-interval-0s-timeout-40s) pcs resource op defaults update timeout=240s pcs resource update VirtualIP op monitor interval=10s pcs resource config VirtualIP Resource: VirtualIP (class=ocf provider=heartbeat type=IPaddr2) Attributes: ip=192.168.0.99 cidr_netmask=24 nic=eth2 Operations: start interval=0s timeout=20s (VirtualIP-start-timeout-20s) monitor interval=10s (VirtualIP-monitor-interval-10s) stop interval=0s timeout=40s (VirtualIP-name-stop-interval-0s-timeout-40s) pcs resource op defaults set create id=podman-timeout meta timeout=90s rule resource ::podman pcs resource op defaults set create id=stop-timeout meta timeout=120s rule op stop pcs resource op defaults set create id=podman-stop-timeout meta timeout=120s rule resource ::podman and op stop pcs resource op defaults set create id=op-set-1 score=100 meta timeout=30s rule op monitor and resource ocf:pacemaker:podman Warning: Defaults do not apply to resources which override them with their own defined values pcs resource op defaults config Meta Attrs: op-set-1 score=100 timeout=30s Rule: boolean-op=and score=INFINITY Expression: op monitor Expression: resource ocf:pacemaker:podman pcs resource op defaults config --output-format=cmd pcs -- resource op defaults set create id=op-set-1 score=100 meta timeout=30s rule 'op monitor and resource ocf:pacemaker:podman' pcs resource op defaults config --output-format=json {"instance_attributes": [], "meta_attributes": [{"id": "op-set-1", "options": {"score": "100"}, "rule": {"id": "op-set-1-rule", "type": "RULE", "in_effect": "UNKNOWN", "options": {"boolean-op": "and", "score": "INFINITY"}, "date_spec": null, "duration": null, "expressions": [{"id": "op-set-1-rule-op-monitor", "type": "OP_EXPRESSION", "in_effect": "UNKNOWN", "options": {"name": "monitor"}, "date_spec": null, "duration": null, "expressions": [], "as_string": "op monitor"}, {"id": "op-set-1-rule-rsc-ocf-pacemaker-podman", "type": "RSC_EXPRESSION", "in_effect": "UNKNOWN", "options": {"class": "ocf", "provider": "pacemaker", "type": "podman"}, "date_spec": null, "duration": null, "expressions": [], "as_string": "resource ocf:pacemaker:podman"}], "as_string": "op monitor and resource ocf:pacemaker:podman"}, "nvpairs": [{"id": "op-set-1-timeout", "name": "timeout", "value": "30s"}]}]} pcs resource create VirtualIP ocf:heartbeat:IPaddr2 ip=192.168.0.99 cidr_netmask=24 nic=eth2 pcs resource op add VirtualIP monitor interval=60s OCF_CHECK_LEVEL=10 pcs property set property = value pcs property set symmetric-cluster=false pcs property unset property pcs property set symmetic-cluster= pcs property list pcs property list --all pcs property show property pcs property show cluster-infrastructure Cluster Properties: cluster-infrastructure: cman pcs property [list|show] --defaults pcs property set migration-limit=10 pcs property config --output-format=cmd pcs property set --force -- migration-limit=10 placement-strategy=minimal pcs property set shutdown-lock=true pcs property list --all | grep shutdown-lock shutdown-lock: true shutdown-lock-limit: 0 pcs status Full List of Resources: * first (ocf::pacemaker:Dummy): Started z3.example.com * second (ocf::pacemaker:Dummy): Started z2.example.com * third (ocf::pacemaker:Dummy): Started z1.example.com * fourth (ocf::pacemaker:Dummy): Started z2.example.com * fifth (ocf::pacemaker:Dummy): Started z1.example.com pcs cluster stop z1.example.com Stopping Cluster (pacemaker) Stopping Cluster (corosync) pcs status Node List: * Online: [ z2.example.com z3.example.com ] * OFFLINE: [ z1.example.com ] Full List of Resources: * first (ocf::pacemaker:Dummy): Started z3.example.com * second (ocf::pacemaker:Dummy): Started z2.example.com * third (ocf::pacemaker:Dummy): Stopped z1.example.com (LOCKED) * fourth (ocf::pacemaker:Dummy): Started z3.example.com * fifth (ocf::pacemaker:Dummy): Stopped z1.example.com (LOCKED) pcs cluster start z1.example.com Starting Cluster pcs status Node List: * Online: [ z1.example.com z2.example.com z3.example.com ] Full List of Resources: .. * first (ocf::pacemaker:Dummy): Started z3.example.com * second (ocf::pacemaker:Dummy): Started z2.example.com * third (ocf::pacemaker:Dummy): Started z1.example.com * fourth (ocf::pacemaker:Dummy): Started z3.example.com * fifth (ocf::pacemaker:Dummy): Started z1.example.com pcs node utilization node1 cpu=2 memory=2048 pcs node utilization node2 cpu=4 memory=2048 pcs resource utilization dummy-small cpu=1 memory=1024 pcs resource utilization dummy-medium cpu=2 memory=2048 pcs resource utilization dummy-large cpu=3 memory=3072 pcs property set placement-strategy=balanced virsh dumpxml guest1 > /etc/pacemaker/guest1.xml pcs resource create VM VirtualDomain config=/etc/pacemaker/guest1.xml migration_transport=ssh meta allow-migrate=true pcs quorum update [auto_tie_breaker=[0|1]] [last_man_standing=[0|1]] [last_man_standing_window=[ time-in-ms ] [wait_for_all=[0|1]] pcs quorum update wait_for_all=1 Checking corosync is not running on nodes Error: node1: corosync is running Error: node2: corosync is running pcs cluster stop --all node2: Stopping Cluster (pacemaker) node1: Stopping Cluster (pacemaker) node1: Stopping Cluster (corosync) node2: Stopping Cluster (corosync) pcs quorum update wait_for_all=1 Checking corosync is not running on nodes node2: corosync is not running node1: corosync is not running Sending updated corosync.conf to nodes node1: Succeeded node2: Succeeded pcs quorum config Options: wait_for_all: 1 pcs quorum [config] pcs quorum status pcs quorum expected-votes votes pcs quorum unblock dnf install corosync-qdevice dnf install corosync-qdevice dnf install pcs corosync-qnetd systemctl start pcsd.service systemctl enable pcsd.service pcs qdevice setup model net --enable --start Quorum device 'net' initialized quorum device enabled Starting quorum device quorum device started pcs qdevice status net --full QNetd address: *:5403 TLS: Supported (client certificate required) Connected clients: 0 Connected clusters: 0 Maximum send/receive size: 32768/32768 bytes firewall-cmd --permanent --add-service=high-availability firewall-cmd --add-service=high-availability pcs host auth qdevice Username: hacluster Password: qdevice: Authorized pcs quorum config Options: pcs quorum status Quorum information ------------------ Date: Wed Jun 29 13:15:36 2016 Quorum provider: corosync_votequorum Nodes: 2 Node ID: 1 Ring ID: 1/8272 Quorate: Yes Votequorum information ---------------------- Expected votes: 2 Highest expected: 2 Total votes: 2 Quorum: 1 Flags: 2Node Quorate Membership information ---------------------- Nodeid Votes Qdevice Name 1 1 NR node1 (local) 2 1 NR node2 pcs quorum device add model net host=qdevice algorithm=ffsplit Setting up qdevice certificates on nodes node2: Succeeded node1: Succeeded Enabling corosync-qdevice node1: corosync-qdevice enabled node2: corosync-qdevice enabled Sending updated corosync.conf to nodes node1: Succeeded node2: Succeeded Corosync configuration reloaded Starting corosync-qdevice node1: corosync-qdevice started node2: corosync-qdevice started pcs quorum config Options: Device: Model: net algorithm: ffsplit host: qdevice pcs quorum status Quorum information ------------------ Date: Wed Jun 29 13:17:02 2016 Quorum provider: corosync_votequorum Nodes: 2 Node ID: 1 Ring ID: 1/8272 Quorate: Yes Votequorum information ---------------------- Expected votes: 3 Highest expected: 3 Total votes: 3 Quorum: 2 Flags: Quorate Qdevice Membership information ---------------------- Nodeid Votes Qdevice Name 1 1 A,V,NMW node1 (local) 2 1 A,V,NMW node2 0 1 Qdevice pcs quorum device status Qdevice information ------------------- Model: Net Node ID: 1 Configured node list: 0 Node ID = 1 1 Node ID = 2 Membership node list: 1, 2 Qdevice-net information ---------------------- Cluster name: mycluster QNetd host: qdevice:5403 Algorithm: ffsplit Tie-breaker: Node with lowest node ID State: Connected pcs qdevice status net --full QNetd address: *:5403 TLS: Supported (client certificate required) Connected clients: 2 Connected clusters: 1 Maximum send/receive size: 32768/32768 bytes Cluster "mycluster": Algorithm: ffsplit Tie-breaker: Node with lowest node ID Node ID 2: Client address: ::ffff:192.168.122.122:50028 HB interval: 8000ms Configured node list: 1, 2 Ring ID: 1.2050 Membership node list: 1, 2 TLS active: Yes (client certificate verified) Vote: ACK (ACK) Node ID 1: Client address: ::ffff:192.168.122.121:48786 HB interval: 8000ms Configured node list: 1, 2 Ring ID: 1.2050 Membership node list: 1, 2 TLS active: Yes (client certificate verified) Vote: ACK (ACK) pcs qdevice start net pcs qdevice stop net pcs qdevice enable net pcs qdevice disable net pcs qdevice kill net pcs quorum device update model algorithm=lms Sending updated corosync.conf to nodes node1: Succeeded node2: Succeeded Corosync configuration reloaded Reloading qdevice configuration on nodes node1: corosync-qdevice stopped node2: corosync-qdevice stopped node1: corosync-qdevice started node2: corosync-qdevice started pcs quorum device remove Sending updated corosync.conf to nodes node1: Succeeded node2: Succeeded Corosync configuration reloaded Disabling corosync-qdevice node1: corosync-qdevice disabled node2: corosync-qdevice disabled Stopping corosync-qdevice node1: corosync-qdevice stopped node2: corosync-qdevice stopped Removing qdevice certificates from nodes node1: Succeeded node2: Succeeded pcs quorum device status Error: Unable to get quorum status: corosync-qdevice-tool: Can't connect to QDevice socket (is QDevice running?): No such file or directory pcs qdevice destroy net Stopping quorum device quorum device stopped quorum device disabled Quorum device 'net' configuration files removed install --mode=0755 /usr/share/pacemaker/alerts/alert_file.sh.sample /var/lib/pacemaker/alert_file.sh touch /var/log/pcmk_alert_file.log chown hacluster:haclient /var/log/pcmk_alert_file.log chmod 600 /var/log/pcmk_alert_file.log pcs alert create id=alert_file description="Log events to a file." path=/var/lib/pacemaker/alert_file.sh pcs alert recipient add alert_file id=my-alert_logfile value=/var/log/pcmk_alert_file.log install --mode=0755 /usr/share/pacemaker/alerts/alert_snmp.sh.sample /var/lib/pacemaker/alert_snmp.sh pcs alert create id=snmp_alert path=/var/lib/pacemaker/alert_snmp.sh meta timestamp-format="%Y-%m-%d,%H:%M:%S.%01N" pcs alert recipient add snmp_alert value=192.168.1.2 pcs alert Alerts: Alert: snmp_alert (path=/var/lib/pacemaker/alert_snmp.sh) Meta options: timestamp-format=%Y-%m-%d,%H:%M:%S.%01N. Recipients: Recipient: snmp_alert-recipient (value=192.168.1.2) install --mode=0755 /usr/share/pacemaker/alerts/alert_smtp.sh.sample /var/lib/pacemaker/alert_smtp.sh pcs alert create id=smtp_alert path=/var/lib/pacemaker/alert_smtp.sh options email_sender=donotreply@example.com pcs alert recipient add smtp_alert value=admin@example.com pcs alert Alerts: Alert: smtp_alert (path=/var/lib/pacemaker/alert_smtp.sh) Options: email_sender=donotreply@example.com Recipients: Recipient: smtp_alert-recipient (value=admin@example.com) pcs alert create path= path [id= alert-id ] [description= description ] [options [ option = value ]...] [meta [ meta-option = value ]...] pcs alert create id=my_alert path=/path/to/myscript.sh pcs alert [config|show] pcs alert update alert-id [path= path ] [description= description ] [options [ option = value ]...] [meta [ meta-option = value ]...] pcs alert remove alert-id pcs alert recipient add alert-id value= recipient-value [id= recipient-id ] [description= description ] [options [ option = value ]...] [meta [ meta-option = value ]...] pcs alert recipient update recipient-id [value= recipient-value ] [description= description ] [options [ option = value ]...] [meta [ meta-option = value ]...] pcs alert recipient remove recipient-id pcs alert recipient add my-alert value=my-alert-recipient id=my-recipient-id options value=some-address pcs alert create id=my-alert path=/path/to/myscript.sh meta timeout=15s pcs alert recipient add my-alert value=someuser@example.com id=my-alert-recipient1 meta timestamp-format="%D %H:%M" pcs alert recipient add my-alert value=otheruser@example.com id=my-alert-recipient2 meta timestamp-format="%c" pcs alert create path=/my/path pcs alert recipient add alert value=rec_value pcs alert recipient add alert value=rec_value2 id=my-recipient pcs alert config Alerts: Alert: alert (path=/my/path) Recipients: Recipient: alert-recipient (value=rec_value) Recipient: my-recipient (value=rec_value2) pcs alert create id=my-alert path=/path/to/script description=alert_description options option1=value1 opt=val meta timeout=50s timestamp-format="%H%B%S" pcs alert recipient add my-alert value=my-other-recipient pcs alert Alerts: Alert: alert (path=/my/path) Recipients: Recipient: alert-recipient (value=rec_value) Recipient: my-recipient (value=rec_value2) Alert: my-alert (path=/path/to/script) Description: alert_description Options: opt=val option1=value1 Meta options: timestamp-format=%H%B%S timeout=50s Recipients: Recipient: my-alert-recipient (value=my-other-recipient) pcs alert update my-alert options option1=newvalue1 meta timestamp-format="%H%M%S" pcs alert recipient update my-alert-recipient options option1=new meta timeout=60s pcs alert Alerts: Alert: alert (path=/my/path) Recipients: Recipient: alert-recipient (value=rec_value) Recipient: my-recipient (value=rec_value2) Alert: my-alert (path=/path/to/script) Description: alert_description Options: opt=val option1=newvalue1 Meta options: timestamp-format=%H%M%S timeout=50s Recipients: Recipient: my-alert-recipient (value=my-other-recipient) Options: option1=new Meta options: timeout=60s pcs alert recipient remove my-recipient pcs alert Alerts: Alert: alert (path=/my/path) Recipients: Recipient: alert-recipient (value=rec_value) Alert: my-alert (path=/path/to/script) Description: alert_description Options: opt=val option1=newvalue1 Meta options: timestamp-format="%M%B%S" timeout=50s Recipients: Recipient: my-alert-recipient (value=my-other-recipient) Options: option1=new Meta options: timeout=60s pcs alert remove myalert pcs alert Alerts: Alert: alert (path=/my/path) Recipients: Recipient: alert-recipient (value=rec_value) dnf install -y booth-site dnf install -y booth-site dnf install -y booth-site dnf install -y booth-site dnf install -y pcs booth-core booth-arbitrator firewall-cmd --permanent --add-service=high-availability firewall-cmd --add-service=high-availability pcs booth setup sites 192.168.11.100 192.168.22.100 arbitrators 192.168.99.100 pcs booth ticket add apacheticket pcs booth sync pcs host auth cluster1-node1 pcs booth pull cluster1-node1 pcs host auth cluster1-node1 pcs booth pull cluster1-node1 pcs booth sync pcs booth start pcs booth enable pcs booth create ip 192.168.11.100 pcs booth create ip 192.168.22.100 pcs constraint ticket add apacheticket apachegroup pcs constraint ticket add apacheticket apachegroup pcs constraint ticket [show] pcs booth ticket grant apacheticket dnf install pacemaker-remote resource-agents pcs systemctl start pcsd.service systemctl enable pcsd.service firewall-cmd --add-port 3121/tcp --permanent firewall-cmd --add-port 2224/tcp --permanent firewall-cmd --reload pcs host auth nodename pcs cluster node add-guest nodename resource_id [ options ] pcs resource create webserver apache configfile=/etc/httpd/conf/httpd.conf op monitor interval=30s pcs constraint location webserver prefers nodename firewall-cmd --permanent --add-service=high-availability success firewall-cmd --reload success dnf install -y pacemaker-remote resource-agents pcs systemctl start pcsd.service systemctl enable pcsd.service pcs host auth remote1 pcs cluster node add-remote remote1 pcs resource create webserver apache configfile=/etc/httpd/conf/httpd.conf op monitor interval=30s pcs constraint location webserver prefers remote1 \#==#==# Pacemaker Remote # Specify a custom port for Pacemaker Remote connections PCMK_remote_port=3121 pcs resource disable resourcename pcs resource enable resourcename pcs node standby node | --all pcs node unstandby node | --all pcs resource move resource_id [ destination_node ] [--promoted] [--strict] [--wait[= n ]] pcs resource relocate run [ resource1 ] [ resource2 ] pcs resource disable resource_id [--wait[= n ]] pcs resource enable resource_id [--wait[= n ]] pcs resource ban resource_id [ node ] [--promoted] [lifetime= lifetime ] [--wait[= n ]] pcs resource debug-start resource_id pcs resource unmanage resource1 [ resource2 ] pcs resource manage resource1 [ resource2 ] pcs property set maintenance-mode=true pcs property set maintenance-mode=false pcs property unset property pcs property set symmetric-cluster= pcs resource disable resourcename pcs resource enable resourcename pcs cluster stop pcs cluster start pcs cluster config uuid generate pcs cluster config uuid generate --force pcs host auth z1.example.com z2.example.com z3.example.com z4.example.com -u hacluster -p password z1.example.com: Authorized z2.example.com: Authorized z3.example.com: Authorized z4.example.com: Authorized pcs cluster setup PrimarySite z1.example.com z2.example.com --start {...} Cluster has been successfully set up. Starting cluster on hosts: 'z1.example.com', 'z2.example.com' pcs cluster setup DRSite z3.example.com z4.example.com --start {...} Cluster has been successfully set up. Starting cluster on hosts: 'z3.example.com', 'z4.example.com' pcs dr set-recovery-site z3.example.com Sending 'disaster-recovery config' to 'z3.example.com', 'z4.example.com' z3.example.com: successful distribution of the file 'disaster-recovery config' z4.example.com: successful distribution of the file 'disaster-recovery config' Sending 'disaster-recovery config' to 'z1.example.com', 'z2.example.com' z1.example.com: successful distribution of the file 'disaster-recovery config' z2.example.com: successful distribution of the file 'disaster-recovery config' pcs dr config Local site: Role: Primary Remote site: Role: Recovery Nodes: z3.example.com z4.example.com pcs dr status --- Local cluster - Primary site --- Cluster name: PrimarySite WARNINGS: No stonith devices and stonith-enabled is not false Cluster Summary: * Stack: corosync * Current DC: z2.example.com (version 2.0.3-2.el8-2c9cea563e) - partition with quorum * Last updated: Mon Dec 9 04:10:31 2019 * Last change: Mon Dec 9 04:06:10 2019 by hacluster via crmd on z2.example.com * 2 nodes configured * 0 resource instances configured Node List: * Online: [ z1.example.com z2.example.com ] Full List of Resources: * No resources Daemon Status: corosync: active/disabled pacemaker: active/disabled pcsd: active/enabled --- Remote cluster - Recovery site --- Cluster name: DRSite WARNINGS: No stonith devices and stonith-enabled is not false Cluster Summary: * Stack: corosync * Current DC: z4.example.com (version 2.0.3-2.el8-2c9cea563e) - partition with quorum * Last updated: Mon Dec 9 04:10:34 2019 * Last change: Mon Dec 9 04:09:55 2019 by hacluster via crmd on z4.example.com * 2 nodes configured * 0 resource instances configured Node List: * Online: [ z3.example.com z4.example.com ] Full List of Resources: * No resources Daemon Status: corosync: active/disabled pacemaker: active/disabled pcsd: active/enabled