Chapter 3. Performing maintenance on the undercloud and overcloud with Instance HA
Chapter 3. Performing maintenance on the undercloud and overcloud with Instance HA To perform maintenance on the undercloud and overcloud, you must shut down and start up the undercloud and overcloud nodes in a specific order to ensure minimal issues when your start your overcloud. You can also perform maintenance on a specific Compute or Controller node by stopping the node and disabling the Pacemaker resources on the node. 3.1. Prerequisites A running undercloud and overcloud with Instance HA enabled. 3.2. Undercloud and overcloud shutdown order To shut down the Red Hat OpenStack Platform environment, you must shut down the overcloud and undercloud in the following order: Shut down instances on overcloud Compute nodes Shut down Compute nodes Stop all high availability and OpenStack Platform services on Controller nodes Shut down Ceph Storage nodes Shut down Controller nodes Shut down the undercloud 3.2.1. Shutting down instances on overcloud Compute nodes As a part of shutting down the Red Hat OpenStack Platform environment, shut down all instances on Compute nodes before shutting down the Compute nodes. Prerequisites An overcloud with active Compute services Procedure Log in to the undercloud as the stack user. Source the credentials file for your overcloud: View running instances in the overcloud: Stop each instance in the overcloud: Repeat this step for each instance until you stop all instances in the overcloud. 3.2.2. Stopping instance HA services on overcloud Compute nodes As a part of shutting down the Red Hat OpenStack Platform environment, you must shut down all Instance HA services that run on Compute nodes before stopping the instances and shutting down the Compute nodes. Prerequisites An overcloud with active Compute services Instance HA is enabled on Compute nodes Procedure Log in as the root user to an overcloud node that runs Pacemaker. Disable the Pacemaker remote resource on each Compute node: Identify the Pacemaker Remote resource on Compute nodes: These resources use the ocf::pacemaker:remote agent and are usually named after the Compute node host format, such as overcloud-novacomputeiha-0 . Disable each Pacemaker Remote resource. The following example shows how to disable the resource for overcloud-novacomputeiha-0 : Disable the Compute node STONITH devices: Identify the Compute node STONITH devices: Disable each Compute node STONITH device: 3.2.3. Shutting down Compute nodes As a part of shutting down the Red Hat OpenStack Platform environment, log in to and shut down each Compute node. Prerequisites Shut down all instances on the Compute nodes Procedure Log in as the root user to a Compute node. Shut down the node: Perform these steps for each Compute node until you shut down all Compute nodes. 3.2.4. Stopping services on Controller nodes As a part of shutting down the Red Hat OpenStack Platform environment, stop services on the Controller nodes before shutting down the nodes. This includes Pacemaker and systemd services. Prerequisites An overcloud with active Pacemaker services Procedure Log in as the root user to a Controller node. Stop the Pacemaker cluster. This command stops the cluster on all nodes. Wait until the Pacemaker services stop and check that the services stopped. Check the Pacemaker status: Check that no Pacemaker services are running in Podman: Stop the Red Hat OpenStack Platform services: Wait until the services stop and check that services are no longer running in Podman: 3.2.5. Shutting down Ceph Storage nodes As a part of shutting down the Red Hat OpenStack Platform environment, disable Ceph Storage services then log in to and shut down each Ceph Storage node. Prerequisites A healthy Ceph Storage cluster Ceph MON services are running on standalone Ceph MON nodes or on Controller nodes Procedure Log in as the root user to a node that runs Ceph MON services, such as a Controller node or a standalone Ceph MON node. Check the health of the cluster. In the following example, the podman command runs a status check within a Ceph MON container on a Controller node: Ensure that the status is HEALTH_OK . Set the noout , norecover , norebalance , nobackfill , nodown , and pause flags for the cluster. In the following example, the podman commands set these flags through a Ceph MON container on a Controller node: Shut down each Ceph Storage node: Log in as the root user to a Ceph Storage node. Shut down the node: Perform these steps for each Ceph Storage node until you shut down all Ceph Storage nodes. Shut down any standalone Ceph MON nodes: Log in as the root user to a standalone Ceph MON node. Shut down the node: Perform these steps for each standalone Ceph MON node until you shut down all standalone Ceph MON nodes. Additional resources "What is the procedure to shutdown and bring up the entire ceph cluster?" 3.2.6. Shutting down Controller nodes As a part of shutting down the Red Hat OpenStack Platform environment, log in to and shut down each Controller node. Prerequisites Stop the Pacemaker cluster Stop all Red Hat OpenStack Platform services on the Controller nodes Procedure Log in as the root user to a Controller node. Shut down the node: Perform these steps for each Controller node until you shut down all Controller nodes. 3.2.7. Shutting down the undercloud As a part of shutting down the Red Hat OpenStack Platform environment, log in to the undercloud node and shut down the undercloud. Prerequisites A running undercloud Procedure Log in to the undercloud as the stack user. Shut down the undercloud: 3.3. Performing system maintenance After you completely shut down the undercloud and overcloud, perform any maintenance to the systems in your environment and then start up the undercloud and overcloud. 3.4. Undercloud and overcloud startup order To start the Red Hat OpenStack Platform environment, you must start the undercloud and overcloud in the following order: Start the undercloud Start Controller nodes Start Ceph Storage nodes Start Compute nodes Start instances on overcloud Compute nodes 3.4.1. Starting the undercloud As a part of starting the Red Hat OpenStack Platform environment, power on the undercloud node, log in to the undercloud, and check the undercloud services. Prerequisites The undercloud is powered down. Procedure Power on the undercloud and wait until the undercloud boots. Verification Log in to the undercloud host as the stack user. Source the stackrc undercloud credentials file: Check the services on the undercloud: Create and validate a static inventory file named inventory.yaml : Check that all services and containers are active and healthy: Additional resources Using the validation framework 3.4.2. Starting Controller nodes As a part of starting the Red Hat OpenStack Platform environment, power on each Controller node and check the non-Pacemaker services on the node. Prerequisites Powered down Controller nodes Procedure Power on each Controller node. Verification Log in to each Controller node as the root user. Check the services on the Controller node: Only non-Pacemaker based services are running. Wait until the Pacemaker services start and check that the services started: Note If your environment uses Instance HA, the Pacemaker resources do not start until you start the Compute nodes or perform a manual unfence operation with the pcs stonith confirm <compute_node> command. You must run this command on each Compute node that uses Instance HA. 3.4.3. Starting Ceph Storage nodes As a part of starting the Red Hat OpenStack Platform environment, power on the Ceph MON and Ceph Storage nodes and enable Ceph Storage services. Prerequisites A powered down Ceph Storage cluster Ceph MON services are enabled on powered down standalone Ceph MON nodes or on powered on Controller nodes Procedure If your environment has standalone Ceph MON nodes, power on each Ceph MON node. Power on each Ceph Storage node. Log in as the root user to a node that runs Ceph MON services, such as a Controller node or a standalone Ceph MON node. Check the status of the cluster nodes. In the following example, the podman command runs a status check within a Ceph MON container on a Controller node: Ensure that each node is powered on and connected. Unset the noout , norecover , norebalance , nobackfill , nodown and pause flags for the cluster. In the following example, the podman commands unset these flags through a Ceph MON container on a Controller node: Verification Check the health of the cluster. In the following example, the podman command runs a status check within a Ceph MON container on a Controller node: Ensure the status is HEALTH_OK . Additional resources "What is the procedure to shutdown and bring up the entire ceph cluster?" 3.4.4. Starting Compute nodes As a part of starting the Red Hat OpenStack Platform environment, power on each Compute node and check the services on the node. Prerequisites Powered down Compute nodes Procedure Power on each Compute node. Verification Log in to each Compute as the root user. Check the services on the Compute node: 3.4.5. Starting instance HA services on overcloud Compute nodes As a part of starting the Red Hat OpenStack Platform environment, start all Instance HA services on the Compute nodes. Prerequisites An overcloud with running Compute nodes Instance HA is enabled on Compute nodes Procedure Log in as the root user to an overcloud node that runs Pacemaker. Enable the STONITH device for a Compute node: Identify the Compute node STONITH device: Clear any STONITH errors for the Compute node: This command returns the node to a clean STONITH state. Enable the Compute node STONITH device: Perform these steps for each Compute node with STONITH. Enable the Pacemaker remote resource on each Compute node: Identify the Pacemaker remote resources on Compute nodes: These resources use the ocf::pacemaker:remote agent and are usually named after the Compute node host format, such as overcloud-novacomputeiha-0 . Enable each Pacemaker Remote resource. The following example shows how to enable the resource for overcloud-novacomputeiha-0 : Perform these steps for each Compute node with Pacemaker remote management. Wait until the Pacemaker services start and check that the services started: If any Pacemaker resources fail to start during the startup process, reset the status and the fail count of the resource: Note Some services might require more time to start, such as fence_compute and fence_kdump . 3.4.6. Starting instances on overcloud Compute nodes As a part of starting the Red Hat OpenStack Platform environment, start the instances on on Compute nodes. Prerequisites An active overcloud with active nodes Procedure Log in to the undercloud as the stack user. Source the credentials file for your overcloud: View running instances in the overcloud: Start an instance in the overcloud:
source ~/overcloudrc openstack server list --all-projects openstack server stop <INSTANCE> pcs resource status pcs resource disable overcloud-novacomputeiha-0 pcs stonith status pcs stonith disable <STONITH_DEVICE> shutdown -h now pcs cluster stop --all pcs status podman ps --filter "name=.*-bundle.*" systemctl stop 'tripleo_*' podman ps sudo podman exec -it ceph-mon-controller-0 ceph status sudo podman exec -it ceph-mon-controller-0 ceph osd set noout sudo podman exec -it ceph-mon-controller-0 ceph osd set norecover sudo podman exec -it ceph-mon-controller-0 ceph osd set norebalance sudo podman exec -it ceph-mon-controller-0 ceph osd set nobackfill sudo podman exec -it ceph-mon-controller-0 ceph osd set nodown sudo podman exec -it ceph-mon-controller-0 ceph osd set pause shutdown -h now shutdown -h now shutdown -h now sudo shutdown -h now source ~/stackrc systemctl list-units 'tripleo_*' tripleo-ansible-inventory --static-yaml-inventory inventory.yaml openstack tripleo validator run --group pre-introspection -i inventory.yaml openstack tripleo validator run --validation service-status --limit undercloud -i inventory.yaml systemctl -t service pcs status sudo podman exec -it ceph-mon-controller-0 ceph status sudo podman exec -it ceph-mon-controller-0 ceph osd unset noout sudo podman exec -it ceph-mon-controller-0 ceph osd unset norecover sudo podman exec -it ceph-mon-controller-0 ceph osd unset norebalance sudo podman exec -it ceph-mon-controller-0 ceph osd unset nobackfill sudo podman exec -it ceph-mon-controller-0 ceph osd unset nodown sudo podman exec -it ceph-mon-controller-0 ceph osd unset pause sudo podman exec -it ceph-mon-controller-0 ceph status systemctl -t service pcs stonith status pcs stonith confirm <COMPUTE_NODE> pcs stonith enable <STONITH_DEVICE> pcs resource status pcs resource enable overcloud-novacomputeiha-0 pcs status pcs resource cleanup source ~/overcloudrc openstack server list --all-projects openstack server start <INSTANCE>