Chapter 10. Rebooting the environment
Chapter 10. Rebooting the environment A situation might occur where you need to reboot the environment. For example, when you might need to modify the physical servers, or you might need to recover from a power outage. In this situation, it is important to make sure your Ceph Storage nodes boot correctly. Make sure to boot the nodes in the following order: Boot all Ceph Monitor nodes first - This ensures the Ceph Monitor service is active in your high availability cluster. By default, the Ceph Monitor service is installed on the Controller node. If the Ceph Monitor is separate from the Controller in a custom role, make sure this custom Ceph Monitor role is active. Boot all Ceph Storage nodes - This ensures the Ceph OSD cluster can connect to the active Ceph Monitor cluster on the Controller nodes. 10.1. Rebooting a Ceph Storage (OSD) cluster Complete the following steps to reboot a cluster of Ceph Storage (OSD) nodes. Prerequisites On a Ceph Monitor or Controller node that is running the ceph-mon service, check that the Red Hat Ceph Storage cluster status is healthy and the pg status is active+clean : USD sudo podman exec -it ceph-mon-controller-0 ceph -s If the Ceph cluster is healthy, it returns a status of HEALTH_OK . If the Ceph cluster status is unhealthy, it returns a status of HEALTH_WARN or HEALTH_ERR . For troubleshooting guidance, see the Red Hat Ceph Storage 4 Troubleshooting Guide . Procedure Log in to a Ceph Monitor or Controller node that is running the ceph-mon service, and disable Ceph Storage cluster rebalancing temporarily: USD sudo podman exec -it ceph-mon-controller-0 ceph osd set noout USD sudo podman exec -it ceph-mon-controller-0 ceph osd set norebalance Note If you have a multistack or distributed compute node (DCN) architecture, you must specify the cluster name when you set the noout and norebalance flags. For example: sudo podman exec -it ceph-mon-controller-0 ceph osd set noout --cluster <cluster_name> Select the first Ceph Storage node that you want to reboot and log in to the node. Reboot the node: Wait until the node boots. Log into the node and check the cluster status: USD sudo podman exec -it ceph-mon-controller-0 ceph status Check that the pgmap reports all pgs as normal ( active+clean ). Log out of the node, reboot the node, and check its status. Repeat this process until you have rebooted all Ceph Storage nodes. When complete, log in to a Ceph Monitor or Controller node that is running the ceph-mon service and re-enable cluster rebalancing: USD sudo podman exec -it ceph-mon-controller-0 ceph osd unset noout USD sudo podman exec -it ceph-mon-controller-0 ceph osd unset norebalance Note If you have a multistack or distributed compute node (DCN) architecture, you must specify the cluster name when you unset the noout and norebalance flags. For example: sudo podman exec -it ceph-mon-controller-0 ceph osd set noout --cluster <cluster_name> Perform a final status check to verify that the cluster reports HEALTH_OK : USD sudo podman exec -it ceph-mon-controller-0 ceph status If a situation occurs where all overcloud nodes boot at the same time, the Ceph OSD services might not start correctly on the Ceph Storage nodes. In this situation, reboot the Ceph Storage OSDs so they can connect to the Ceph Monitor service. Verify a HEALTH_OK status of the Ceph Storage node cluster with the following command:
sudo podman exec -it ceph-mon-controller-0 ceph -s sudo podman exec -it ceph-mon-controller-0 ceph osd set noout sudo podman exec -it ceph-mon-controller-0 ceph osd set norebalance sudo reboot sudo podman exec -it ceph-mon-controller-0 ceph status sudo podman exec -it ceph-mon-controller-0 ceph osd unset noout sudo podman exec -it ceph-mon-controller-0 ceph osd unset norebalance sudo podman exec -it ceph-mon-controller-0 ceph status sudo ceph status