Chapter 13. Upgrading Red Hat Ceph Storage 5 to 6
Chapter 13. Upgrading Red Hat Ceph Storage 5 to 6 You can upgrade the Red Hat Ceph Storage cluster from Release 5 to 6 after all other upgrade tasks are completed. Prerequisites The upgrade from Red Hat OpenStack Platform 16.2 to 17.1 is complete. All Controller nodes and Red Hat Ceph Storage nodes are upgraded to Red Hat Enterprise Linux (RHEL) 9.2. For more information, see Upgrading the control plane operating system . In HCI environments, all Compute nodes must also be upgraded to RHEL 9.2. For more information, see Upgrading the Compute node operating system . The current Red Hat Ceph Storage 5 cluster is healthy. 13.1. Director-deployed Red Hat Ceph Storage environments Perform the following tasks if Red Hat Ceph Storage is director-deployed in your environment. 13.1.1. Updating the cephadm client Before you upgrade the Red Hat Ceph Storage cluster, you must update the cephadm package in the overcloud nodes to Release 6. Prerequisites Confirm that the health status of the Red Hat Ceph Storage cluster is HEALTH_OK . Log in to a Controller node and use the command sudo cephadm shell - ceph -s to confirm the cluster health. If the status is not HEALTH_OK , correct any issues before continuing with this procedure. Procedure Create a playbook to enable the Red Hat Ceph Storage (tool only) repositories in the Controller nodes. It should contain the following information: Run the playbook: ansible-playbook -i ~/overcloud-deploy/<stack>/tripleo-ansible-inventory.yaml <playbook_file_name> --limit <controller_role> Replace <stack> with the name of your stack. Replace <playbook_file_name > with the name of the playbook created in the step. Replace <controller_role> with the role applied to Controller nodes. Use the --limit option to apply the content to Controller nodes only. Log in to a Controller node. Verify that the cephadm package is updated to Release 6: USD sudo dnf info cephadm | grep -i version 13.1.2. Updating the Red Hat Ceph Storage container image The container-image-prepare.yaml file contains the ContainerImagePrepare parameter and defines the Red Hat Ceph Storage containers. This file is used by the tripleo-container-image prepare command to define the rules for obtaining container images for the undercloud and overcloud. Update this file with the correct image version before updating your environment. Procedure Locate your container preparation file. The default name of this file is containers-prepare-parameter.yaml . Edit the container preparation file. Locate the ceph_tag parameter. The current entry should be similar to the following example: Update the ceph_tag parameter for Red Hat Ceph Storage 6: Edit the containers-image-prepare.yaml file and replace the Red Hat Ceph monitoring stack container related parameters with the following content: Save the file. 13.1.3. Running the container image prepare Complete the container image preparation process by running the director container preparation command. This prepares all container image configurations for the overcloud and retrieves the latest Red Hat Ceph Storage 6 container image. Note If you are using Red Hat Satellite Server to host RPMs and container images for your Red Hat OpenStack Platform (RHOSP) environment, do not perform this procedure. Update Satellite to include the Red Hat Ceph Storage 6 container image and update your containers-prepare-parameter.yaml file to reference the URL of the container image that is hosted on the Satellite server. Procedure Log in to the undercloud host as the stack user. Source the stackrc undercloud credentials file: Run the container preparation command: USD openstack tripleo container image prepare -e <container_preparation_file> Replace <container_preparation_file> with the name of your file. The default file is containers-prepare-parameter.yaml . Verify that the new Red Hat Ceph Storage image is present in the undercloud registry: USD openstack tripleo container image list -f value | awk -F '//' '/ceph/ {print USD2}' Verify that the new Red Hat Ceph Storage monitoring stack images are present in the undercloud registry: USD openstack tripleo container image list -f value | awk -F '//' '/grafana|prometheus|alertmanager|node-exporter/ {print USD2}' 13.1.4. Configuring Ceph Manager with Red Hat Ceph Storage 6 monitoring stack images Procedure Log in to a Controller node. List the current images from the Ceph Manager configuration: Update the Ceph Manager configuration for the monitoring stack services to use Red Hat Ceph Storage 6 images: Replace <alertmanager_image> with the new alertmanager image. Replace <grafana_image> with the new grafana image. Replace <node_exporter_image> with the new node exporter image. Replace <prometheus_image> with the new prometheus image. The following is an example of the alert manager update command: Verify that the new image references are updated in the Red Hat Ceph Storage cluster: 13.1.5. Upgrading to Red Hat Ceph Storage 6 with Orchestrator Upgrade to Red Hat Ceph Storage 6 by using the Orchestrator capabilities of the cephadm command. Prerequisities On a Monitor or Controller node that is running the ceph-mon service, confirm the Red Hat Ceph Storage cluster status by using the sudo cephadm --shell ceph status command . This command returns one of three responses: HEALTH_OK - The cluster is healthy. Proceed with the cluster upgrade. HEALTH_WARN - The cluster is unhealthy. Do not proceed with the cluster upgrade before the blocking issues are resolved. For troubleshooting guidance, see Red Hat Ceph Storage 5 Troubleshooting Guide . HEALTH_ERR - The cluster is unhealthy. Do not proceed with the cluster upgrade before the blocking issues are resolved. For troubleshooting guidance, see Red Hat Ceph Storage 5 Troubleshooting Guide . Procedure Log in to a Controller node. Upgrade the cluster to the latest Red Hat Ceph Storage version by using Upgrade a Red Hat Ceph Storage cluster using cephadm in the Red Hat Ceph Storage 6 Upgrade Guide . Important Steps 1 to 4 of the upgrade procedure presented in Upgrading the Red Hat Ceph Storage cluster are not necessary when upgrading a Red Hat Ceph Storage cluster deployed with Red Hat OpenStack by director. Start at step 5 of the procedure. Wait until the Red Hat Ceph Storage container upgrade completes. Note Monitor the status upgrade by using the command sudo cephadm shell - ceph orch upgrade status . 13.1.6. Upgrading NFS Ganesha when moving from Red Hat Ceph Storage 5 to 6 When Red Hat Ceph Storage is upgraded from Release 4 to 5, NFS Ganesha is not adopted by the Orchestrator. This means it remains under director control and must be moved manually to Release 6. Red Hat Ceph Storage 5 based NFS Ganesha with a Red Hat Ceph Storage 6 cluster is only supported during the upgrade period. Once you upgrade the Red Hat Ceph Storage cluster to 6, you must upgrade NFS Ganesha to use a Release 6 based container image. Note This procedure only applies to environments that are using the Shared File Systems service (manila) with CephFS NFS. Upgradng the Red Hat Ceph Storage container for NFS Ganesha is mandatory in these environments. Procedure Log in to a Controller node. Inspect the ceph-nfs service: USD sudo pcs status | grep ceph-nfs Inspect the ceph-nfs systemd unit to confirm that it contains the Red Hat Ceph Storage 5 container image and tag: Create a file called /home/stack/ganesha_update_extravars.yaml with the following content: Replace <ceph_image_name> with the name of the Red Hat Ceph Storage container image. Replace <ceph_image_namespace> with the name of the Red Hat Ceph Storage container namespace. Replace <ceph_image_tag> with the name of the Red Hat Ceph Storage container tag. For example, in a typical environment, this content would have the following values: Save the file. Run the ceph-update-ganesha.yml playbook and provide the ganesha_update_extravars.yaml playbook for additional command parameters: Replace <stack> with the name of the overcloud stack. Verify that the ceph-nfs service is running: USD sudo pcs status | grep ceph-nfs Verify that the ceph-nfs systemd unit contains the Red Hat Ceph Storage 6 container image and tag: USD cat /etc/systemd/system/ceph-nfs@.service | grep rhceph 13.1.7. Configuring Red Hat Ceph Storage Object Gateway to use Identity service (keystone) authentication When Red Hat Ceph Storage is upgraded from Release 4 to 5 to 6, more options must be set to configure the use of the Identity service than in older releases. For the procedure to set these options, see Configuring the Ceph Object Gateway to use Keystone authentication in the Object Gateway Guide . 13.2. External Red Hat Ceph Storage cluster environment Perform the following tasks if your Red Hat Ceph Storage cluster is external to your Red Hat OpenStack Platform deployment in your environment. 13.2.1. Updating the Red Hat Ceph Storage container image The container-image-prepare.yaml file contains the ContainerImagePrepare parameter and defines the Red Hat Ceph Storage containers. This file is used by the tripleo-container-image prepare command to define the rules for obtaining container images for the undercloud and overcloud. Update this file with the correct image version before updating your environment. Procedure Locate your container preparation file. The default name of this file is containers-prepare-parameter.yaml . Edit the container preparation file. Locate the ceph_tag parameter. The current entry should be similar to the following example: Update the ceph_tag parameter for Red Hat Ceph Storage 6: Edit the containers-image-prepare.yaml file and replace the Red Hat Ceph monitoring stack container related parameters with the following content: Save the file. 13.2.2. Running the container image prepare Complete the container image preparation process by running the director container preparation command. This prepares all container image configurations for the overcloud and retrieves the latest Red Hat Ceph Storage 6 container image. Note If you are using Red Hat Satellite Server to host RPMs and container images for your Red Hat OpenStack Platform (RHOSP) environment, do not perform this procedure. Update Satellite to include the Red Hat Ceph Storage 6 container image and update your containers-prepare-parameter.yaml file to reference the URL of the container image that is hosted on the Satellite server. Procedure Log in to the undercloud host as the stack user. Source the stackrc undercloud credentials file: Run the container preparation command: USD openstack tripleo container image prepare -e <container_preparation_file> Replace <container_preparation_file> with the name of your file. The default file is containers-prepare-parameter.yaml . Verify that the new Red Hat Ceph Storage image is present in the undercloud registry: USD openstack tripleo container image list -f value | awk -F '//' '/ceph/ {print USD2}' Verify that the new Red Hat Ceph Storage monitoring stack images are present in the undercloud registry: USD openstack tripleo container image list -f value | awk -F '//' '/grafana|prometheus|alertmanager|node-exporter/ {print USD2}' 13.2.3. Upgrading NFS Ganesha when moving from Red Hat Ceph Storage 5 to 6 When Red Hat Ceph Storage is upgraded from Release 4 to 5, NFS Ganesha is not adopted by the Orchestrator. This means it remains under director control and must be moved manually to Release 6. Red Hat Ceph Storage 5 based NFS Ganesha with a Red Hat Ceph Storage 6 cluster is only supported during the upgrade period. Once you upgrade the Red Hat Ceph Storage cluster to 6, you must upgrade NFS Ganesha to use a Release 6 based container image. Note This procedure only applies to environments that are using the Shared File Systems service (manila) with CephFS NFS. Upgradng the Red Hat Ceph Storage container for NFS Ganesha is mandatory in these environments. Procedure Log in to a Controller node. Inspect the ceph-nfs service: USD sudo pcs status | grep ceph-nfs Inspect the ceph-nfs systemd unit to confirm that it contains the Red Hat Ceph Storage 5 container image and tag: Create a file called /home/stack/ganesha_update_extravars.yaml with the following content: Replace <ceph_image_name> with the name of the Red Hat Ceph Storage container image. Replace <ceph_image_namespace> with the name of the Red Hat Ceph Storage container namespace. Replace <ceph_image_tag> with the name of the Red Hat Ceph Storage container tag. For example, in a typical environment, this content would have the following values: Save the file. Run the ceph-update-ganesha.yml playbook and provide the ganesha_update_extravars.yaml playbook for additional command parameters: Replace <stack> with the name of the overcloud stack. Verify that the ceph-nfs service is running: USD sudo pcs status | grep ceph-nfs Verify that the ceph-nfs systemd unit contains the Red Hat Ceph Storage 6 container image and tag: USD cat /etc/systemd/system/ceph-nfs@.service | grep rhceph 13.2.4. Configuring Red Hat Ceph Storage Object Gateway to use Identity service (keystone) authentication When Red Hat Ceph Storage is upgraded from Release 4 to 5 to 6, more options must be set to configure the use of the Identity service than in older releases. For the procedure to set these options, see Configuring the Ceph Object Gateway to use Keystone authentication in the Object Gateway Guide .
- hosts: all gather_facts: false tasks: - name: Enable RHCS 6 tools repo ansible.builtin.command: | subscription-manager repos --disable=rhceph-5-tools-for-rhel-9-x86_64-rpms --enable=rhceph-6-tools-for-rhel-9-x86_64-rpms become: true - name: Update cephadm ansible.builtin.package: name: cephadm state: latest become: true ceph_namespace: registry.redhat.io ceph_image: rhceph-6-rhel9 ceph_tag: '5' ceph_namespace: registry.redhat.io ceph_image: rhceph-6-rhel9 ceph_tag: '6' ceph_alertmanager_image: ose-prometheus-alertmanager ceph_alertmanager_namespace: registry.redhat.io/openshift4 ceph_alertmanager_tag: v4.12 ceph_grafana_image: rhceph-6-dashboard-rhel9 ceph_grafana_namespace: registry.redhat.io/rhceph ceph_grafana_tag: latest ceph_node_exporter_image: ose-prometheus-node-exporter ceph_node_exporter_namespace: registry.redhat.io/openshift4 ceph_node_exporter_tag: v4.12 ceph_prometheus_image: ose-prometheus ceph_prometheus_namespace: registry.redhat.io/openshift4 ceph_prometheus_tag: v4.12 source ~/stackrc sudo cephadm shell -- ceph config dump | grep image sudo cephadm shell -- ceph config set mgr mgr/cephadm/container_image_alertmanager <alertmanager_image> sudo cephadm shell -- ceph config set mgr mgr/cephadm/container_image_grafana <grafana_image> sudo cephadm shell -- ceph config set mgr mgr/cephadm/container_image_node_exporter <node_exporter_image> sudo cephadm shell -- ceph config set mgr mgr/cephadm/container_image_prometheus <prometheus_image> sudo cephadm shell -- ceph config set mgr mgr/cephadm/container_image_alertmanager undercloud-0.ctlplane.redhat.local:8787/rh-osbs/openshift-ose-prometheus-alertmanager:v4.12 sudo cephadm shell -- ceph config dump | grep image cat /etc/systemd/system/ceph-nfs@.service | grep -i container_image tripleo_cephadm_container_image: <ceph_image_name> tripleo_cephadm_container_ns: <ceph_image_namespace> tripleo_cephadm_container_tag: <ceph_image_tag> tripleo_cephadm_container_image: rhceph-6-rhel9 tripleo_cephadm_container_ns: undercloud-0.ctlplane.redhat.local:8787 tripleo_cephadm_container_tag: '6' ansible-playbook -i USDHOME/overcloud-deploy/<stack>/config-download/<stack>/tripleo-ansible-inventory.yaml /usr/share/ansible/tripleo-playbooks/ceph-update-ganesha.yml -e @USDHOME/overcloud-deploy/<stack>/config-download/<stack>/global_vars.yaml -e @USDHOME/overcloud-deploy/<stack>/config-download/<stack>/cephadm/cephadm-extra-vars-heat.yml -e @USDHOME/ganesha_update_extravars.yaml ceph_namespace: registry.redhat.io ceph_image: rhceph-6-rhel9 ceph_tag: '5' ceph_namespace: registry.redhat.io ceph_image: rhceph-6-rhel9 ceph_tag: '6' ceph_alertmanager_image: ose-prometheus-alertmanager ceph_alertmanager_namespace: registry.redhat.io/openshift4 ceph_alertmanager_tag: v4.12 ceph_grafana_image: rhceph-6-dashboard-rhel9 ceph_grafana_namespace: registry.redhat.io/rhceph ceph_grafana_tag: latest ceph_node_exporter_image: ose-prometheus-node-exporter ceph_node_exporter_namespace: registry.redhat.io/openshift4 ceph_node_exporter_tag: v4.12 ceph_prometheus_image: ose-prometheus ceph_prometheus_namespace: registry.redhat.io/openshift4 ceph_prometheus_tag: v4.12 source ~/stackrc cat /etc/systemd/system/ceph-nfs@.service | grep -i container_image tripleo_cephadm_container_image: <ceph_image_name> tripleo_cephadm_container_ns: <ceph_image_namespace> tripleo_cephadm_container_tag: <ceph_image_tag> tripleo_cephadm_container_image: rhceph-6-rhel9 tripleo_cephadm_container_ns: undercloud-0.ctlplane.redhat.local:8787 tripleo_cephadm_container_tag: '6' ansible-playbook -i USDHOME/overcloud-deploy/<stack>/config-download/<stack>/tripleo-ansible-inventory.yaml /usr/share/ansible/tripleo-playbooks/ceph-update-ganesha.yml -e @USDHOME/overcloud-deploy/<stack>/config-download/<stack>/global_vars.yaml -e @USDHOME/overcloud-deploy/<stack>/config-download/<stack>/cephadm/cephadm-extra-vars-heat.yml -e @USDHOME/ganesha_update_extravars.yaml