Chapter 8. Manually upgrading a Red Hat Ceph Storage cluster and operating system
Chapter 8. Manually upgrading a Red Hat Ceph Storage cluster and operating system Normally, using ceph-ansible , it is not possible to upgrade Red Hat Ceph Storage and Red Hat Enterprise Linux to a new major release at the same time. For example, if you are on Red Hat Enterprise Linux 7, using ceph-ansible , you must stay on that version. As a system administrator, you can do this manually, however. Use this chapter to manually upgrade a Red Hat Ceph Storage cluster at version 4.1 or 3.3z6 running on Red Hat Enterprise Linux 7.9, to a Red Hat Ceph Storage cluster at version 4.2 running on Red Hat Enterprise Linux 8.4. Important To upgrade a containerized Red Hat Ceph Storage cluster at version 3.x or 4.x to a version 4.2, see the following three sections, Supported Red Hat Ceph Storage upgrade scenarios , Preparing for an upgrade , and Upgrading the storage cluster using Ansible in the Red Hat Ceph Storage Installation Guide . To migrate existing systemd templates, run docker-to-podman playbook: Where user is the Ansible user. Important If a node is collocated with more than one daemon, follow the specific section in this chapter , for the daemons collocated in the node. For example a node collocated with the Ceph Monitor daemon and the OSD daemon: see Manually upgrading Ceph Monitor nodes and their operating systems and Manually upgrading Ceph OSD nodes and their operating systems . Important Manually upgrading Ceph OSD nodes and their operating systems will not work with encrypted OSD partitions as the Leapp upgrade utility does not support upgrading with OSD encryption. 8.1. Prerequisites A running Red Hat Ceph Storage cluster. The nodes are running Red Hat Enterprise Linux 7.9. The nodes are using Red Hat Ceph Storage version 3.3z6 or 4.1 Access to the installation source for Red Hat Enterprise Linux 8.3. 8.2. Manually upgrading Ceph Monitor nodes and their operating systems As a system administrator, you can manually upgrade the Ceph Monitor software on a Red Hat Ceph Storage cluster node and the Red Hat Enterprise Linux operating system to a new major release at the same time. Important Perform the procedure on only one Monitor node at a time. To prevent cluster access issues, ensure the current upgraded Monitor node has returned to normal operation prior to proceeding to the node. Prerequisites A running Red Hat Ceph Storage cluster. The nodes are running Red Hat Enterprise Linux 7.9. The nodes are using Red Hat Ceph Storage version 3.3z6 or 4.1 Access to the installation source for Red Hat Enterprise Linux 8.3. Procedure Stop the monitor service: Syntax Replace MONITOR_ID with the Monitor's ID number. If using Red Hat Ceph Storage 3, disable the Red Hat Ceph Storage 3 repositories. Disable the tools repository: Disable the mon repository: If using Red Hat Ceph Storage 4, disable the Red Hat Ceph Storage 4 repositories. Disable the tools repository: Disable the mon repository: Install the leapp utility. See Upgrading from Red Hat Enterprise Linux 7 to Red Hat Enterprise Linux 8 . Run through the leapp preupgrade checks. See Assessing upgradability from the command line . Set PermitRootLogin yes in /etc/ssh/sshd_config . Restart the OpenSSH SSH daemon: Remove the iSCSI module from the Linux kernel: Perform the upgrade by following Performing the upgrade from RHEL 7 to RHEL 8 . Reboot the node. Enable the repositories for Red Hat Ceph Storage 4 for Red Hat Enterprise Linux 8. Enable the tools repository: Enable the mon repository: Install the ceph-mon package: If the manager service is colocated with the monitor service, install the ceph-mgr package: Restore the ceph-client-admin.keyring and ceph.conf files from a Monitor node which has not been upgraded yet or from a node that has already had those files restored. Switch any existing CRUSH buckets to the latest bucket type straw2 . Once all the daemons are updated after upgrading from Red Hat Ceph Storage 3 to Red Hat Ceph Storage 4, run the following steps: Enable the messenger v2 protocol, msgr2 : This instructs all Ceph Monitors that bind to the old default port of 6789, to also bind to the new port of 3300. Important Ensure all the Ceph Monitors are upgraded from Red Hat Ceph Storage 3 to Red Hat Ceph Storage 4 before performing any further Ceph Monitor configuration. Verify the status of the monitor: Note Running nautilus OSDs does not bind to their v2 address automatically. They must be restarted. For each host upgraded from Red Hat Ceph Storage 3 to Red Hat Ceph Storage 4, update the ceph.conf file to either not specify any monitor port or reference both the v2 and v1 addresses and ports. Import any configuration options in ceph.conf file into the storage cluster's configuration database. Example Check the storage cluster's configuration database. Example Optional: After upgrading to Red Hat Ceph Storage 4, create a minimal ceph.conf file for each host: Example Install the leveldb package: Start the monitor service: If the manager service is colocated with the monitor service, start the manager service too: Verify the monitor service came back up and is in quorum. On the mon: line under services: , ensure the node is listed as in quorum and not as out of quorum . Example If the manager service is colocated with the monitor service, verify it is up too: Look for the manager's node name on the mgr: line under services . Example Repeat the above steps on all Monitor nodes until they have all been upgraded. Additional Resources See Manually upgrading a Red Hat Ceph Storage cluster and operating system in the Installation Guide for more information. See Upgrading from Red Hat Enterprise Linux 7 to Red Hat Enterprise Linux 8 for more information. 8.3. Manually upgrading Ceph OSD nodes and their operating systems As a system administrator, you can manually upgrade the Ceph OSD software on a Red Hat Ceph Storage cluster node and the Red Hat Enterprise Linux operating system to a new major release at the same time. Important This procedure should be performed for each OSD node in the Ceph cluster, but typically only for one OSD node at a time. A maximum of one failure domains worth of OSD nodes may be performed in parallel. For example, if per-rack replication is in use, one entire rack's OSD nodes can be upgraded in parallel. To prevent data access issues, ensure the current OSD node's OSDs have returned to normal operation and all of the cluster's PGs are in the active+clean state prior to proceeding to the OSD. Important This procedure will not work with encrypted OSD partitions as the Leapp upgrade utility does not support upgrading with OSD encryption. Important If the OSDs were created using ceph-disk , and are still managed by ceph-disk , you must use ceph-volume to take over management of them. This is covered in an optional step below. Prerequisites A running Red Hat Ceph Storage cluster. The nodes are running Red Hat Enterprise Linux 7.9. The nodes are using Red Hat Ceph Storage version 3.3z6 or 4.0 Access to the installation source for Red Hat Enterprise Linux 8.3. Procedure Set the OSD noout flag to prevent OSDs from getting marked down during the migration: Set the OSD nobackfill , norecover , norrebalance , noscrub and nodeep-scrub flags to avoid unnecessary load on the cluster and to avoid any data reshuffling when the node goes down for migration: Gracefully shut down all the OSD processes on the node: If using Red Hat Ceph Storage 3, disable the Red Hat Ceph Storage 3 repositories. Disable the tools repository: Disable the osd repository: If using Red Hat Ceph Storage 4, disable the Red Hat Ceph Storage 4 repositories. Disable the tools repository: Disable the osd repository: Install the leapp utility. See Upgrading from Red Hat Enterprise Linux 7 to Red Hat Enterprise Linux 8 . Run through the leapp preupgrade checks. See Assessing upgradability from the command line . Set PermitRootLogin yes in /etc/ssh/sshd_config . Restart the OpenSSH SSH daemon: Remove the iSCSI module from the Linux kernel: Perform the upgrade by following Performing the upgrade from RHEL 7 to RHEL 8 . Reboot the node. Enable the repositories for Red Hat Ceph Storage 4 for Red Hat Enterprise Linux 8. Enable the tools repository: Enable the osd repository: Install the ceph-osd package: Install the leveldb package: Restore the ceph.conf file from a node which has not been upgraded yet or from a node that has already had those files restored. Unset the noout , nobackfill , norecover , norrebalance , noscrub and nodeep-scrub flags: Switch any existing CRUSH buckets to the latest bucket type straw2 . Optional: If the OSDs were created using ceph-disk , and are still managed by ceph-disk , you must use ceph-volume to take over management of them. Mount each object storage device: Syntax Replace DRIVE with the storage device name and partition number. Replace OSD_ID with the OSD ID. Example Verify the ID_NUMBER is correct. Syntax Replace OSD_ID with the OSD ID. Example Repeat the above steps for any additional object store devices. Scan the newly mounted devices: Syntax Replace OSD_ID with the OSD ID. Example Repeat the above step for any additional object store devices. Activate the device: Syntax Replace OSD_ID with the OSD ID and UUID with the UUID printed in the scan output from earlier. Example Repeat the above step for any additional object store devices. Optional: If your OSDs were created with ceph-volume and you did not complete the step, start the OSD service now: Activate the OSDs: BlueStore Verify that the OSDs are up and in , and that they are in the active+clean state. On the osd: line under services: , ensure that all OSDs are up and in : Example Repeat the above steps on all OSD nodes until they have all been upgraded. If upgrading from Red Hat Ceph Storage 3, disallow pre-Nautilus OSDs and enable the Nautilus-only functionality: Note Failure to execute this step makes it impossible for OSDs to communicate after msgrv2 is enabled. Once all the daemons are updated after upgrading from Red Hat Ceph Storage 3 to Red Hat Ceph Storage 4, run the following steps: Enable the messenger v2 protocol, msgr2 : This instructs all Ceph Monitors that bind to the old default port of 6789, to also bind to the new port of 3300. On every node, import any configuration options in ceph.conf file into the storage cluster's configuration database: Example Note When you assimilate a config into your monitors, for example, if you have different config values set for the same set of options, the end result depends on the order in which the files are assimilated. Check the storage cluster's configuration database: Example Additional Resources See Manually upgrading a Red Hat Ceph Storage cluster and operating system in the Installation Guide for more information. See Upgrading from Red Hat Enterprise Linux 7 to Red Hat Enterprise Linux 8 for more information. 8.4. Manually upgrading Ceph Object Gateway nodes and their operating systems As a system administrator, you can manually upgrade the Ceph Object Gateway (RGW) software on a Red Hat Ceph Storage cluster node and the Red Hat Enterprise Linux operating system to a new major release at the same time. Important This procedure should be performed for each RGW node in the Ceph cluster, but only for one RGW node at a time. Ensure the current upgraded RGW has returned to normal operation prior to proceeding to the node to prevent any client access issues. Prerequisites A running Red Hat Ceph Storage cluster. The nodes are running Red Hat Enterprise Linux 7.9. The nodes are using Red Hat Ceph Storage version 3.3z6 or 4.1 Access to the installation source for Red Hat Enterprise Linux 8.3. Procedure Stop the Ceph Object Gateway service: If using Red Hat Ceph Storage 3, disable the Red Hat Ceph Storage 3 tool repository: If using Red Hat Ceph Storage 4, disable the Red Hat Ceph Storage 4 tools repository: Install the leapp utility. See Upgrading from Red Hat Enterprise Linux 7 to Red Hat Enterprise Linux 8 . Run through the leapp preupgrade checks. See Assessing upgradability from the command line . Set PermitRootLogin yes in /etc/ssh/sshd_config . Restart the OpenSSH SSH daemon: Remove the iSCSI module from the Linux kernel: Perform the upgrade by following Performing the upgrade from RHEL 7 to RHEL 8 . Reboot the node. Enable the tools repositories for Red Hat Ceph Storage 4 for Red Hat Enterprise Linux 8. Install the ceph-radosgw package: Optional: Install the packages for any Ceph services that are colocated on this node. Enable additional Ceph repositories if needed. Optional: Install the leveldb package which is needed by other Ceph services. Restore the ceph-client-admin.keyring and ceph.conf files from a node which has not been upgraded yet or from a node that has already had those files restored. Start the RGW service: Switch any existing CRUSH buckets to the latest bucket type straw2 . Verify the daemon is active: There is an rgw: line under services: . Example Repeat the above steps on all Ceph Object Gateway nodes until they have all been upgraded. Additional Resources See Manually upgrading a Red Hat Ceph Storage cluster and operating system in the Installation Guide for more information. See Upgrading from Red Hat Enterprise Linux 7 to Red Hat Enterprise Linux 8 for more information. 8.5. Manually upgrading the Ceph Dashboard node and its operating system As a system administrator, you can manually upgrade the Ceph Dashboard software on a Red Hat Ceph Storage cluster node and the Red Hat Enterprise Linux operating system to a new major release at the same time. Prerequisites A running Red Hat Ceph Storage cluster. The node is running Red Hat Enterprise Linux 7.9. The node is running Red Hat Ceph Storage version 3.3z6 or 4.1 Access to the installation source for Red Hat Enterprise Linux 8.3. Procedure Uninstall the existing dashboard from the cluster. Change to the /usr/share/cephmetrics-ansible directory: Run the purge.yml Ansible playbook: If using Red Hat Ceph Storage 3, disable the Red Hat Ceph Storage 3 tools repository: If using Red Hat Ceph Storage 4, disable the Red Hat Ceph Storage 4 tools repository: Install the leapp utility. See Upgrading from Red Hat Enterprise Linux 7 to Red Hat Enterprise Linux 8 . Run through the leapp preupgrade checks. See Assessing upgradability from the command line . Set PermitRootLogin yes in /etc/ssh/sshd_config . Restart the OpenSSH SSH daemon: Remove the iSCSI module from the Linux kernel: Perform the upgrade by following Performing the upgrade from RHEL 7 to RHEL 8 . Reboot the node. Enable the tools repository for Red Hat Ceph Storage 4 for Red Hat Enterprise Linux 8: Enable the Ansible repository: Configure ceph-ansible to manage the cluster. It will install the dashboard. Follow the instructions in Installing Red Hat Ceph Storage using Ansible , including the prerequisites. After you run ansible-playbook site.yml as a part of the above procedures, the URL for the dashboard will be printed. See Installing dashboard using Ansible in the Dashboard guide for more information on locating the URL and accessing the dashboard. Additional Resources See Manually upgrading a Red Hat Ceph Storage cluster and operating system in the Installation Guide for more information. See Upgrading from Red Hat Enterprise Linux 7 to Red Hat Enterprise Linux 8 for more information. See Installing dashboard using Ansible in the Dashboard guide for more information. 8.6. Manually upgrading Ceph Ansible nodes and reconfiguring settings Manually upgrade the Ceph Ansible software on a Red Hat Ceph Storage cluster node and the Red Hat Enterprise Linux operating system to a new major release at the same time. This procedure applies to both bare-metal and container deployments, unless specified. Important Before upgrading hostOS on the Ceph Ansible node, take a backup of group_vars and hosts file. Use the created backup before re-configuring the Ceph Ansible node. Prerequisites A running Red Hat Ceph Storage cluster. The node is running Red Hat Enterprise Linux 7.9. The node is running Red Hat Ceph Storage version 3.3z6 or 4.1 Access to the installation source for Red Hat Enterprise Linux 8.3. Procedure Enable the tools repository for Red Hat Ceph Storage 4 for Red Hat Enterprise Linux 8: Enable the Ansible repository: Configure ceph-ansible to manage the storage cluster. It will install the dashboard. Follow the instructions in Installing Red Hat Ceph Storage using Ansible , including the prerequisites. After you run ansible-playbook site.yml as a part of the above procedures, the URL for the dashboard will be printed. See Installing dashboard using Ansible in the Dashboard guide for more information on locating the URL and accessing the dashboard. Additional Resources See Manually upgrading a Red Hat Ceph Storage cluster and operating system in the Installation Guide for more information. See Upgrading from Red Hat Enterprise Linux 7 to Red Hat Enterprise Linux 8 for more information. See Installing dashboard using Ansible in the Dashboard guide for more information. 8.7. Manually upgrading the Ceph File System Metadata Server nodes and their operating systems You can manually upgrade the Ceph File System (CephFS) Metadata Server (MDS) software on a Red Hat Ceph Storage cluster and the Red Hat Enterprise Linux operating system to a new major release at the same time. Important Before you upgrade the storage cluster, reduce the number of active MDS ranks to one per file system. This eliminates any possible version conflicts between multiple MDS. In addition, take all standby nodes offline before upgrading. This is because the MDS cluster does not possess built-in versioning or file system flags. Without these features, multiple MDS might communicate using different versions of the MDS software, and could cause assertions or other faults to occur. Prerequisites A running Red Hat Ceph Storage cluster. The nodes are running Red Hat Enterprise Linux 7.9. The nodes are using Red Hat Ceph Storage version 3.3z6 or 4.1. Access to the installation source for Red Hat Enterprise Linux 8.3. Root-level access to all nodes in the storage cluster. Important The underlying XFS filesystem must be formatted with ftype=1 or with d_type support. Run the command xfs_info /var to ensure the ftype is set to 1 . If the value of ftype is not 1 , attach a new disk or create a volume. On top of this new device, create a new XFS filesystem and mount it on /var/lib/containers . Starting with Red Hat Enterprise Linux 8, mkfs.xfs enables ftype=1 by default. Procedure Reduce the number of active MDS ranks to 1: Syntax Example Wait for the cluster to stop all of the MDS ranks. When all of the MDS have stopped, only rank 0 should be active. The rest should be in standby mode. Check the status of the file system: Use systemctl to take all standby MDS offline: Confirm that only one MDS is online, and that it has rank 0 for the file system: Disable the tools repository for the operating system version: If you are upgrading from Red Hat Ceph Storage 3 on RHEL 7, disable the Red Hat Ceph Storage 3 tools repository: If you are using Red Hat Ceph Storage 4, disable the Red Hat Ceph Storage 4 tools repository: Install the leapp utility. For more information about leapp , refer to Upgrading from Red Hat Enterprise Linux 7 to Red Hat Enterprise Linux 8 . Run through the leapp preupgrade checks. For more information, refer to Assessing upgradability from the command line . Edit /etc/ssh/sshd_config and set PermitRootLogin to yes . Restart the OpenSSH SSH daemon: Remove the iSCSI module from the Linux kernel: Perform the upgrade. See Performing the upgrade from RHEL 7 to RHEL 8 . Reboot the MDS node. Enable the tools repositories for Red Hat Ceph Storage 4 for Red Hat Enterprise Linux 8: Install the ceph-mds package: Optional: Install the packages for any Ceph services that are colocated on this node. Enable additional Ceph repositories, if needed. Optional: Install the leveldb package, which is needed by other Ceph services: Restore the ceph-client-admin.keyring and ceph.conf files from a node that has not been upgraded yet, or from a node that has already had those files restored. Switch any existing CRUSH buckets to the latest bucket type straw2 . Start the MDS service: Verify that the daemon is active: Follow the same processes for the standby daemons. When you have finished restarting all of the MDS in standby, restore the value of max_mds for your cluster: Syntax Example 8.8. Recovering from an operating system upgrade failure on an OSD node As a system administrator, if you have a failure when using the procedure Manually upgrading Ceph OSD nodes and their operating systems , you can recover from the failure using the following procedure. In the procedure you will do a fresh install of Red Hat Enterprise Linux 8.4 on the node and still be able to recover the OSDs without any major backfilling of data besides the writes to the OSDs that were down while they were out. Important DO NOT touch the media backing the OSDs or their respective wal.db or block.db databases. Prerequisites A running Red Hat Ceph Storage cluster. An OSD node that failed to upgrade. Access to the installation source for Red Hat Enterprise Linux 8.4. Procedure Perform a standard installation of Red Hat Enterprise Linux 8.4 on the failed node and enable the Red Hat Enterprise Linux repositories. Performing a standard RHEL installation Enable the repositories for Red Hat Ceph Storage 4 for Red Hat Enterprise Linux 8. Enable the tools repository: Enable the osd repository: Install the ceph-osd package: Restore the ceph.conf file to /etc/ceph from a node which has not been upgraded yet or from a node that has already had those files restored. Start the OSD service: Activate the object store devices: Watch the recovery of the OSDs and cluster backfill writes to recovered OSDs: Monitor the output until all PGs are in state active+clean . Additional Resources See Manually upgrading a Red Hat Ceph Storage cluster and operating system in the Installation Guide for more information. See Upgrading from Red Hat Enterprise Linux 7 to Red Hat Enterprise Linux 8 for more information. 8.9. Additional Resources If you do not need to upgrade the operating system to a new major release, see Upgrading a Red Hat Ceph Storage cluster .
[user@admin ceph-ansible]USD ansible-playbook infrastructure-playbooks/docker-to-podman.yml -i hosts systemctl stop ceph-mon@ MONITOR_ID subscription-manager repos --disable=rhel-7-server-rhceph-3-tools-rpms subscription-manager repos --disable=rhel-7-server-rhceph-3-mon-rpms subscription-manager repos --disable=rhel-7-server-rhceph-4-tools-rpms subscription-manager repos --disable=rhel-7-server-rhceph-4-mon-rpms systemctl restart sshd.service modprobe -r iscsi subscription-manager repos --enable=rhceph-4-tools-for-rhel-8-x86_64-rpms subscription-manager repos --enable=rhceph-4-mon-for-rhel-8-x86_64-rpms dnf install ceph-mon dnf install ceph-mgr ceph osd getcrushmap -o backup-crushmap ceph osd crush set-all-straw-buckets-to-straw2 ceph mon enable-msgr2 ceph mon dump ceph config assimilate-conf -i /etc/ceph/ceph.conf ceph config dump ceph config generate-minimal-conf > /etc/ceph/ceph.conf.new mv /etc/ceph/ceph.conf.new /etc/ceph/ceph.conf dnf install leveldb systemctl start ceph-mon.target systemctl start ceph-mgr.target ceph -s mon: 3 daemons, quorum ceph4-mon,ceph4-mon2,ceph4-mon3 (age 2h) ceph -s mgr: ceph4-mon(active, since 2h), standbys: ceph4-mon3, ceph4-mon2 ceph osd set noout ceph osd set nobackfill ceph osd set norecover ceph osd set norebalance ceph osd set noscrub ceph osd set nodeep-scrub systemctl stop ceph-osd.target subscription-manager repos --disable=rhel-7-server-rhceph-3-tools-rpms subscription-manager repos --disable=rhel-7-server-rhceph-3-osd-rpms subscription-manager repos --disable=rhel-7-server-rhceph-4-tools-rpms subscription-manager repos --disable=rhel-7-server-rhceph-4-osd-rpms systemctl restart sshd.service modprobe -r iscsi subscription-manager repos --enable=rhceph-4-tools-for-rhel-8-x86_64-rpms subscription-manager repos --enable=rhceph-4-osd-for-rhel-8-x86_64-rpms dnf install ceph-osd dnf install leveldb ceph osd unset noout ceph osd unset nobackfill ceph osd unset norecover ceph osd unset norebalance ceph osd unset noscrub ceph osd unset nodeep-scrub ceph osd getcrushmap -o backup-crushmap ceph osd crush set-all-straw-buckets-to-straw2 /dev/ DRIVE /var/lib/ceph/osd/ceph- OSD_ID mount /dev/sdb1 /var/lib/ceph/osd/ceph-0 cat /var/lib/ceph/osd/ceph- OSD_ID /whoami cat /var/lib/ceph/osd/ceph-0/whoami 0 ceph-volume simple scan /var/lib/ceph/osd/ceph- OSD_ID ceph-volume simple scan /var/lib/ceph/osd/ceph-0 stderr: lsblk: /var/lib/ceph/osd/ceph-0: not a block device stderr: lsblk: /var/lib/ceph/osd/ceph-0: not a block device stderr: Unknown device, --name=, --path=, or absolute path in /dev/ or /sys expected. Running command: /usr/sbin/cryptsetup status /dev/sdb1 --> OSD 0 got scanned and metadata persisted to file: /etc/ceph/osd/0-0c9917f7-fce8-42aa-bdec-8c2cf2d536ba.json --> To take over management of this scanned OSD, and disable ceph-disk and udev, run: --> ceph-volume simple activate 0 0c9917f7-fce8-42aa-bdec-8c2cf2d536ba ceph-volume simple activate OSD_ID UUID ceph-volume simple activate 0 0c9917f7-fce8-42aa-bdec-8c2cf2d536ba Running command: /usr/bin/ln -snf /dev/sdb2 /var/lib/ceph/osd/ceph-0/journal Running command: /usr/bin/chown -R ceph:ceph /dev/sdb2 Running command: /usr/bin/systemctl enable ceph-volume@simple-0-0c9917f7-fce8-42aa-bdec-8c2cf2d536ba stderr: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-volume@simple-0-0c9917f7-fce8-42aa-bdec-8c2cf2d536ba.service /usr/lib/systemd/system/ceph-volume@.service. Running command: /usr/bin/ln -sf /dev/null /etc/systemd/system/ceph-disk@.service --> All ceph-disk systemd units have been disabled to prevent OSDs getting triggered by UDEV events Running command: /usr/bin/systemctl enable --runtime ceph-osd@0 stderr: Created symlink /run/systemd/system/ceph-osd.target.wants/ceph-osd@0.service /usr/lib/systemd/system/ceph-osd@.service. Running command: /usr/bin/systemctl start ceph-osd@0 --> Successfully activated OSD 0 with FSID 0c9917f7-fce8-42aa-bdec-8c2cf2d536ba systemctl start ceph-osd.target ceph-volume lvm activate --all ceph -s osd: 3 osds: 3 up (since 8s), 3 in (since 3M) ceph osd require-osd-release nautilus ceph mon enable-msgr2 ceph config assimilate-conf -i /etc/ceph/ceph.conf ceph config dump systemctl stop ceph-radosgw.target subscription-manager repos --disable=rhel-7-server-rhceph-3-tools-rpms subscription-manager repos --disable=rhel-7-server-rhceph-4-tools-rpms systemctl restart sshd.service modprobe -r iscsi subscription-manager repos --enable=rhceph-4-tools-for-rhel-8-x86_64-rpms dnf install ceph-radosgw dnf install leveldb systemctl start ceph-radosgw.target ceph osd getcrushmap -o backup-crushmap ceph osd crush set-all-straw-buckets-to-straw2 ceph -s rgw: 1 daemon active (jb-ceph4-rgw.rgw0) cd /usr/share/cephmetrics-ansible ansible-playbook -v purge.yml subscription-manager repos --disable=rhel-7-server-rhceph-3-tools-rpms subscription-manager repos --disable=rhel-7-server-rhceph-4-tools-rpms systemctl restart sshd.service modprobe -r iscsi subscription-manager repos --enable=rhceph-4-tools-for-rhel-8-x86_64-rpms subscription-manager repos --enable=ansible-2.9-for-rhel-8-x86_64-rpms subscription-manager repos --enable=rhceph-4-tools-for-rhel-8-x86_64-rpms subscription-manager repos --enable=ansible-2.9-for-rhel-8-x86_64-rpms ceph fs set FILE_SYSTEM_NAME max_mds 1 ceph fs set fs1 max_mds 1 ceph status systemctl stop ceph-mds.target ceph status subscription-manager repos --disable=rhel-7-server-rhceph-3-tools-rpms subscription-manager repos --disable=rhel-7-server-rhceph-4-tools-rpms systemctl restart sshd.service modprobe -r iscsi subscription-manager repos --enable=rhceph-4-tools-for-rhel-8-x86_64-rpms dnf install ceph-mds -y dnf install leveldb ceph osd getcrushmap -o backup-crushmap ceph osd crush set-all-straw-buckets-to-straw2 systemctl restart ceph-mds.target ceph -s ceph fs set FILE_SYSTEM_NAME max_mds ORIGINAL_VALUE ceph fs set fs1 max_mds 5 subscription-manager repos --enable=rhceph-4-tools-for-rhel-8-x86_64-rpms subscription-manager repos --enable=rhceph-4-osd-for-rhel-8-x86_64-rpms dnf install ceph-osd systemctl start ceph-osd.target ceph-volume lvm activate --all ceph -w