Chapter 1. Upgrading a self-hosted engine environment
Chapter 1. Upgrading a self-hosted engine environment 1.1. Upgrading a self-Hosted engine from Red Hat Virtualization 4.3 to 4.4 Upgrading a self-hosted engine environment from version 4.3 to 4.4 involves the following steps: Upgrade Considerations When planning to upgrade, see Red Hat Virtualization 4.4 upgrade considerations and known issues . When upgrading from Open Virtual Network (OVN) and Open vSwitch (OvS) 2.11 to OVN 2021 and OvS 2.15, the process is transparent to the user as long as the following conditions are met: The Manager is upgraded first. The ovirt-provider-ovn security groups must be disabled, before the host upgrade, for all OVN networks that are expected to work between hosts with OVN/OvS version 2.11. The hosts are upgraded to match OVN version 2021 or higher and OvS version 2.15. You must complete this step in the Administration Portal, so you can properly reconfigure OVN and refresh the certificates. The host is rebooted after an upgrade. Note To verify whether the provider and OVN were configured successfully on the host, check the OVN configured flag on the General tab for the host. If the OVN Configured is set to No , click Management Refresh Capabilities . This setting is also available in the REST API. If refreshing the capabilities fails, you can configure OVN by reinstalling the host from Manager 4.4 or higher. Make sure you meet the prerequisites, including enabling the correct repositories Use the Log Collection Analysis tool and Image Discrepancies tool to check for issues that might prevent a successful upgrade Migrate any virtual machines that are running on the same host as the Manager virtual machine to another host in the same cluster Place the environment in global maintenance mode Update the 4.3 Manager to the latest version of 4.3 Upgrade the Manager from 4.3 to 4.4 Upgrade the self-hosted engine nodes, and any standard hosts, while reducing virtual machine downtime (Optional) Upgrade RHVH while preserving local storage Update the compatibility version of the clusters Reboot any running or suspended virtual machines to update their configuration Update the compatibility version of the data centers 1.1.1. Prerequisites Plan for any necessary virtual machine downtime. After you update the clusters' compatibility versions during the upgrade, a new hardware configuration is automatically applied to each virtual machine once it reboots. You must reboot any running or suspended virtual machines as soon as possible to apply the configuration changes. Ensure your environment meets the requirements for Red Hat Virtualization 4.4. For a complete list of prerequisites, see the Planning and Prerequisites Guide . When upgrading Red Hat Virtualization Manager, it is recommended that you use one of the existing hosts. If you decide to use a new host, you must assign a unique name to the new host and then add it to the existing cluster before you begin the upgrade procedure. 1.1.2. Analyzing the Environment It is recommended to run the Log Collection Analysis tool and the Image Discrepancies tool prior to performing updates and for troubleshooting. These tools analyze your environment for known issues that might prevent you from performing an update, and provide recommendations to resolve them. 1.1.3. Log Collection Analysis tool Run the Log Collection Analysis tool prior to performing updates and for troubleshooting. The tool analyzes your environment for known issues that might prevent you from performing an update, and provides recommendations to resolve them. The tool gathers detailed information about your system and presents it as an HTML file. Prerequisites Ensure the Manager has the correct repositories enabled. For the list of required repositories, see Enabling the Red Hat Virtualization Manager Repositories for Red Hat Virtualization 4.3. Updates to the Red Hat Virtualization Manager are released through the Content Delivery Network. Procedure Install the Log Collection Analysis tool on the Manager machine: Run the tool: A detailed report is displayed. By default, the report is saved to a file called analyzer_report.html . To save the file to a specific location, use the --html flag and specify the location: # rhv-log-collector-analyzer --live --html=/ directory / filename .html You can use the ELinks text mode web browser to read the analyzer reports within the terminal. To install the ELinks browser: Launch ELinks and open analyzer_report.html . To navigate the report, use the following commands in ELinks: Insert to scroll up Delete to scroll down PageUp to page up PageDown to page down Left Bracket to scroll left Right Bracket to scroll right 1.1.3.1. Monitoring snapshot health with the image discrepancies tool The RHV Image Discrepancies tool analyzes image data in the Storage Domain and RHV Database. It alerts you if it finds discrepancies in volumes and volume attributes, but does not fix those discrepancies. Use this tool in a variety of scenarios, such as: Before upgrading versions, to avoid carrying over broken volumes or chains to the new version. Following a failed storage operation, to detect volumes or attributes in a bad state. After restoring the RHV database or storage from backup. Periodically, to detect potential problems before they worsen. To analyze a snapshot- or live storage migration-related issues, and to verify system health after fixing these types of problems. Prerequisites Required Versions: this tool was introduced in RHV version 4.3.8 with rhv-log-collector-analyzer-0.2.15-0.el7ev . Because data collection runs simultaneously at different places and is not atomic, stop all activity in the environment that can modify the storage domains. That is, do not create or remove snapshots, edit, move, create, or remove disks. Otherwise, false detection of inconsistencies may occur. Virtual Machines can remain running normally during the process. Procedure To run the tool, enter the following command on the RHV Manager: If the tool finds discrepancies, rerun it to confirm the results, especially if there is a chance some operations were performed while the tool was running. Note This tool includes any Export and ISO storage domains and may report discrepancies for them. If so, these can be ignored, as these storage domains do not have entries for images in the RHV database. Understanding the results The tool reports the following: If there are volumes that appear on the storage but are not in the database, or appear in the database but are not on the storage. If some volume attributes differ between the storage and the database. Sample output: 1.1.4. Migrating virtual machines from the self-hosted engine host Only the Manager virtual machine should remain on the host until after you have finished upgrading the host. Migrate any virtual machines other than the Manager virtual machine to another host in the same cluster. You can use Live Migration to minimize virtual machine down-time. For more information, see Migrating Virtual Machines Between Hosts in the Virtual Machine Management Guide for more information. 1.1.5. Enabling global maintenance mode You must place the self-hosted engine environment in global maintenance mode before performing any setup or upgrade tasks on the Manager virtual machine. Procedure Log in to one of the self-hosted engine nodes and enable global maintenance mode: # hosted-engine --set-maintenance --mode=global Confirm that the environment is in global maintenance mode before proceeding: # hosted-engine --vm-status You should see a message indicating that the cluster is in global maintenance mode. You can now update the Manager to the latest version of 4.3. 1.1.6. Updating the Red Hat Virtualization Manager Prerequisites Ensure the Manager has the correct repositories enabled . For the list of required repositories, see Enabling the Red Hat Virtualization Manager Repositories for Red Hat Virtualization 4.3. Updates to the Red Hat Virtualization Manager are released through the Content Delivery Network. Procedure On the Manager machine, check if updated packages are available: Update the setup packages: # yum update ovirt\*setup\* rh\*vm-setup-plugins Update the Red Hat Virtualization Manager with the engine-setup script. The engine-setup script prompts you with some configuration questions, then stops the ovirt-engine service, downloads and installs the updated packages, backs up and updates the database, performs post-installation configuration, and starts the ovirt-engine service. When the script completes successfully, the following message appears: Note The engine-setup script is also used during the Red Hat Virtualization Manager installation process, and it stores the configuration values supplied. During an update, the stored values are displayed when previewing the configuration, and might not be up to date if engine-config was used to update configuration after installation. For example, if engine-config was used to update SANWipeAfterDelete to true after installation, engine-setup will output "Default SAN wipe after delete: False" in the configuration preview. However, the updated values will not be overwritten by engine-setup . Important The update process might take some time. Do not stop the process before it completes. Update the base operating system and any optional packages installed on the Manager: Important If you encounter a required Ansible package conflict during the update, see Cannot perform yum update on my RHV manager (ansible conflict) . Important If any kernel packages were updated, reboot the machine to complete the update. You can now upgrade the Manager to 4.4. 1.1.7. Upgrading the Red Hat Virtualization Manager from 4.3 to 4.4 The Red Hat Virtualization Manager 4.4 is only supported on Red Hat Enterprise Linux versions 8.2 to 8.6. You need to do a clean installation of Red Hat Enterprise Linux 8.6, or Red Hat Virtualization Host on the self-hosted engine host, even if you are using the same physical machine that you use to run the RHV 4.3 self-hosted engine. The upgrade process requires restoring Red Hat Virtualization Manager 4.3 backup files onto the Red Hat Virtualization Manager 4.4 virtual machine. Prerequisites All data centers and clusters in the environment must have the cluster compatibility level set to version 4.2 or 4.3. All virtual machines in the environment must have the cluster compatibility level set to version 4.3. Make note of the MAC address of the self-hosted engine if you are using DHCP and want to use the same IP address. The deploy script prompts you for this information. During the deployment you need to provide a new storage domain for the Manager machine. The deployment script renames the 4.3 storage domain and retains its data to enable disaster recovery. Set the cluster scheduling policy to cluster_maintenance in order to prevent automatic virtual machine migration during the upgrade. Caution In an environment with multiple highly available self-hosted engine nodes, you need to detach the storage domain hosting the version 4.3 Manager after upgrading the Manager to 4.4. Use a dedicated storage domain for the 4.4 self-hosted engine deployment. If you use an external CA to sign HTTPS certificates, follow the steps in Replacing the Red Hat Virtualization Manager CA Certificate in the Administration Guide . The backup and restore include the 3rd-party certificate, so you should be able to log in to the Administration portal after the upgrade. Ensure the CA certificate is added to system-wide trust stores of all clients to ensure the foreign menu of virt-viewer works. See BZ#1313379 for more information. Note Connected hosts and virtual machines can continue to work while the Manager is being upgraded. Procedure Log in to the Manager virtual machine and shut down the engine service. # systemctl stop ovirt-engine Back up the Red Hat Virtualization Manager 4.3 environment. # engine-backup --scope=all --mode=backup --file=backup.bck --log=backuplog.log Copy the backup file to a storage device outside of the RHV environment. Shut down the self-hosted engine. # shutdown Note If you want to reuse the self-hosted engine virtual machine to deploy the Red Hat Virtualization Manager 4.4, note the MAC address of the self-hosted engine network interface before you shut it down. Make sure that the self-hosted engine is shut down. # hosted-engine --vm-status | grep -E 'Engine status|Hostname' Note If any of the hosts report the detail field as Up , log in to that specific host and shut it down with the hosted-engine --vm-shutdown command. Install RHVH 4.4 or Red Hat Enterprise Linux 8.6 on the existing node currently running the Manager virtual machine to use it as the self-hosted engine deployment host. See Installing the Self-hosted Engine Deployment Host for more information. Note It is recommended that you use one of the existing hosts. If you decide to use a new host, you must assign a unique name to the new host and then add it to the existing cluster before you begin the upgrade procedure. Install the self-hosted engine deployment tool. # yum install ovirt-hosted-engine-setup Copy the backup file to the host. Log in to the Manager host and deploy the self-hosted engine with the backup file: # hosted-engine --deploy --restore-from-file=/ path /backup.bck Note tmux enables the deployment script to continue if the connection to the server is interrupted, so you can reconnect and attach to the deployment and continue. Otherwise, if the connection is interrupted during deployment, the deployment fails. To run the deployment script using tmux , enter the tmux command before you run the deployment script: # tmux # hosted-engine --deploy --restore-from-file=backup.bck The deployment script automatically disables global maintenance mode and calls the HA agent to start the self-hosted engine virtual machine. The upgraded host with the 4.4 self-hosted engine reports that HA mode is active, but the other hosts report that global maintenance mode is still enabled as they are still connected to the old self-hosted engine storage. Detach the storage domain that hosts the Manager 4.3 machine. For details, see Detaching a Storage Domain from a Data Center in the Administration Guide . Log in to the Manager virtual machine and shut down the engine service. # systemctl stop ovirt-engine Ensure the Manager has the correct repositories enabled. For the list of required repositories, see Enabling the Red Hat Virtualization Manager Repositories for Red Hat Virtualization 4.4. Updates to the Red Hat Virtualization Manager are released through the Content Delivery Network. Install optional extension packages if they were installed on the Red Hat Virtualization Manager 4.3 machine. # yum install ovirt-engine-extension-aaa-ldap ovirt-engine-extension-aaa-misc Note The ovirt-engine-extension-aaa-ldap is deprecated. For new installations, use Red Hat Single Sign On. For more information, see Installing and Configuring Red Hat Single Sign-On in the Administration Guide . Note The configuration for these package extensions must be manually reapplied because they are not migrated as part of the backup and restore process. Configure the Manager by running the engine-setup command: # engine-setup The Red Hat Virtualization Manager 4.4 is now installed, with the cluster compatibility version set to 4.2 or 4.3, whichever was the preexisting cluster compatibility version. Additional resources Installing Red Hat Virtualization as a self-hosted engine using the command line You can now update the self-hosted engine nodes, and then any standard hosts. The procedure is the same for both host types. 1.1.8. Migrating hosts and virtual machines from RHV 4.3 to 4.4 You can migrate hosts and virtual machines from Red Hat Virtualization 4.3 to 4.4 such that you minimize the downtime of virtual machines in your environment. This process requires migrating all virtual machines from one host so as to make that host available to upgrade to RHV 4.4. After the upgrade, you can reattach the host to the Manager. Warning When installing or reinstalling the host's operating system, Red Hat strongly recommends that you first detach any existing non-OS storage that is attached to the host to avoid accidental initialization of these disks, and with that, potential data loss. Note CPU-passthrough virtual machines might not migrate properly from RHV 4.3 to RHV 4.4. RHV 4.3 and RHV 4.4 are based on RHEL 7 and RHEL 8, respectively, which have different kernel versions with different CPU flags and microcodes. This can cause problems in migrating CPU-passthrough virtual machines. Prerequisites Hosts for RHV 4.4 require Red Hat Enterprise Linux versions 8.2 to 8.6. A clean installation of Red Hat Enterprise Linux 8.6, or Red Hat Virtualization Host 4.4 is required, even if you are using the same physical machine that you use to run hosts for RHV 4.3. Red Hat Virtualization Manager 4.4 is installed and running. The compatibility level of the data center and cluster to which the hosts belong is set to 4.2 or 4.3. All data centers and clusters in the environment must have the cluster compatibility level set to version 4.2 or 4.3 before you start the procedure. Procedure Pick a host to upgrade and migrate that host's virtual machines to another host in the same cluster. You can use Live Migration to minimize virtual machine downtime. For more information, see Migrating Virtual Machines Between Hosts in the Virtual Machine Management Guide . Put the host into maintenance mode and remove the host from the Manager. For more information, see Removing a Host in the Administration Guide . Install Red Hat Enterprise Linux 8.6, or RHVH 4.4. For more information, see Installing Hosts for Red Hat Virtualization in one of the Installing Red Hat Virtualization guides. Install the appropriate packages to enable the host for RHV 4.4. For more information, see Installing Hosts for Red Hat Virtualization in one of the Installing Red Hat Virtualization guides. Add this host to the Manager, assigning it to the same cluster. You can now migrate virtual machines onto this host. For more information, see Adding Standard Hosts to the Manager in one of the Installing Red Hat Virtualization guides. Repeat these steps to migrate virtual machines and upgrade hosts for the rest of the hosts in the same cluster, one by one, until all are running Red Hat Virtualization 4.4. Additional resources Installing Red Hat Virtualization as a self-hosted engine using the command line Installing Red Hat Virtualization as a standalone Manager with local databases Installing Red Hat Virtualization as a standalone Manager with remote databases 1.1.9. Upgrading RHVH while preserving local storage Environments with local storage cannot migrate virtual machines to a host in another cluster because the local storage is not shared with other storage domains. To upgrade RHVH 4.3 hosts that have a local storage domain, reinstall the host while preserving the local storage, create a new local storage domain in the 4.4 environment, and import the local storage into the new domain. Prerequisites Red Hat Virtualization Manager 4.4 is installed and running. The compatibility level of the data center and cluster to which the host belongs is set to 4.2 or 4.3. Procedure Ensure that the local storage on the RHVH 4.3 host's local storage is in maintenance mode before starting this process. Complete these steps: Open the Data Centers tab. Click the Storage tab in the Details pane and select the storage domain in the results list. Click Maintenance . Reinstall the Red Hat Virtualization Host, as described in Installing Red Hat Virtualization Host in the Installation Guide . Important When selecting the device on which to install RHVH from the Installation Destination screen, do not select the device(s) storing the virtual machines. Only select the device where the operating system should be installed. If you are using Kickstart to install the host, ensure that you preserve the devices containing the virtual machines by adding the following to the Kickstart file, replacing `device` with the relevant device. # clearpart --all --drives= device For more information on using Kickstart, see Kickstart references in Red Hat Enterprise Linux 8 Performing an advanced RHEL installation . On the reinstalled host, create a directory, for example /data in which to recover the environment. # mkdir /data Mount the local storage in the new directory. In our example, /dev/sdX1 is the local storage: # mount /dev/sdX1 /data Set the following permissions for the new directory. # chown -R 36:36 /data # chmod -R 0755 /data Red Hat recommends that you also automatically mount the local storage via /etc/fstab in case the server requires a reboot: # blkid | grep -i sdX1 /dev/sdX1: UUID="a81a6879-3764-48d0-8b21-2898c318ef7c" TYPE="ext4" # vi /etc/fstab UUID="a81a6879-3764-48d0-8b21-2898c318ef7c" /data ext4 defaults 0 0 In the Administration Portal, create a data center and select Local in the Storage Type drop-down menu. Configure a cluster on the new data center. See Creating a New Cluster in the Administration Guide for more information. Add the host to the Manager. See Adding Standard Hosts to the Red Hat Virtualization Manager in one of the Installing Red Hat Virtualization guides for more information. On the host, create a new directory that will be used to create the initial local storage domain. For example: # mkdir -p /localfs # chown 36:36 /localfs # chmod -R 0755 /localfs In the Administration Portal, open the Storage tab and click New Domain to create a new local storage domain. Set the name to localfs and set the path to /localfs . Once the local storage is active, click Import Domain and set the domain's details. For example, define Data as the name, Local on Host as the storage type and /data as the path. Click OK to confirm the message that appears informing you that storage domains are already attached to the data center. Activate the new storage domain: Open the Data Centers tab. Click the Storage tab in the details pane and select the new data storage domain in the results list. Click Activate . Once the new storage domain is active, import the virtual machines and their disks: In the Storage tab, select data . Select the VM Import tab in the details pane, select the virtual machines and click Import . See Importing Virtual Machines from a Data Domain in the Virtual Machine Management Guide for more details. Once you have ensured that all virtual machines have been successfully imported and are functioning properly, you can move localfs to maintenance mode. Click the Storage tab and select localfs from the results list. Click the Data Center tab in the details pane. Click Maintenance, then click OK to move the storage domain to maintenance mode. Click Detach . The Detach Storage confirmation window opens. Click OK . You have now upgraded the host to version 4.4, created a new local storage domain, and imported the 4.3 storage domain and its virtual machines. 1.1.10. Changing the Cluster Compatibility Version Red Hat Virtualization clusters have a compatibility version. The cluster compatibility version indicates the features of Red Hat Virtualization supported by all of the hosts in the cluster. The cluster compatibility is set according to the version of the least capable host operating system in the cluster. Prerequisites To change the cluster compatibility level, you must first update all the hosts in your cluster to a level that supports your desired compatibility level. Check if there is an icon to the host indicating an update is available. Limitations Virtio NICs are enumerated as a different device after upgrading the cluster compatibility level to 4.6. Therefore, the NICs might need to be reconfigured. Red Hat recommends that you test the virtual machines before you upgrade the cluster by setting the cluster compatibility level to 4.6 on the virtual machine and verifying the network connection. If the network connection for the virtual machine fails, configure the virtual machine with a custom emulated machine that matches the current emulated machine, for example pc-q35-rhel8.3.0 for 4.5 compatibility version, before upgrading the cluster. Procedure In the Administration Portal, click Compute Clusters . Select the cluster to change and click Edit . On the General tab, change the Compatibility Version to the desired value. Click OK . The Change Cluster Compatibility Version confirmation dialog opens. Click OK to confirm. Important An error message might warn that some virtual machines and templates are incorrectly configured. To fix this error, edit each virtual machine manually. The Edit Virtual Machine window provides additional validations and warnings that show what to correct. Sometimes the issue is automatically corrected and the virtual machine's configuration just needs to be saved again. After editing each virtual machine, you will be able to change the cluster compatibility version. 1.1.11. Changing Virtual Machine Cluster Compatibility After updating a cluster's compatibility version, you must update the cluster compatibility version of all running or suspended virtual machines by rebooting them from the Administration Portal, or using the REST API, or from within the guest operating system. Virtual machines that require a reboot are marked with the pending changes icon ( ). The Manager virtual machine does not need to be rebooted. Although you can wait to reboot the virtual machines at a convenient time, rebooting immediately is highly recommended so that the virtual machines use the latest configuration. Any virtual machine that has not been rebooted runs with the configuration, and subsequent configuration changes made to the virtual machine might overwrite its pending cluster compatibility changes. Procedure In the Administration Portal, click Compute Virtual Machines . Check which virtual machines require a reboot. In the Vms: search bar, enter the following query: next_run_config_exists=True The search results show all virtual machines with pending changes. Select each virtual machine and click Restart . Alternatively, if necessary you can reboot a virtual machine from within the virtual machine itself. When the virtual machine starts, the new compatibility version is automatically applied. Note You cannot change the cluster compatibility version of a virtual machine snapshot that is in preview. You must first commit or undo the preview. 1.1.12. Changing the Data Center Compatibility Version Red Hat Virtualization data centers have a compatibility version. The compatibility version indicates the version of Red Hat Virtualization with which the data center is intended to be compatible. All clusters in the data center must support the desired compatibility level. Prerequisites To change the data center compatibility level, you must first update the compatibility version of all clusters and virtual machines in the data center. Procedure In the Administration Portal, click Compute Data Centers . Select the data center to change and click Edit . Change the Compatibility Version to the desired value. Click OK . The Change Data Center Compatibility Version confirmation dialog opens. Click OK to confirm.
yum install rhv-log-collector-analyzer rhv-log-collector-analyzer --live rhv-log-collector-analyzer --live --html=/ directory / filename .html yum install -y elinks elinks /home/user1/analyzer_report.html rhv-image-discrepancies Checking storage domain c277ad93-0973-43d9-a0ca-22199bc8e801 Looking for missing images No missing images found Checking discrepancies between SD/DB attributes image ef325650-4b39-43cf-9e00-62b9f7659020 has a different attribute capacity on storage(2696984576) and on DB(2696986624) image 852613ce-79ee-4adc-a56a-ea650dcb4cfa has a different attribute capacity on storage(5424252928) and on DB(5424254976) Checking storage domain c64637b4-f0e8-408c-b8af-6a52946113e2 Looking for missing images No missing images found Checking discrepancies between SD/DB attributes No discrepancies found hosted-engine --set-maintenance --mode=global hosted-engine --vm-status engine-upgrade-check yum update ovirt\*setup\* rh\*vm-setup-plugins engine-setup Execution of setup completed successfully yum update --nobest systemctl stop ovirt-engine engine-backup --scope=all --mode=backup --file=backup.bck --log=backuplog.log shutdown hosted-engine --vm-status | grep -E 'Engine status|Hostname' yum install ovirt-hosted-engine-setup hosted-engine --deploy --restore-from-file=/ path /backup.bck tmux hosted-engine --deploy --restore-from-file=backup.bck systemctl stop ovirt-engine yum install ovirt-engine-extension-aaa-ldap ovirt-engine-extension-aaa-misc engine-setup clearpart --all --drives= device mkdir /data mount /dev/sdX1 /data chown -R 36:36 /data chmod -R 0755 /data blkid | grep -i sdX1 /dev/sdX1: UUID="a81a6879-3764-48d0-8b21-2898c318ef7c" TYPE="ext4" vi /etc/fstab UUID="a81a6879-3764-48d0-8b21-2898c318ef7c" /data ext4 defaults 0 0 mkdir -p /localfs chown 36:36 /localfs chmod -R 0755 /localfs next_run_config_exists=True