Chapter 6. Updating
Chapter 6. Updating 6.1. Updating OpenShift Virtualization Learn how to keep OpenShift Virtualization updated and compatible with OpenShift Container Platform. 6.1.1. About updating OpenShift Virtualization When you install OpenShift Virtualization, you select an update channel and an approval strategy. The update channel determines the versions that OpenShift Virtualization will be updated to. The approval strategy setting determines whether updates occur automatically or require manual approval. Both settings can impact supportability. 6.1.1.1. Recommended settings To maintain a supportable environment, use the following settings: Update channel: stable Approval strategy: Automatic With these settings, the update process automatically starts when a new version of the Operator is available in the stable channel. This ensures that your OpenShift Virtualization and OpenShift Container Platform versions remain compatible, and that your version of OpenShift Virtualization is suitable for production environments. Note Each minor version of OpenShift Virtualization is supported only if you run the corresponding OpenShift Container Platform version. For example, you must run OpenShift Virtualization 4.15 on OpenShift Container Platform 4.15. 6.1.1.2. What to expect The amount of time an update takes to complete depends on your network connection. Most automatic updates complete within fifteen minutes. Updating OpenShift Virtualization does not interrupt network connections. Data volumes and their associated persistent volume claims are preserved during an update. Important If you have virtual machines running that use hostpath provisioner storage, they cannot be live migrated and might block an OpenShift Container Platform cluster update. As a workaround, you can reconfigure the virtual machines so that they can be powered off automatically during a cluster update. Set the evictionStrategy field to None and the runStrategy field to Always . 6.1.1.3. How updates work Operator Lifecycle Manager (OLM) manages the lifecycle of the OpenShift Virtualization Operator. The Marketplace Operator, which is deployed during OpenShift Container Platform installation, makes external Operators available to your cluster. OLM provides z-stream and minor version updates for OpenShift Virtualization. Minor version updates become available when you update OpenShift Container Platform to the minor version. You cannot update OpenShift Virtualization to the minor version without first updating OpenShift Container Platform. 6.1.1.4. RHEL 9 compatibility OpenShift Virtualization 4.15 is based on Red Hat Enterprise Linux (RHEL) 9. You can update to OpenShift Virtualization 4.15 from a version that was based on RHEL 8 by following the standard OpenShift Virtualization update procedure. No additional steps are required. As in versions, you can perform the update without disrupting running workloads. OpenShift Virtualization 4.15 supports live migration from RHEL 8 nodes to RHEL 9 nodes. 6.1.1.4.1. RHEL 9 machine type All VM templates that are included with OpenShift Virtualization now use the RHEL 9 machine type by default: machineType: pc-q35-rhel9.<y>.0 , where <y> is a single digit corresponding to the latest minor version of RHEL 9. For example, the value pc-q35-rhel9.2.0 is used for RHEL 9.2. Updating OpenShift Virtualization does not change the machineType value of any existing VMs. These VMs continue to function as they did before the update. You can optionally change a VM's machine type so that it can benefit from RHEL 9 improvements. Important Before you change a VM's machineType value, you must shut down the VM. 6.1.2. Monitoring update status To monitor the status of a OpenShift Virtualization Operator update, watch the cluster service version (CSV) PHASE . You can also monitor the CSV conditions in the web console or by running the command provided here. Note The PHASE and conditions values are approximations that are based on available information. Prerequisites Log in to the cluster as a user with the cluster-admin role. Install the OpenShift CLI ( oc ). Procedure Run the following command: USD oc get csv -n openshift-cnv Review the output, checking the PHASE field. For example: Example output VERSION REPLACES PHASE 4.9.0 kubevirt-hyperconverged-operator.v4.8.2 Installing 4.9.0 kubevirt-hyperconverged-operator.v4.9.0 Replacing Optional: Monitor the aggregated status of all OpenShift Virtualization component conditions by running the following command: USD oc get hyperconverged kubevirt-hyperconverged -n openshift-cnv \ -o=jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.message}{"\n"}{end}' A successful upgrade results in the following output: Example output ReconcileComplete True Reconcile completed successfully Available True Reconcile completed successfully Progressing False Reconcile completed successfully Degraded False Reconcile completed successfully Upgradeable True Reconcile completed successfully 6.1.3. VM workload updates When you update OpenShift Virtualization, virtual machine workloads, including libvirt , virt-launcher , and qemu , update automatically if they support live migration. Note Each virtual machine has a virt-launcher pod that runs the virtual machine instance (VMI). The virt-launcher pod runs an instance of libvirt , which is used to manage the virtual machine (VM) process. You can configure how workloads are updated by editing the spec.workloadUpdateStrategy stanza of the HyperConverged custom resource (CR). There are two available workload update methods: LiveMigrate and Evict . Because the Evict method shuts down VMI pods, only the LiveMigrate update strategy is enabled by default. When LiveMigrate is the only update strategy enabled: VMIs that support live migration are migrated during the update process. The VM guest moves into a new pod with the updated components enabled. VMIs that do not support live migration are not disrupted or updated. If a VMI has the LiveMigrate eviction strategy but does not support live migration, it is not updated. If you enable both LiveMigrate and Evict : VMIs that support live migration use the LiveMigrate update strategy. VMIs that do not support live migration use the Evict update strategy. If a VMI is controlled by a VirtualMachine object that has runStrategy: Always set, a new VMI is created in a new pod with updated components. Migration attempts and timeouts When updating workloads, live migration fails if a pod is in the Pending state for the following periods: 5 minutes If the pod is pending because it is Unschedulable . 15 minutes If the pod is stuck in the pending state for any reason. When a VMI fails to migrate, the virt-controller tries to migrate it again. It repeats this process until all migratable VMIs are running on new virt-launcher pods. If a VMI is improperly configured, however, these attempts can repeat indefinitely. Note Each attempt corresponds to a migration object. Only the five most recent attempts are held in a buffer. This prevents migration objects from accumulating on the system while retaining information for debugging. 6.1.3.1. Configuring workload update methods You can configure workload update methods by editing the HyperConverged custom resource (CR). Prerequisites To use live migration as an update method, you must first enable live migration in the cluster. Note If a VirtualMachineInstance CR contains evictionStrategy: LiveMigrate and the virtual machine instance (VMI) does not support live migration, the VMI will not update. Procedure To open the HyperConverged CR in your default editor, run the following command: USD oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv Edit the workloadUpdateStrategy stanza of the HyperConverged CR. For example: apiVersion: hco.kubevirt.io/v1beta1 kind: HyperConverged metadata: name: kubevirt-hyperconverged spec: workloadUpdateStrategy: workloadUpdateMethods: 1 - LiveMigrate 2 - Evict 3 batchEvictionSize: 10 4 batchEvictionInterval: "1m0s" 5 # ... 1 The methods that can be used to perform automated workload updates. The available values are LiveMigrate and Evict . If you enable both options as shown in this example, updates use LiveMigrate for VMIs that support live migration and Evict for any VMIs that do not support live migration. To disable automatic workload updates, you can either remove the workloadUpdateStrategy stanza or set workloadUpdateMethods: [] to leave the array empty. 2 The least disruptive update method. VMIs that support live migration are updated by migrating the virtual machine (VM) guest into a new pod with the updated components enabled. If LiveMigrate is the only workload update method listed, VMIs that do not support live migration are not disrupted or updated. 3 A disruptive method that shuts down VMI pods during upgrade. Evict is the only update method available if live migration is not enabled in the cluster. If a VMI is controlled by a VirtualMachine object that has runStrategy: Always configured, a new VMI is created in a new pod with updated components. 4 The number of VMIs that can be forced to be updated at a time by using the Evict method. This does not apply to the LiveMigrate method. 5 The interval to wait before evicting the batch of workloads. This does not apply to the LiveMigrate method. Note You can configure live migration limits and timeouts by editing the spec.liveMigrationConfig stanza of the HyperConverged CR. To apply your changes, save and exit the editor. 6.1.3.2. Viewing outdated VM workloads You can view a list of outdated virtual machine (VM) workloads by using the CLI. Note If there are outdated virtualization pods in your cluster, the OutdatedVirtualMachineInstanceWorkloads alert fires. Procedure To view a list of outdated virtual machine instances (VMIs), run the following command: USD oc get vmi -l kubevirt.io/outdatedLauncherImage --all-namespaces Note To ensure that VMIs update automatically, configure workload updates. 6.1.4. Control Plane Only updates Every even-numbered minor version of OpenShift Container Platform, including 4.10 and 4.12, is an Extended Update Support (EUS) version. However, because Kubernetes design mandates serial minor version updates, you cannot directly update from one EUS version to the . After you update from the source EUS version to the odd-numbered minor version, you must sequentially update OpenShift Virtualization to all z-stream releases of that minor version that are on your update path. When you have upgraded to the latest applicable z-stream version, you can then update OpenShift Container Platform to the target EUS minor version. When the OpenShift Container Platform update succeeds, the corresponding update for OpenShift Virtualization becomes available. You can now update OpenShift Virtualization to the target EUS version. For more information about EUS versions, see the Red Hat OpenShift Container Platform Life Cycle Policy . 6.1.4.1. Prerequisites Before beginning a Control Plane Only update, you must: Pause worker nodes' machine config pools before you start a Control Plane Only update so that the workers are not rebooted twice. Disable automatic workload updates before you begin the update process. This is to prevent OpenShift Virtualization from migrating or evicting your virtual machines (VMs) until you update to your target EUS version. Note By default, OpenShift Virtualization automatically updates workloads, such as the virt-launcher pod, when you update the OpenShift Virtualization Operator. You can configure this behavior in the spec.workloadUpdateStrategy stanza of the HyperConverged custom resource. Learn more about Performing a Control Plane Only update . 6.1.4.2. Preventing workload updates during a Control Plane Only update When you update from one Extended Update Support (EUS) version to the , you must manually disable automatic workload updates to prevent OpenShift Virtualization from migrating or evicting workloads during the update process. Prerequisites You are running an EUS version of OpenShift Container Platform and want to update to the EUS version. You have not yet updated to the odd-numbered version in between. You read "Preparing to perform a Control Plane Only update" and learned the caveats and requirements that pertain to your OpenShift Container Platform cluster. You paused the worker nodes' machine config pools as directed by the OpenShift Container Platform documentation. It is recommended that you use the default Automatic approval strategy. If you use the Manual approval strategy, you must approve all pending updates in the web console. For more details, refer to the "Manually approving a pending Operator update" section. Procedure Run the following command and record the workloadUpdateMethods configuration: USD oc get kv kubevirt-kubevirt-hyperconverged \ -n openshift-cnv -o jsonpath='{.spec.workloadUpdateStrategy.workloadUpdateMethods}' Turn off all workload update methods by running the following command: USD oc patch hyperconverged kubevirt-hyperconverged -n openshift-cnv \ --type json -p '[{"op":"replace","path":"/spec/workloadUpdateStrategy/workloadUpdateMethods", "value":[]}]' Example output hyperconverged.hco.kubevirt.io/kubevirt-hyperconverged patched Ensure that the HyperConverged Operator is Upgradeable before you continue. Enter the following command and monitor the output: USD oc get hyperconverged kubevirt-hyperconverged -n openshift-cnv -o json | jq ".status.conditions" Example 6.1. Example output [ { "lastTransitionTime": "2022-12-09T16:29:11Z", "message": "Reconcile completed successfully", "observedGeneration": 3, "reason": "ReconcileCompleted", "status": "True", "type": "ReconcileComplete" }, { "lastTransitionTime": "2022-12-09T20:30:10Z", "message": "Reconcile completed successfully", "observedGeneration": 3, "reason": "ReconcileCompleted", "status": "True", "type": "Available" }, { "lastTransitionTime": "2022-12-09T20:30:10Z", "message": "Reconcile completed successfully", "observedGeneration": 3, "reason": "ReconcileCompleted", "status": "False", "type": "Progressing" }, { "lastTransitionTime": "2022-12-09T16:39:11Z", "message": "Reconcile completed successfully", "observedGeneration": 3, "reason": "ReconcileCompleted", "status": "False", "type": "Degraded" }, { "lastTransitionTime": "2022-12-09T20:30:10Z", "message": "Reconcile completed successfully", "observedGeneration": 3, "reason": "ReconcileCompleted", "status": "True", "type": "Upgradeable" 1 } ] 1 The OpenShift Virtualization Operator has the Upgradeable status. Manually update your cluster from the source EUS version to the minor version of OpenShift Container Platform: USD oc adm upgrade Verification Check the current version by running the following command: USD oc get clusterversion Note Updating OpenShift Container Platform to the version is a prerequisite for updating OpenShift Virtualization. For more details, refer to the "Updating clusters" section of the OpenShift Container Platform documentation. Update OpenShift Virtualization. With the default Automatic approval strategy, OpenShift Virtualization automatically updates to the corresponding version after you update OpenShift Container Platform. If you use the Manual approval strategy, approve the pending updates by using the web console. Monitor the OpenShift Virtualization update by running the following command: USD oc get csv -n openshift-cnv Update OpenShift Virtualization to every z-stream version that is available for the non-EUS minor version, monitoring each update by running the command shown in the step. Confirm that OpenShift Virtualization successfully updated to the latest z-stream release of the non-EUS version by running the following command: USD oc get hyperconverged kubevirt-hyperconverged -n openshift-cnv -o json | jq ".status.versions" Example output [ { "name": "operator", "version": "4.15.9" } ] Wait until the HyperConverged Operator has the Upgradeable status before you perform the update. Enter the following command and monitor the output: USD oc get hyperconverged kubevirt-hyperconverged -n openshift-cnv -o json | jq ".status.conditions" Update OpenShift Container Platform to the target EUS version. Confirm that the update succeeded by checking the cluster version: USD oc get clusterversion Update OpenShift Virtualization to the target EUS version. With the default Automatic approval strategy, OpenShift Virtualization automatically updates to the corresponding version after you update OpenShift Container Platform. If you use the Manual approval strategy, approve the pending updates by using the web console. Monitor the OpenShift Virtualization update by running the following command: USD oc get csv -n openshift-cnv The update completes when the VERSION field matches the target EUS version and the PHASE field reads Succeeded . Restore the workloadUpdateMethods configuration that you recorded from step 1 with the following command: USD oc patch hyperconverged kubevirt-hyperconverged -n openshift-cnv --type json -p \ "[{\"op\":\"add\",\"path\":\"/spec/workloadUpdateStrategy/workloadUpdateMethods\", \"value\":{WorkloadUpdateMethodConfig}}]" Example output hyperconverged.hco.kubevirt.io/kubevirt-hyperconverged patched Verification Check the status of VM migration by running the following command: USD oc get vmim -A steps You can now unpause the worker nodes' machine config pools. 6.1.5. Advanced options The stable release channel and the Automatic approval strategy are recommended for most OpenShift Virtualization installations. Use other settings only if you understand the risks. 6.1.5.1. Changing update settings You can change the update channel and approval strategy for your OpenShift Virtualization Operator subscription by using the web console. Prerequisites You have installed the OpenShift Virtualization Operator. You have administrator permissions. Procedure Click Operators Installed Operators . Select OpenShift Virtualization from the list. Click the Subscription tab. In the Subscription details section, click the setting that you want to change. For example, to change the approval strategy from Manual to Automatic , click Manual . In the window that opens, select the new update channel or approval strategy. Click Save . 6.1.5.2. Manual approval strategy If you use the Manual approval strategy, you must manually approve every pending update. If OpenShift Container Platform and OpenShift Virtualization updates are out of sync, your cluster becomes unsupported. To avoid risking the supportability and functionality of your cluster, use the Automatic approval strategy. If you must use the Manual approval strategy, maintain a supportable cluster by approving pending Operator updates as soon as they become available. 6.1.5.3. Manually approving a pending Operator update If an installed Operator has the approval strategy in its subscription set to Manual , when new updates are released in its current update channel, the update must be manually approved before installation can begin. Prerequisites An Operator previously installed using Operator Lifecycle Manager (OLM). Procedure In the Administrator perspective of the OpenShift Container Platform web console, navigate to Operators Installed Operators . Operators that have a pending update display a status with Upgrade available . Click the name of the Operator you want to update. Click the Subscription tab. Any updates requiring approval are displayed to Upgrade status . For example, it might display 1 requires approval . Click 1 requires approval , then click Preview Install Plan . Review the resources that are listed as available for update. When satisfied, click Approve . Navigate back to the Operators Installed Operators page to monitor the progress of the update. When complete, the status changes to Succeeded and Up to date . 6.1.6. Additional resources Performing a Control Plane Only update What are Operators? Operator Lifecycle Manager concepts and resources Cluster service versions (CSVs) About live migration Configuring eviction strategies Configuring live migration limits and timeouts
oc get csv -n openshift-cnv VERSION REPLACES PHASE 4.9.0 kubevirt-hyperconverged-operator.v4.8.2 Installing 4.9.0 kubevirt-hyperconverged-operator.v4.9.0 Replacing oc get hyperconverged kubevirt-hyperconverged -n openshift-cnv -o=jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.message}{"\n"}{end}' ReconcileComplete True Reconcile completed successfully Available True Reconcile completed successfully Progressing False Reconcile completed successfully Degraded False Reconcile completed successfully Upgradeable True Reconcile completed successfully oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv apiVersion: hco.kubevirt.io/v1beta1 kind: HyperConverged metadata: name: kubevirt-hyperconverged spec: workloadUpdateStrategy: workloadUpdateMethods: 1 - LiveMigrate 2 - Evict 3 batchEvictionSize: 10 4 batchEvictionInterval: "1m0s" 5 oc get vmi -l kubevirt.io/outdatedLauncherImage --all-namespaces oc get kv kubevirt-kubevirt-hyperconverged -n openshift-cnv -o jsonpath='{.spec.workloadUpdateStrategy.workloadUpdateMethods}' oc patch hyperconverged kubevirt-hyperconverged -n openshift-cnv --type json -p '[{"op":"replace","path":"/spec/workloadUpdateStrategy/workloadUpdateMethods", "value":[]}]' hyperconverged.hco.kubevirt.io/kubevirt-hyperconverged patched oc get hyperconverged kubevirt-hyperconverged -n openshift-cnv -o json | jq ".status.conditions" [ { "lastTransitionTime": "2022-12-09T16:29:11Z", "message": "Reconcile completed successfully", "observedGeneration": 3, "reason": "ReconcileCompleted", "status": "True", "type": "ReconcileComplete" }, { "lastTransitionTime": "2022-12-09T20:30:10Z", "message": "Reconcile completed successfully", "observedGeneration": 3, "reason": "ReconcileCompleted", "status": "True", "type": "Available" }, { "lastTransitionTime": "2022-12-09T20:30:10Z", "message": "Reconcile completed successfully", "observedGeneration": 3, "reason": "ReconcileCompleted", "status": "False", "type": "Progressing" }, { "lastTransitionTime": "2022-12-09T16:39:11Z", "message": "Reconcile completed successfully", "observedGeneration": 3, "reason": "ReconcileCompleted", "status": "False", "type": "Degraded" }, { "lastTransitionTime": "2022-12-09T20:30:10Z", "message": "Reconcile completed successfully", "observedGeneration": 3, "reason": "ReconcileCompleted", "status": "True", "type": "Upgradeable" 1 } ] oc adm upgrade oc get clusterversion oc get csv -n openshift-cnv oc get hyperconverged kubevirt-hyperconverged -n openshift-cnv -o json | jq ".status.versions" [ { "name": "operator", "version": "4.15.9" } ] oc get hyperconverged kubevirt-hyperconverged -n openshift-cnv -o json | jq ".status.conditions" oc get clusterversion oc get csv -n openshift-cnv oc patch hyperconverged kubevirt-hyperconverged -n openshift-cnv --type json -p "[{\"op\":\"add\",\"path\":\"/spec/workloadUpdateStrategy/workloadUpdateMethods\", \"value\":{WorkloadUpdateMethodConfig}}]" hyperconverged.hco.kubevirt.io/kubevirt-hyperconverged patched oc get vmim -A