Chapter 12. Performing post-upgrade actions
Chapter 12. Performing post-upgrade actions After you have completed the overcloud upgrade, you must perform some post-upgrade configuration to ensure that your environment is fully supported and ready for future operations. Important If you run additional overcloud commands after the upgrade from Red Hat OpenStack Platform 16.2 to 17.1, you must consider the following: Overcloud commands that you run after the upgrade must include the YAML files that you created or updated during the upgrade process. For example, to provision overcloud nodes during a scale-up operation, use the /home/stack/tripleo-[stack]-baremetal-deploy.yaml file instead of the /home/stack/templates/overcloud-baremetal-deployed.yaml file. Include all the options that you passed to the last run of the openstack overcloud upgrade prepare command, except for the system_upgrade.yaml file and the upgrades-environment.yaml file. 12.1. Performing post-upgrade tasks on the operating system If you upgraded the operating system on your hosts to Red Hat Enterprise Linux 9.2, you must perform post-upgrade tasks such as removing any remaining Leapp packages. For more information on these tasks, see Performing post-upgrade tasks in Upgrading from RHEL 8 to RHEL 9 . 12.2. Upgrading the overcloud images You must replace your current overcloud images with new versions. The new images ensure that the director can introspect and provision your nodes using the latest version of Red Hat OpenStack Platform software. Note You must use the new version of the overcloud images if you redeploy your overcloud. For more information on installing overcloud images, see Installing the overcloud images in Installing and managing Red Hat OpenStack Platform with director . Prerequisites You have upgraded the undercloud to the latest version. Procedure Remove any existing images from the images directory on the stack user's home ( /home/stack/images ): Extract the archives: Import the images into director: The command completes the following tasks: Converts the image format from QCOW to RAW. Provides status updates about the upload of the image. 12.3. Updating CPU pinning parameters You must migrate the CPU pinning configuration from the NovaVcpuPinSet parameter to the following parameters after completing the upgrade to Red Hat OpenStack Platform 17.1: NovaComputeCpuDedicatedSet Sets the dedicated (pinned) CPUs. NovaComputeCpuSharedSet Sets the shared (unpinned) CPUs. Procedure Log in to the undercloud as the stack user. If your Compute nodes support simultaneous multithreading (SMT) but you created instances with the hw:cpu_thread_policy=isolated policy, you must perform one of the following options: Create a new flavor that does not set the hw:cpu_thread_policy thread policy and resize the instances with that flavor: Source your overcloud authentication file: Create a flavor with the default thread policy, prefer : Note When you resize an instance, you must use a new flavor. You cannot reuse the current flavor. For more information, see Resizing an instance in the Creating and managing instances guide. Convert the instances to use the new flavor: Repeat this step for all pinned instances that use the hw:cpu_thread_policy=isolated policy. Migrate instances from the Compute node and disable SMT on the Compute node: Source your overcloud authentication file: Disable the Compute node from accepting new virtual machines: Migrate all instances from the Compute node. For more information on instance migration, see Migrating virtual machine instances between Compute nodes . Reboot the Compute node and disable SMT in the BIOS of the Compute node. Boot the Compute node. Re-enable the Compute node: Source the stackrc file: Edit the environment file that contains the NovaVcpuPinSet parameter. Migrate the CPU pinning configuration from the NovaVcpuPinSet parameter to NovaComputeCpuDedicatedSet and NovaComputeCpuSharedSet : Migrate the value of NovaVcpuPinSet to NovaComputeCpuDedicatedSet for hosts that were previously used for pinned instances. Migrate the value of NovaVcpuPinSet to NovaComputeCpuSharedSet for hosts that were previously used for unpinned instances. If there is no value set for NovaVcpuPinSet, then all Compute node cores should be assigned to either NovaComputeCpuDedicatedSet or NovaComputeCpuSharedSet , depending on the type of instances you intend to host on the nodes. For example, your environment file might contain the following pinning configuration: To migrate the configuration to a pinned configuration, set the NovaComputeCpuDedicatedSet parameter and unset the NovaVcpuPinSet parameter: To migrate the configuration to an unpinned configuration, set the NovaComputeCpuSharedSet parameter and unset the NovaVcpuPinSet parameter: Important Ensure the configuration of either NovaComputeCpuDedicatedSet or NovaComputeCpuSharedSet matches the configuration defined in NovaVcpuPinSet . To change the configuration for either of these, or to configure both NovaComputeCpuDedicatedSet or NovaComputeCpuSharedSet , ensure the Compute nodes with the pinning configuration are not running any instances before updating the configuration. Save the file. Run the deployment command to update the overcloud with the new CPU pinning parameters. Additional resources Configuring CPU pinning on Compute nodes 12.4. Updating the default machine type for hosts after an upgrade to RHOSP 17 The machine type of an instance is a virtual chipset that provides certain default devices, such as a PCIe graphics card or Ethernet controller. Cloud users can specify the machine type for their instances by using an image with the hw_machine_type metadata property that they require. Cloud administrators can use the Compute parameter NovaHWMachineType to configure each Compute node architecture with a default machine type to apply to instances hosted on that architecture. If the hw_machine_type image property is not provided when launching the instance, the default machine type for the host architecture is applied to the instance. Red Hat OpenStack Platform (RHOSP) 17 is based on RHEL 9. The pc-i440fx QEMU machine type is deprecated in RHEL 9, therefore the default machine type for x86_64 instances that run on RHEL 9 has changed from pc to q35 . Based on this change in RHEL 9, the default value for machine type x86_64 has also changed from pc in RHOSP 16 to q35 in RHOSP 17. From RHOSP 16.2 and later, the Compute service records the instance machine type within the system metadata of the instance when it launches an instance. This means that it is now possible to change the NovaHWMachineType during the lifetime of a RHOSP deployment without affecting the machine type of existing instances. The Compute service records the machine type of instances that are not in a SHELVED_OFFLOADED state. Therefore, after an upgrade to RHOSP 17 you must manually record the machine type of instances that are in SHELVED_OFFLOADED state, and verify that all instances within the environment or specific cell have had a machine type recorded. After you have updated the system metadata for each instance with the machine types, you can update the NovaHWMachineType parameter to the RHOSP 17 default, q35 , without affecting the machine type of existing instances. Note From RHOSP OSP17.0 onwards, Q35 is the default machine type. The Q35 machine type uses PCIe ports. You can manage the number of PCIe port devices by configuring the heat parameter NovaLibvirtNumPciePorts . The number of devices that can attach to a PCIe port is fewer than instances running on versions. If you want to use more devices, you must use the hw_disk_bus=scsi or hw_scsi_model=virtio-scsi image property. For more information, see Metadata properties for virtual hardware . Prerequisites Upgrade all Compute nodes to RHEL 9.2. For more information about upgrading Compute nodes, see Upgrading all Compute nodes to RHEL 9.2 . Procedure Log in to the undercloud as the stack user. Source the stackrc file. Log in to a Controller node as the heat-admin user: Replace <controller_ip> with the IP address of the Controller node. Retrieve the list of instances that have no machine type set: Check the NovaHWMachineType parameter in the nova-hw-machine-type-upgrade.yaml file for the default machine type for the instance host. The default value for the NovaHWMachineType parameter in RHOSP 16.2 is as follows: x86_64=pc-i440fx-rhel7.6.0,aarch64=virt-rhel7.6.0,ppc64=pseries-rhel7.6.0,ppc64le=pseries-rhel7.6.0 Update the system metadata of each instance with the default instance machine type: Replace <instance_uuid> with the UUID of the instance. Replace <machine_type> with the machine type to record for the instance. Warning If you set the machine type to something other than the machine type of the image on which the instance was booted, the existing instance might fail to boot. Confirm that the machine type is recorded for all instances: This command returns a warning if an instance is found without a machine type. If you get this warning, repeat this procedure from step 4. Change the default value of NovaHWMachineType in a Compute environment file to x86_64=q35 and deploy the overcloud. Verification Create an instance that has the default machine type: Replace <flavor> with the name or ID of a flavor for the instance. Replace <image> with the name or ID of an image that does not set hw_machine_type . Replace <network> with the name or ID of the network to connect the instance to. Verify that the instance machine type is set to the default value: Replace <instance_uuid> with the UUID of the instance. Hard reboot an instance with a machine type of x86_64=pc-i440fx : Replace <instance_uuid> with the UUID of the instance. Verify that the instance machine type has not been changed: Replace <instance_uuid> with the UUID of the instance. 12.5. Re-enabling fencing in the overcloud Before you upgraded the overcloud, you disabled fencing in Disabling fencing in the overcloud . After you upgrade your environment, re-enable fencing to protect your data if a node fails. Procedure Log in to the undercloud host as the stack user. Source the stackrc undercloud credentials file: Log in to a Controller node and run the Pacemaker command to re-enable fencing: USD ssh tripleo-admin@<controller_ip> "sudo pcs property set stonith-enabled=true" Replace <controller_ip> with the IP address of a Controller node. You can find the IP addresses of your Controller nodes with the openstack server list command. If you use SBD fencing, reset the watchdog timer device interval to its original value before you disabled it: Replace <interval> with the original value of the watchdog timer device, for example, 10 . In the fencing.yaml environment file, set the EnableFencing parameter to true . Additional Resources Fencing Controller nodes with STONITH 12.6. Compressing Red Hat OpenStack Platform dashboard files After the Red Hat OpenStack Platform (RHOSP) upgrade, if your RHOSP dashboard (horizon) has errors that are similar to the following example, you must compress your files manually. Static file compression does not run automatically. You must repeat this procedure on every horizon container that you upgraded. Procedure Enter the shell for the horizon container: Navigate to the directory that contains the files to compress: Run the compression:
rm -rf ~/images/* cd ~/images for i in /usr/share/rhosp-director-images/overcloud-full-latest-17.1.tar /usr/share/rhosp-director-images/ironic-python-agent-latest-17.1.tar; do tar -xvf USDi; done cd ~ (undercloud) [stack@director images]USD openstack overcloud image upload --image-path /home/stack/images/ --update-existing source ~/overcloudrc (overcloud) USD openstack flavor create <flavor> (overcloud) USD openstack server resize --flavor <flavor> <server> (overcloud) USD openstack server resize confirm <server> source ~/overcloudrc (overcloud) USD openstack compute service list (overcloud) USD openstack compute service set <hostname> nova-compute --disable (overcloud) USD openstack compute service set <hostname> nova-compute --enable source ~/stackrc parameter_defaults: NovaVcpuPinSet: 1,2,3,5,6,7 parameter_defaults: NovaComputeCpuDedicatedSet: 1,2,3,5,6,7 NovaVcpuPinSet: "" parameter_defaults: NovaComputeCpuSharedSet: 1,2,3,5,6,7 NovaVcpuPinSet: "" (undercloud) USD openstack overcloud deploy --stack _STACK NAME_ --templates -e /home/stack/templates/<compute_environment_file>.yaml source ~/stackrc (undercloud)USD metalsmith list ssh heat-admin@<controller_ip> [heat-admin@<controller_ip> ~]USD sudo podman exec -i -u root nova_api nova-manage libvirt list_unset_machine_type [heat-admin@<controller_ip> ~]USD sudo podman exec -i -u root nova_api nova-manage libvirt update_machine_type <instance_uuid> <machine_type> [heat-admin@<controller_ip> ~]USD sudo podman exec -i -u root nova_api nova-status upgrade check (overcloud)USD openstack server create --flavor <flavor> --image <image> --network <network> --wait defaultMachineTypeInstance [heat-admin@<controller_ip> ~]USD sudo podman exec -i -u root nova_api nova-manage libvirt get_machine_type <instance_uuid> (overcloud)USD openstack server reboot --hard <instance_uuid> [heat-admin@<controller_ip> ~]USD sudo podman exec -i -u root nova_api nova-manage libvirt get_machine_type <instance_uuid> source ~/stackrc ssh tripleo-admin@<controller_ip> "sudo pcs property set stonith-enabled=true" pcs property set stonith-watchdog-timeout=<interval> compressor.exceptions.OfflineGenerationError: You have offline compression enabled but key "dbf52fe9eafa4b50d57c151a16962bcb02dfc37de3ae4fde450231af213e84a9" is missing from offline manifest. You may need to run "python manage.py compress". Here is the original content: podman exec -it horizon /bin/bash cd /usr/bin/ python3 manage.py compress