Chapter 10. Troubleshooting
Chapter 10. Troubleshooting You can refer to the following tips to troubleshoot upgrading from RHEL 7 to RHEL 8. 10.1. Troubleshooting resources You can refer to the following troubleshooting resources. Console output By default, only error and critical log level messages are printed to the console output by the Leapp utility. To change the log level, use the --verbose or --debug options with the leapp upgrade command. In verbose mode, Leapp prints info, warning, error, and critical messages. In debug mode, Leapp prints debug, info, warning, error, and critical messages. Logs The /var/log/leapp/leapp-upgrade.log file lists issues found during the initramfs phase. The /var/log/leapp/dnf-debugdata/ directory contains transaction debug data. This directory is present only if the leapp upgrade command is executed with the --debug option. The /var/log/leapp/answerfile contains questions required to be answered by Leapp . The journalctl utility provides complete logs. Reports The /var/log/leapp/leapp-report.txt file lists issues found during the pre-upgrade phase. The report is also available in the web console, see Assessing upgradability and applying automated remediations through the web console . The /var/log/leapp/leapp-report.json file lists issues found during the pre-upgrade phase in a machine-readable format, which enables you to process the report using custom scripts. For more information, see Automating your Red Hat Enterprise Linux pre-upgrade report workflow . 10.2. Troubleshooting tips You can refer to the following troubleshooting tips. Pre-upgrade phase Verify that your system meets all conditions listed in Planning an upgrade . Make sure you have followed all steps described in Preparing for the upgrade for example, your system does not use more than one Network Interface Card (NIC) with a name based on the prefix used by the kernel ( eth ). Make sure you have answered all questions required by Leapp in the /var/log/leapp/answerfile file. If any answers are missing, Leapp inhibits the upgrade. Example questions: Disable pam_pkcs11 module in PAM configuration? Disable pam_krb5 module in PAM configuration? Configure PAM and nsswitch.conf with the following authselect call? Make sure you have resolved all problems identified in the pre-upgrade report, located at /var/log/leapp/leapp-report.txt . To achieve this, you can also use the web console, as described in Assessing upgradability and applying automated remediations through the web console . Example 10.1. Leapp answerfile The following is an example of an unedited /var/log/leapp/answerfile file that has one unanswered question: The Label field specifies the question that requires an answer. In this example, the question is Disable pam_pkcs11 module in PAM configuration? To answer the question, uncomment the confirm line and enter an answer of True or False . In this example, the selected answer is True : Download phase If a problem occurs during downloading RPM packages, examine transaction debug data located in the /var/log/leapp/dnf-debugdata/ directory. Note The /var/log/leapp/dnf-debugdata/ directory is empty or does not exist if no transaction debug data was produced. This might occur when the required repositories are not available. initramfs phase During this phase, potential failures redirect you to the Dracut shell. Check the Journal log: Alternatively, restart the system from the Dracut shell using the reboot command and check the /var/log/leapp/leapp-upgrade.log file. Post-upgrade phase If your system seems to be successfully upgraded but booted with the old RHEL 7 kernel, restart the system and check the kernel version of the default entry in GRUB. Make sure you have followed the recommended steps in Verifying the post-upgrade state of the RHEL 8 system . If your application or a service stops working or behaves incorrectly after you have switched SELinux to enforcing mode, search for denials using the ausearch , journalctl , or dmesg utilities: The most common problems are caused by incorrect labeling. See Troubleshooting problems related to SELinux for more details. 10.3. Known issues The following are known issues you might encounter when upgrading from RHEL 7 to RHEL 8. NIC teaming currently does not work when the in-place upgrade is performed while Network Manager is disabled or not installed. If your RHEL 7 system uses a device driver that is provided by Red Hat but is not available in RHEL 8, Leapp inhibits the upgrade. However, if the RHEL 7 system uses a third-party device driver that Leapp does not have data for in the /etc/leapp/files/device_driver_deprecation_data.json file, Leapp does not detect such a driver and proceeds with the upgrade. Consequently, the system might fail to boot after the upgrade. You cannot perform an in-place upgrade when the winbind and wins Samba modules are used in the /etc/nsswitch.conf file. The upgrade transaction fails with the following error messages and Leapp inhibits the upgrade: To work around this problem, configure the system so that it uses only local providers for the user , groups , and hosts database during the update: Open the system /etc/nsswitch.conf configuration file and search for entries that contain the winbind or wins strings. If you find such entries, create a backup of /etc/nsswitch.conf . Edit /etc/nsswitch.conf and remove winbind or wins from the entries that contain them. Perform an in-place upgrade. After the upgrade, add the winbind and wins strings to their entries in /etc/nsswitch.conf , based on your system configuration requirements. (BZ#1410154) The Leapp utility does not change customized authentication configuration during the upgrade process. If you used the deprecated authconfig utility to configure authentication on your RHEL 7 system, authentication on RHEL 8 might not work correctly. To ensure that your custom configuration functions properly on the RHEL 8 system, re-configure your RHEL 8 system with the authselect utility. Important During the in-place upgrade, the deprecated pam_krb5 or pam_pkcs11 pluggable authentication modules (PAM) are removed. As a result, if the PAM configuration on your RHEL 7 system contains the pam_krb5 or pam_pkcs11 modules and if these modules have the required or requisite control values, you might be locked out of the system if you perform the in-place upgrade. To work around this problem, reconfigure your RHEL 7 system to not use pam_krb5 or pam_pkcs11 before you start the upgrade process. If the name of a third-party package (not signed by Red Hat) installed on your system is the same as the name of a package provided by Red Hat, the in-place upgrade fails. To work around this problem, select one of the following options prior to upgrading: Remove the third-party package Replace the third-party package with the package provided by Red Hat For security reasons, support for single-DES (DES) and triple-DES (3DES) encryption types has been removed from RHEL 8. RHEL 7 Identity Management (IdM), however, still supports 3DES encryption. Upgrading IdM clients or migrating the whole IdM environment from RHEL 7 to RHEL 8 is possible because both versions of RHEL prefer stronger AES encryption types by default: Version of IdM Default encryption types Additional supported encryption types RHEL 7 aes256-cts aes128-cts camellia256-cts camellia128-cts des3-hmac arcfour-hmac RHEL 8 aes256-cts aes128-cts aes256-sha2 aes128-sha2 camellia256-cts camellia128-cts arcfour-hmac [a] [a] RC4 encryption has been deprecated and disabled by default in RHEL 8 because it is considered less secure than the newer AES-128 and AES-256 encryption types. For more information about enabling RC4 support for compatibility with legacy Active Directory environments, see Ensuring support for common encryption types in AD and RHEL . If you manually configured a non-IdM Kerberos Distribution Center (KDC), any services, or any users to only use DES or 3DES encryption, you might experience service interruptions after updating to the latest Kerberos packages in RHEL 8, such as: Kerberos authentication errors unknown enctype encryption errors KDCs with DES-encrypted Database Master Keys ( K/M ) fail to start Red Hat recommends you do not use DES or 3DES encryption in your environment. For more information about re-keying Kerberos principals to use stronger encryption types, see Retiring DES from MIT Kerberos Documentation. The in-place upgrade might fail on systems with Software Redundant Array of Independent Disks (RAID). ( RHEL-3279 ) Systems with a disabled GRUB boot loader specification, such as systems using Puppet, cannot create new initramfs for newer kernels. To work around this problem, manually remove packages and the old kernel from the boot loader entry as described in Chapter 6: Performing post-upgrade tasks . (BZ# 1955099 ) The Relax-and-Recover (ReaR) utility is not available on the IBM Z architecture. As a result, IBM Z systems cannot be completely remediated by the OpenSCAP suite and might not be fully compliant with security baselines. (BZ# 1958939 ) During the in-place upgrade, the Leapp utility usually preserves the network interface controller (NIC) names between RHEL 7 and RHEL 8. However, on some systems, such as systems with network bonding, the NIC names require updating between RHEL 7 and RHEL 8. On those systems, perform the following steps: Set the LEAPP_NO_NETWORK_RENAMING=1 environment variable to prevent the Leapp utility from incorrectly preserving the original RHEL 7 NIC names. Perform the in-place upgrade. Verify that your network is working correctly. If needed, manually update the network configuration. (BZ# 1919382 ) After the in-place upgrade, SSH keys are no longer auto-generated if the system meets the following conditions: The system is on a cloud. The cloud-init package is installed. The ssh_genkeytypes configuration is set to ~ in the /etc/cloud/cloud.cfg file, which is the default. This issue prevents the system from connecting by using SSH if the original keys have been removed. For more information about preventing this issue, see the Red Hat Knowledgebase solution, Unable to SSH to new Virtual Machine after upgrading the template to RHEL 8.7 or 9 . (BZ# 2210012 ) VMWare virtual machines that were created at Hardware Level 13 and are booting with UEFI might experience issues during the upgrade because the NVRAM file is too small. For more information, see the Red Hat Knowledgebase solution VMWare: Getting "No space left on device" when executing efibootmgr or mokutil command to add entries . ( RHEL-3362 ) The upgrade might fail if you are upgrading by using RHUI with an ISO image. You can work around this issue by not using the --iso option with the upgrade or see the Red Hat Knowledgebase solution Offline Leapp upgrade using ISO fails with "Failed to synchronize cache for repo 'rhul-microsoft-azure-rhel8', ignoring this repo . ( RHEL-3296 ) The pre-upgrade process might fail with the following error messages: MountError: failed to create mount target directory ... If this issue occurs, export the LEAPP_OVL_IMG_FS_EXT4=1 environment variable. For more information, see the Red Hat Knowledgebase solution Leapp can fail with a MountError (OverlayFS + XFS ftype=1) . ( RHEL-3330 ) If too many file systems are mounted, the pre-upgrade process might fail with the following error message: If this issue occurs, complete the following steps: Unmount any file systems that are not related to the system partitions and are not needed during the upgrade process. Comment out the unmounted file systems' entries in the /etc/fstab file to prevent them from being mounted during the upgrade process. Restore the original file system configuration after the upgrade. ( RHEL-3320 ) If you do not have the /etc/sysconfig/kernel system configuration file on your system, the upgrade fails, resulting in a broken system. To prevent this issue, create the file manually with the expected configuration. For more information, see Verifying the boot loader . ( RHEL-22306 ) If any of the mounted file systems that are defined in the /etc/fstab file do not have the shared propagation flag set, the upgrade might fail. To prevent this issue, remount these file systems to set them as shared: Replace mountpoint with the mountpoint of each file system. For more information, see the Red Hat Knowledgebase solution Leapp "Can not load RPM file" during the DNF transaction check . ( RHEL-23449 ) The upgrade might fail if limited resources have been set for the upgrade process. For example, the maximum number of open files descriptors and maximum size of files written by the process and its children attributes can be reached by the upgrade process if they are set. To prevent these issues, increase or remove these limits before the upgrade process. For more information, see Red Hat Knowledgebase solutions Why does leapp preupgrade fail with sqlite3.OperationalError: unable to open database file traceback error ? and Ensure that there is enough diskspace in /var/lib/leapp/scratch/diskimages/root_boot at least XXX mib are needed . ( RHEL-16881 , RHEL-26459 ) If the jbcs-httpd24-brotli RPM is installed on the system, for example, if your system uses JBoss Core Services Web Server, the upgrade might fail. To prevent this issue, remove the package before the upgrade. Alternatively, if you cannot remove the package before the upgrade, run the following command to ensure that the brotli package is installed during the upgrade: ( RHEL-22542 ) Systems with network cards that are not on a PCI bus, such as RoCE on IBM Z machines, might experience connectivity loss after the upgrade due to changed NIC names. To prevent this problem, complete the following steps before upgrading: Add net.ifnames=0 to the kernel boot parameters: On IBM Z systems, enter the following command to make the change effective: Reboot your system. ( RHEL-22371 ) If your RHEL 7 system uses a device driver that is provided by Red Hat but is not available in RHEL 8, the driver must be unloaded before the in-place upgrade. However, when some device drivers, for example the mptcl driver, are used by other services, the drivers might be reloaded automatically. For more information to resolve this problem, see the Red Hat Knowledgebase solution IPU is inhibited due to mptctl module that cannot be unloaded . ( RHEL-15894 ) If you are upgrading by using RHUI, files in the /usr/share/leapp-repository/repositories/system_upgrade/common/files/rhui/ directory are incorrectly reported as custom files in the pre-upgrade report. Unless you modified these files manually, you can ignore the warnings about these files in the report and the in-place upgrade will not be affected. ( RHEL-40115 ) The Active Directory authentication might be broken after the upgrade. For more information, see the Red Hat Knowledgebase solution Post LEAPP upgrade from RHEL 7 to RHEL 8, Active Directory authentication is not working . The logrotate configuration on RHEL 8 differs from RHEL 7. As a result, if you customized the logrotate configuration, logrotate might fail after the upgrade. For more information to fix this issue, see the Red Hat Knowledgebase solution logrotate exited abnormally after inplace upgrade from RHEL 7 to RHEL 8 . The upgrade might fail if any file systems without storage are defined in the /etc/fstab file with a storage. Such setups were typical for legacy systems that use SysVinit , but they are invalid for systems with systemd . To prevent this problem, remove these entries from the /etc/stab file. The following mount points are generally used for file systems without storage: /dev /dev/pts /dev/shm /proc /run /sys /sys/firmware/efi/efivars /sys/fs/bpf /sys/fs/cgroup /sys/fs/pstore /sys/fs/smackfs /sys/kernel/security ( RHEL-56838 ) If you use an HTTP proxy, Red Hat Subscription Manager must be configured to use such a proxy, or the subscription-manager command must be executed with the --proxy <hostname> option. Otherwise, an execution of the subscription-manager command fails. If you use the --proxy option instead of the configuration change, the upgrade process fails because Leapp is unable to detect the proxy. To prevent this problem from occurring, manually edit the rhsm.conf file. For more information, see the Red Hat Knowledgebase solution How to configure HTTP Proxy for Red Hat Subscription Management . (BZ# 1689294 ) For systems that require a proxy to access RHEL 8 content, you usually need to configure the use of the proxy by DNF in the /etc/dnf/dnf.conf configuration file. If the current DNF configuration is not compatible with the DNF version on the target system, specify the valid target configuration in the /etc/leapp/files/dnf.conf configuration file. For more information, see the Red Hat Knowledgebase solution How does Leapp work with a proxy? The upgrade might fail with a MountError message when all of the following conditions are met: A mount point that is defined in /etc/fstab contains underscore characters. Another mount point defined in /etc/stab has the same name if the underscore character is replaced with a forward slash. For example, having both /var/tmp and /var_tmp mount points defined in /etc/fstab causes the upgrade to fail. To prevent this problem, unmount the mount point that contains underscore characters, and comment that mount point out from the /etc/fstab file before the upgrade. You can restore the configuration after the upgrade. ( RHEL-62737 ) 10.4. Obtaining support To open a support case, select RHEL 7 as the product, and provide a sosreport from your system. To generate a sosreport on your system, run: Note that you can leave the case ID empty. For more information about generating a sosreport, see the Red Hat Knowledgebase solution What is an sosreport and how to create one in Red Hat Enterprise Linux? . For more information about opening and managing a support case on the Customer Portal, see the Red Hat Knowledgebase solution, How do I open and manage a support case on the Customer Portal? .
Title: None Reason: Confirmation =================== remove_pam_pkcs11_module_check.confirm ================== Label: Disable pam_pkcs11 module in PAM configuration? If no, the upgrade process will be interrupted. Description: PAM module pam_pkcs11 is no longer available in RHEL-8 since it was replaced by SSSD. Type: bool Default: None Available choices: True/False Unanswered question. Uncomment the following line with your answer confirm = [remove_pam_pkcs11_module_check] Available choices: True/False Unanswered question. Uncomment the following line with your answer confirm = True journalctl ausearch -m AVC,USER_AVC -ts boot journalctl -t setroubleshoot dmesg | grep -i -e selinux -e type=1400 upgrade[469]: STDERR: upgrade[469]: Error in PREIN scriptlet in rpm package unbound-libs upgrade[469]: Error: Transaction failed upgrade[469]: Container el8userspace failed with error code 1. unbound-libs has a PREIN failure OperationalError: unable to open database file Cannot create XFS filesystem in mount -o remount --make-shared <mountpoint> echo brotli >> /etc/leapp/transaction/to_install grubby --update-kernel=ALL --args 'net.ifnames=0' zipl sosreport