Chapter 8. Troubleshooting the Bare Metal Provisioning service
Chapter 8. Troubleshooting the Bare Metal Provisioning service Diagnose issues in an environment that includes the Bare Metal Provisioning service (ironic). 8.1. PXE boot errors Use the following troubleshooting procedures to assess and remedy issues you might encounter with PXE boot. Permission Denied errors If the console of your bare metal node returns a Permission Denied error, ensure that you have applied the appropriate SELinux context to the /httpboot and /tftpboot directories: Boot process freezes at /pxelinux.cfg/XX-XX-XX-XX-XX-XX On the console of your node, if it looks like you receive an IP address but then the process stops, you might be using the wrong PXE boot template in your ironic.conf file. The default template is pxe_config.template , so it is easy to omit the i and inadvertently enter ipxe_config.template instead. 8.2. Login errors after the bare metal node boots Failure to log in to the node when you use the root password that you set during configuration indicates that you are not booted into the deployed image. You might be logged in to the deploy-kernel/deploy-ramdisk image and the system has not yet loaded the correct image. To fix this issue, verify that the PXE Boot Configuration file in the /httpboot/pxelinux.cfg/MAC_ADDRESS on the Compute or Bare Metal Provisioning service node and ensure that all the IP addresses listed in this file correspond to IP addresses on the Bare Metal network. Note The only network that the Bare Metal Provisioning service node uses is the Bare Metal network. If one of the endpoints is not on the network, the endpoint cannot reach the Bare Metal Provisioning service node as a part of the boot process. For example, the kernel line in your file is as follows: Value in the above example kernel line Corresponding information http://192.168.200.2:8088 Parameter http_url in /etc/ironic/ironic.conf file. This IP address must be on the Bare Metal network. 5a6cdbe3-2c90-4a90-b3c6-85b449b30512 UUID of the baremetal node in ironic node-list . deploy_kernel This is the deploy kernel image in the Image service that is copied down as /httpboot/<NODE_UUID>/deploy_kernel . http://192.168.200.2:6385 Parameter api_url in /etc/ironic/ironic.conf file. This IP address must be on the Bare Metal network. ipmi The IPMI Driver in use by the Bare Metal Provisioning service for this node. deploy_ramdisk This is the deploy ramdisk image in the Image service that is copied down as /httpboot/<NODE_UUID>/deploy_ramdisk . If a value does not correspond between the /httpboot/pxelinux.cfg/MAC_ADDRESS and the ironic.conf file: Update the value in the ironic.conf file Restart the Bare Metal Provisioning service Re-deploy the Bare Metal instance 8.3. Boot-to-disk errors on deployed nodes With certain hardware, you might experience a problem with deployed nodes where the nodes cannot boot from disk during successive boot operations as part of a deployment. This usually happens because the BMC does not honor the persistent boot settings that director requests on the nodes. Instead, the nodes boot from a PXE target. In this case, you must update the boot order in the BIOS of the nodes. Set the HDD to be the first boot device, and then PXE as a later option, so that the nodes boot from disk by default, but can boot from the network during introspection or deployment as necessary. Note This error mostly applies to nodes that use LegacyBIOS firmware. 8.4. The Bare Metal Provisioning service does not receive the correct host name If the Bare Metal Provisioning service does not receive the right host name, it means that cloud-init is failing. To fix this, connect the Bare Metal subnet to a router in the OpenStack Networking service. This configuration routes requests to the meta-data agent correctly. 8.5. Invalid OpenStack Identity service credentials when executing Bare Metal Provisioning service commands If you cannot authenticate to the Identity service, check the identity_uri parameter in the ironic.conf file and ensure that you remove the /v2.0 from the keystone AdminURL. For example, set the identity_uri to http://IP:PORT . 8.6. Hardware enrolment Incorrect node registration details can cause issues with enrolled hardware. Ensure that you enter property names and values correctly. When you input property names incorrectly, the system adds the properties to the node details but ignores them. Use the openstack baremetal node set command to update node details. For example, update the amount of memory that the node is registered to use to 2 GB: 8.7. Troubleshooting iDRAC issues Redfish management interface fails to set boot device When you use the idrac-redfish management interface with certain iDRAC firmware versions and attempt to set the boot device on a bare metal server with UEFI boot, iDRAC returns the following error: If you encounter this issue, set the force_persistent_boot_device parameter in the driver-info on the node to Never : Timeout when powering off Some servers can be too slow when powering off, and time out. The default retry count is 6 , which results in a 30 second timeout. To increase the timeout duration to 90 seconds, set the ironic::agent::rpc_response_timeout value to 18 in the undercloud hieradata overrides file and re-run the openstack undercloud install command: Vendor passthrough timeout When iDRAC is not available to execute vendor passthrough commands, these commands take too long and time out: To increase the timeout duration for messaging, increase the value of the ironic::default::rpc_response_timeout parameter in the undercloud hieradata overrides file and re-run the openstack undercloud install command: 8.8. Configuring the server console Console output from overcloud nodes is not always sent to the server console. If you want to view this output in the server console, you must configure the overcloud to use the correct console for your hardware. Use one of the following methods to perform this configuration: Modify the KernelArgs heat parameter for each overcloud role. Customize the overcloud-full.qcow2 image that director uses to provision the overcloud nodes. Prerequisites A successful undercloud installation. For more information, see the Director Installation and Usage guide. Overcloud nodes ready for deployment. Modifying KernelArgs with heat during deployment Log in to the undercloud host as the stack user. Source the stackrc credentials file: Create an environment file overcloud-console.yaml with the following content: Replace <role> with the name of the overcloud role that you want to configure, and replace <console-name> with the ID of the console that you want to use. For example, use the following snippet to configure all overcloud nodes in the default roles to use tty0 : Include the overcloud-console-tty0.yaml file in your deployment command with the -e option. Modifying the overcloud-full.qcow2 image Log in to the undercloud host as the stack user. Source the stackrc credentials file: Modify the kernel arguments in the overcloud-full.qcow2 image to set the correct console for your hardware. For example, set the console to tty0 : Import the image into director: Deploy the overcloud. Verification Log in to an overcloud node from the undercloud: Replace <IP-address> with the IP address of an overcloud node. Inspect the contents of the /proc/cmdline file and ensure that console= parameter is set to the value of the console that you want to use:
semanage fcontext -a -t httpd_sys_content_t "/httpboot(/.*)?" restorecon -r -v /httpboot semanage fcontext -a -t tftpdir_t "/tftpboot(/.*)?" restorecon -r -v /tftpboot grep ^pxe_config_template ironic.conf pxe_config_template=USDpybasedir/drivers/modules/ipxe_config.template kernel http://192.168.200.2:8088/5a6cdbe3-2c90-4a90-b3c6-85b449b30512/deploy_kernel selinux=0 disk=cciss/c0d0,sda,hda,vda iscsi_target_iqn=iqn.2008-10.org.openstack:5a6cdbe3-2c90-4a90-b3c6-85b449b30512 deployment_id= 5a6cdbe3-2c90-4a90-b3c6-85b449b30512 deployment_key=VWDYDVVEFCQJNOSTO9R67HKUXUGP77CK ironic_api_url= http://192.168.200.2:6385 troubleshoot=0 text nofb nomodeset vga=normal boot_option=netboot ip=USD{ip}:USD{next-server}:USD{gateway}:USD{netmask} BOOTIF=USD{mac} ipa-api-url= http://192.168.200.2:6385 ipa-driver-name= ipmi boot_mode=bios initrd= deploy_ramdisk coreos.configdrive=0 || goto deploy openstack baremetal node set --property memory_mb=2048 NODE_UUID Unable to Process the request because the value entered for the parameter Continuous is not supported by the implementation. openstack baremetal node set --driver-info force_persistent_boot_device=Never USD{node_uuid} ironic::agent::rpc_response_timeout: 18 openstack baremetal node passthru call --http-method GET aed58dca-1b25-409a-a32f-3a817d59e1e0 list_unfinished_jobs Timed out waiting for a reply to message ID 547ce7995342418c99ef1ea4a0054572 (HTTP 500) ironic::default::rpc_response_timeout: 600 source stackrc parameter_defaults: <role>Parameters: KernelArgs: "console=<console-name>" parameter_defaults: ControllerParameters: KernelArgs: "console=tty0" ComputeParameters: KernelArgs: "console=tty0" BlockStorageParameters: KernelArgs: "console=tty0" ObjectStorageParameters: KernelArgs: "console=tty0" CephStorageParameters: KernelArgs: "console=tty0" source stackrc virt-customize --selinux-relabel -a overcloud-full.qcow2 --run-command 'grubby --update-kernel=ALL --args="console=tty0"' openstack overcloud image upload --image-path /home/stack/images/overcloud-full.qcow2 ssh heat-admin@<IP-address> [heat-admin@controller-0 ~]USD cat /proc/cmdline BOOT_IMAGE=(hd0,msdos2)/boot/vmlinuz-4.18.0-193.29.1.el8_2.x86_64 root=UUID=0ec3dea5-f293-4729-b676-5d38a611ce81 ro console=tty0 console=ttyS0,115200n81 no_timer_check crashkernel=auto rhgb quiet