Chapter 11. Contacting Red Hat support for service
Chapter 11. Contacting Red Hat support for service If the information in this guide did not help you to solve the problem, this chapter explains how you contact the Red Hat support service. 11.1. Prerequisites Red Hat support account. 11.2. Providing information to Red Hat Support engineers If you are unable to fix problems related to Red Hat Ceph Storage, contact the Red Hat Support Service and provide sufficient amount of information that helps the support engineers to faster troubleshoot the problem you encounter. Prerequisites Root-level access to the node. Red Hat support account. Procedure Open a support ticket on the Red Hat Customer Portal . Ideally, attach an sosreport to the ticket. See the What is a sosreport and how to create one in Red Hat Enterprise Linux 4.6 and later? solution for details. If the Ceph daemons fail with a segmentation fault, consider generating a human-readable core dump file. See Generating readable core dump files for details. 11.3. Generating readable core dump files When a Ceph daemon terminates unexpectedly with a segmentation fault, gather the information about its failure and provide it to the Red Hat Support Engineers. Such information speeds up the initial investigation. Also, the Support Engineers can compare the information from the core dump files with Red Hat Ceph Storage cluster known issues. 11.3.1. Prerequisites Install the ceph-debuginfo package if it is not installed already. Enable the repository containing the ceph-debuginfo package: Red Hat Enterprise Linux 7: Replace DAEMON with osd or mon depending on the type of Ceph node. Red Hat Enterprise Linux 8: Install the ceph-debuginfo package: Ensure that the gdb package is installed and if it is not, install it: Continue with the procedure based on the type of your deployment: Section 11.3.2, "Generating readable core dump files on bare-metal deployments" Section 11.3.3, "Generating readable core dump files in containerized deployments" 11.3.2. Generating readable core dump files on bare-metal deployments Follow this procedure to generate a core dump file if you use Red Hat Ceph Storage on bare-metal. Procedure Enable generating core dump files for Ceph. Set the proper ulimits for the core dump files by adding the following parameter to the /etc/systemd/system.conf file: Comment out the PrivateTmp=true parameter in the Ceph daemon service file, by default located at /lib/systemd/system/ CLUSTER_NAME - DAEMON @.service : Set the suid_dumpable flag to 2 to allow the Ceph daemons to generate dump core files: Adjust the core dump files location: Modify /etc/systemd/coredump.conf file by adding the following lines under section [Coredump] : Reload the systemd service for the changes to take effect: Restart the Ceph daemon for the changes to take effect: Specify the daemon type ( osd or mon ) and its ID (numbers for OSDs, or short host names for Monitors) for example: Reproduce the failure, for example try to start the daemon again. Use the GNU Debugger (GDB) to generate a readable backtrace from an application core dump file: Specify the daemon type and the PID of the failed process, for example: In the GDB command prompt disable paging and enable logging to a file by entering the commands set pag off and set log on : Apply the backtrace command to all threads of the process by entering thr a a bt full : After the backtrace is generated turn off logging by entering set log off : Transfer the log file gdb.txt to the system you access the Red Hat Customer Portal from and attach it to a support ticket. 11.3.3. Generating readable core dump files in containerized deployments Follow this procedure to generate a core dump file if you use Red Hat Ceph Storage in containers. The procedure involves two scenarios of capturing the core dump file: When a Ceph process terminates unexpectedly due to the SIGILL, SIGTRAP, SIGABRT, or SIGSEGV error. or Manually, for example for debugging issues such as Ceph processes are consuming high CPU cycles, or are not responding. Prerequisites Root-level access to the container node running the Ceph containers. Installation of the appropriate debugging packages. Installation of the GNU Project Debugger ( gdb ) package. Procedure If a Ceph process terminates unexpectedly due to the SIGILL, SIGTRAP, SIGABRT, or SIGSEGV error: Set the core pattern to the systemd-coredump service on the node where the container with the failed Ceph process is running, for example: Watch for the container failure due to a Ceph process and search for a core dump file in the /var/lib/systemd/coredump/ directory, for example: To manually capture a core dump file for the Ceph Monitors and Ceph Managers : Get the ceph-mon package details of the Ceph daemon from the container: Red Hat Enterprise Linux 7: Red Hat Enterprise Linux 8: Replace NAME with the name of the Ceph container. Make a backup copy and open for editing the ceph-mon@.service file: In the ceph-mon@.service file, add these three options to the [Service] section, each on a separate line: Example Restart the Ceph Monitor daemon: Syntax Replace MONITOR_ID with the ID number of the Ceph Monitor. Example Install the gdb package inside the Ceph Monitor container: Red Hat Enterprise Linux 7: Red Hat Enterprise Linux 8: Replace MONITOR_ID with the ID number of the Ceph Monitor. Find the process ID: Syntax Replace PROCESS with the name of failed process, for example ceph-mon . Example Generate the core dump file: Syntax Replace ID with the ID of the failed process that you got from the step, for example 18110 : Example Verify that the core dump file has been generated correctly. Example Copy the core dump file outside of the Ceph Monitor container: Red Hat Enterprise Linux 7: Red Hat Enterprise Linux 8: Replace MONITOR_ID with the ID number of the Ceph Monitor and replace MONITOR_PID with the process ID number. Restore the backup copy of the ceph-mon@.service file: Restart the Ceph Monitor daemon: Syntax Replace MONITOR_ID with the ID number of the Ceph Monitor. Example Upload the core dump file for analysis by Red Hat support, see step 4. To manually capture a core dump file for Ceph OSDs : Get the ceph-osd package details of the Ceph daemon from the container: Red Hat Enterprise Linux 7: Red Hat Enterprise Linux 8: Replace NAME with the name of the Ceph container. Install the Ceph package for the same version of the ceph-osd package on the node where the Ceph containers are running: Red Hat Enterprise Linux 7: Red Hat Enterprise Linux 8: If needed, enable the appropriate repository first. See the Enabling the Red Hat Ceph Storage repositories section in the Installation Guide for details. Find the ID of the process that has failed: Replace PROCESS with the name of failed process, for example ceph-osd . Generate the core dump file: Replace ID with the ID of the failed process that you got from the step, for example 18110 : Verify that the core dump file has been generated correctly. Upload the core dump file for analysis by Red Hat support, see the step. Upload the core dump file for analysis to a Red Hat support case. See Providing information to Red Hat Support engineers for details. 11.3.4. Additional Resources The How to use gdb to generate a readable backtrace from an application core solution on the Red Hat Customer Portal The How to enable core file dumps when an application crashes or segmentation faults solution on the Red Hat Customer Portal
subscription-manager repos --enable=rhel-7-server-rhceph-4- DAEMON -debug-rpms subscription-manager repos --enable=rhceph-4-tools-for-rhel-8-x86_64-debug-rpms yum install ceph-debuginfo yum install gdb DefaultLimitCORE=infinity PrivateTmp=true sysctl fs.suid_dumpable=2 sysctl kernel.core_pattern=/tmp/core ProcessSizeMax=8G ExternalSizeMax=8G JournalSizeMax=8G systemctl daemon-reload systemctl restart ceph- DAEMON @ ID systemctl restart ceph-osd@1 gdb /usr/bin/ceph- DAEMON /tmp/core. PID gdb /usr/bin/ceph-osd /tmp/core.123456 (gdb) set pag off (gdb) set log on (gdb) thr a a bt full (gdb) set log off echo "| /usr/lib/systemd/systemd-coredump %P %u %g %s %t %e" > /proc/sys/kernel/core_pattern ls -ltr /var/lib/systemd/coredump total 8232 -rw-r-----. 1 root root 8427548 Jan 22 19:24 core.ceph-osd.167.5ede29340b6c4fe4845147f847514c12.15622.1584573794000000.xz docker exec -it NAME /bin/bash rpm -qa | grep ceph podman exec -it NAME /bin/bash rpm -qa | grep ceph cp /etc/systemd/system/ceph-mon@.service /etc/systemd/system/ceph-mon@.service.orig --pid=host --ipc=host --cap-add=SYS_PTRACE \ [Unit] Description=Ceph Monitor After=docker.service [Service] EnvironmentFile=-/etc/environment ExecStartPre=-/usr/bin/docker rm ceph-mon-%i ExecStartPre=/bin/sh -c '"USD(command -v mkdir)" -p /etc/ceph /var/lib/ceph/mon' ExecStart=/usr/bin/docker run --rm --name ceph-mon-%i --memory=924m --cpu-quota=100000 -v /var/lib/ceph:/var/lib/ceph:z -v /etc/ceph:/etc/ceph:z -v /var/run/ceph:/var/run/ceph:z -v /etc/localtime:/etc/localtime:ro --net=host --privileged=true --ipc=host \ 1 --pid=host \ 2 --cap-add=SYS_PTRACE \ 3 -e IP_VERSION=4 -e MON_IP=10.74.131.17 -e CLUSTER=ceph -e FSID=9448efca-b1a1-45a3-bf7b-b55cba696a6e -e CEPH_PUBLIC_NETWORK=10.74.131.0/24 -e CEPH_DAEMON=MON registry.access.redhat.com/rhceph/rhceph-3-rhel7:latest ExecStop=-/usr/bin/docker stop ceph-mon-%i ExecStopPost=-/bin/rm -f /var/run/ceph/ceph-mon.pd-cephcontainer-mon01.asok Restart=always RestartSec=10s TimeoutStartSec=120 TimeoutStopSec=15 [Install] WantedBy=multi-user.target systemctl restart ceph-mon@ MONITOR_ID systemctl restart ceph-mon@1 docker exec -it ceph-mon- MONITOR_ID /bin/bash sh USD yum install gdb podman exec -it ceph-mon- MONITOR_ID /bin/bash sh USD yum install gdb ps -aef | grep PROCESS | grep -v run ps -aef | grep ceph-mon | grep -v run ceph 15390 15266 0 18:54 ? 00:00:29 /usr/bin/ceph-mon --cluster ceph --setroot ceph --setgroup ceph -d -i 5 ceph 18110 17985 1 19:40 ? 00:00:08 /usr/bin/ceph-mon --cluster ceph --setroot ceph --setgroup ceph -d -i 2 gcore ID gcore 18110 warning: target file /proc/18110/cmdline contained unexpected null characters Saved corefile core.18110 ls -ltr total 709772 -rw-r--r--. 1 root root 726799544 Mar 18 19:46 core.18110 docker cp ceph-mon- MONITOR_ID :/tmp/mon.core. MONITOR_PID /tmp podman cp ceph-mon- MONITOR_ID :/tmp/mon.core. MONITOR_PID /tmp cp /etc/systemd/system/ceph-mon@.service.orig /etc/systemd/system/ceph-mon@.service systemctl restart ceph-mon@ MONITOR_ID systemctl restart ceph-mon@1 docker exec -it NAME /bin/bash rpm -qa | grep ceph podman exec -it NAME /bin/bash rpm -qa | grep ceph yum install ceph-osd dnf install ceph-osd ps -aef | grep PROCESS | grep -v run ps -aef | grep ceph-osd | grep -v run ceph 15390 15266 0 18:54 ? 00:00:29 /usr/bin/ceph-osd --cluster ceph --setroot ceph --setgroup ceph -d -i 5 ceph 18110 17985 1 19:40 ? 00:00:08 /usr/bin/ceph-osd --cluster ceph --setroot ceph --setgroup ceph -d -i 2 gcore ID gcore 18110 warning: target file /proc/18110/cmdline contained unexpected null characters Saved corefile core.18110 ls -ltr total 709772 -rw-r--r--. 1 root root 726799544 Mar 18 19:46 core.18110