title: How to debug with GDB on RT Linux: fails with SIGTRAP
body: <p>I am trying to debug a multi threaded application on RT Linux. On regular Linux, the app runs fine, and GDB also runs fine. On RT Linux the app runs fine, but  under GDB, the app runs for several seconds then terminates and prints:</p>
  
  <p><code>Program terminated with signal SIGTRAP, Trace/breakpoint trap. The program no longer exists.</code></p>
  
  <p>I cannot do a backtrace, or determine what is causing the problem. I suspect it might be some libraries used by gdb, or else maybe memory corruption in the app.</p>
  
  <p>I create over sixty threads, and many more are created by various watchdogs and timers. 
  What I have tried so far:</p>
  
  <ol>
  <li><p>Checking for mismatch between libpthread.so.0 and libthread_db.so.1.  I used </p>
  
  <p><code>objdump -s --section .comment /usr/lib64/libthread_db-1.0.so</code></p></li>
  </ol>
  
  <p>on both libraries, and they both gave the same version of gcc, which is the same gcc I am using to build the app </p>
  
  <p><code>gcc --version gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-36)</code></p>
  
  <ol start="2">
  <li><p>Trying to set a catchpoint for SIGTRAP in gdb with</p>
  
  <p><code>catch signal SIGTRAP 
  commands
  p $_siginfo.si_code 
  end</code></p></li>
  </ol>
  
  <p>This did not alter behaviour of gdb at all.
  Any ideas? New kernel libraries or sources that I should download?</p>
  
  <p><strong>Versions:</strong>
  My original linux was Scientific Linux 7 downloaded from the CERN repo (based on CentOS 7). I also downloaded and installed the prebuilt RT kernel from there.</p>
  
  <p><code># gdb --version
  GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-114.el7</code></p>
  
  <p><code># uname -r
  3.10.0-957.10.1.rt56.921.el7.x86_64</code></p>
  
  <p><code>gcc --version gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-36)</code></p>
  
  <p><strong>Some Progress</strong>
  Several times gdb itself crashed and left a gdb core dump. Backtracing into gdb using the core file, I found the same call stack -- last few functions shown below:</p>
  
  <pre><code>(gdb) bt
  #0  0x00007fc62ca9e207 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
  #1  0x00007fc62ca9f8f8 in __GI_abort () at abort.c:90
  #2  0x000000000069f5e6 in dump_core ()
  #3  0x00000000006a1de5 in internal_vproblem ()
  #4  0x00000000006a1e59 in internal_verror ()
  #5  0x00000000006a1eff in internal_error ()
  #6  0x00000000004d5149 in check_ptrace_stopped_lwp_gone ()
  #7  0x00000000004d51e2 in linux_resume_one_lwp ()
  #8  0x00000000004d6e44 in linux_handle_extended_wait ()
  #9  0x00000000004d9cf9 in linux_nat_wait ()
  #10 0x00000000004e1273 in thread_db_wait ()
  #11 0x0000000000607602 in target_wait ()
  #12 0x00000000005cf815 in wait_for_inferior ()
  </code></pre>
  
  <p>This seems to indicate a  problem with gdb, so I rebuilt gdb using the latest source (8.2.1) and this stopped the gdb crashes. Now GDB stops at many kernel calls (sleep, semwait etc) with a SIGSTOP, I can press continue, but this makes debugging impractical.</p>
  
  <p>If I add the following line to .gdbinit <code>handle SIGSTOP nostop noprint pass</code> then gdb does not stop at kernel calls, but now breakpoints don't work and it is very difficult to stop either gdb or the process being debugged.</p>