title: All threads in wait in core dump file, but someone triggered SIG_ABRT
body: <p>I am supporting an application written in C++ over many years and as of late it has started to crash providing core dumps that we don't know how to handle.
  It runs on an appliance on Ubuntu 14.04.5</p>
  
  <p>When loading the core file in GDB it says that:
  Program terminated with signal SIGABRT, Aborted </p>
  
  <p>I can inspect 230 threads but they are all in wait() in the exact same memory position.</p>
  
  <p>There is a thread with ID 1 that in theory could be the responsible but that thread is also in wait.</p>
  
  <p>So I have two questions basically.</p>
  
  <p>How does the id index of the threads work? 
  Is thread with GDB ID 1 the last active thread? or is that an arbitrary index and the failure can be in any of the other threads?</p>
  
  <p>How can all threads be in wait() when a SIGABRT is triggered?
  Shouldn't the instruction pointer be at the failing command when the OS decided to step in an halt the process? Or is it some sort of deadlock protection?</p>
  
  <p>Any help much appreciated.</p>
  
  <p>Backtrace of thread 1:</p>
  
  <pre><code>#0  0xf771dcd9 in ?? ()
  #1  0xf74ad4ca in _int_free (av=0x38663364, p=&lt;optimized out&gt;,have_lock=-186161432) at malloc.c:3989
  #2  0xf76b41ab in std::string::_Rep::_M_destroy(std::allocator&lt;char&gt; const&amp;) () from /usr/lib32/libstdc++.so.6
  #3  0xf764f82f in operator delete(void*) () from /usr/lib32/libstdc++.so.6
  #4  0xf764f82f in operator delete(void*) () from /usr/lib32/libstdc++.so.6
  #5  0x5685e8b4 in SlimStringMapper::~SlimStringMapper() ()
  #6  0x567d6bc3 in destroy ()
  #7  0x566a40b4 in HttpProxy::getLogonCredentials(HttpClient*, HttpServerTransaction*, std::string const&amp;, std::string const&amp;, std::string&amp;, std::string&amp;) ()
  #8  0x566a5d04 in HttpProxy::add_authorization_header(HttpClient*, HttpServerTransaction*, Hosts::Host*) ()
  #9  0x566af97c in HttpProxy::onClientRequest(HttpClient*, HttpServerTransaction*) ()
  #10 0x566d597e in callOnClientRequest(HttpClient*, HttpServerTransaction*, FastHttpRequest*) ()
  #11 0x566d169f in GateKeeper::onClientRequest(HttpClient*, HttpServerTransaction*) ()
  #12 0x566a2291 in HttpClientThread::run() ()
  #13 0x5682e37c in wa_run_thread ()
  #14 0xf76f6f72 in start_thread (arg=0xec65ab40) at pthread_create.c:312
  #15 0xf75282ae in query_module () at ../sysdeps/unix/syscall-template.S:82
  #16 0xec65ab40 in ?? ()
  </code></pre>
  
  <p>Another thread that should be in wait:</p>
  
  <pre><code>#0  0xf771dcd9 in ?? ()
  #1  0x5682e37c in wa_run_thread ()
  #2  0xf76f6f72 in start_thread (arg=0xf33bdb40) at pthread_create.c:312
  #3  0xf75282ae in query_module () at ../sysdeps/unix/syscall-template.S:82
  #4  0xf33bdb40 in ?? ()
  </code></pre>
  
  <p>Best regards
  Jon</p>