title: socket close in wait call from system - heartbeat mechanism
body: <p>I have a heartbeat mechanism implemented where application 'app1' is sending pulse every 60 seconds to watchdog process 'watch1'. If watch1 doesnt get any message from app1 after 60 seconds it assumes that app1 is dead and it restarts it. But in acual it is alive. This comminication happens through TCP/IP sockets and applications running on same machine. The watch1 process listens on port say 3000 to read the packet from app1. These applications runs on multiple OSs UNix,AIX etc.</p>
  
  <p>But for particular AIX machine the 'app1's socket is getting closed and the watch1 process restarting the app1 again and again. Its multithreading application where app1 creates one thread to send message/packet to watch1.</p>
  
  <p>I put in place a script which every 60 seconds ran ‘procstack’ on the app1 PID to see what was going on. This didn’t provide any pointers I’d hoped for (like being stuck on a lock), rather it began to show some ‘weird’ stack patterns:</p>
  
  <pre><code> ---------- tid# 21692655 (pthread ID:   3342) ----------
   0xd4be4754  SocketBase::close()(0x0) + 0x34 
   0xd0503ca4  _event_wait(??, ??) + 0x2c4
   0xd0511e78  _cond_wait_local(??, ??, ??) + 0x378
   0xd05125d0  _cond_wait(??, ??, ??) + 0xb0
   0xd0512f1c  pthread_cond_timedwait(??, ??, ??) + 0x21c
   0xd4a2cbd8  TaskEvent::Wait(int)(0x32ebc03c, 0x7530) + 0xd8
   0xd4e40b70  SendIAmAliveThreadFunc(void*)(0x32ebc038) + 0xf50
   0xd3dfcdc4  ThreadStartRoutine(void*)(0x32ebbba8) + 0xc4
   0xd04f0c4c  _pthread_body(??) + 0xec 
  </code></pre>
  
  <p>It makes no sense that the kernel service _event_wait() would be calling our code(wrapper class for socket handling) SocketBase::close(). Might be some weird anomaly; might need some AIX expertise to help diagnose.</p>
  
  <p>Is any one aware of this issue or guide on this?</p>
  
  <p><em><strong></em>**<em>*</em>**<em>*</em>*</strong> EDIT1 - 23-11-12 <strong><em>*</em>***</strong></p>
  
  <p>The app1 process is getting terminated with signal 15.</p>
  
  <p>Logs:
  Received signal 15 on thread 515
  Thread 515 does not handle signal 15
  Abnormal termination due to signal 15kill: 7143518: no such process
  Start command failed.
  kill: 7405740: no such process</p>
  
  <p>warning: The core file is not a fullcore. Some info may
  not be available.</p>