title: Tomcat process suddenly crushes
body: <p>Trying to diagnose some bizarre Tomcat 6 and/or JVM errors on a 64-bit linux machine:</p>
  
  <blockquote>
    <p>OS VERSION                       : RED HAT ENTERPRISE LINUX SERVER RELEASE 6.3 (SANTIAGO)<br>
    SERVER MODEL                     : VMWARE VIRTUAL PLATFORM ()</p>
  </blockquote>
  
  <p>I am running some application tests on virtual machines and facing a weird issue.
  The problem occures on certain VMs where exactly 2 process are running: Tomcat Application server process (java process) and a monitor process, at one randomal point, after some random hours of execution the following message can be found on catalina.out (server log) of the VM that fails (the problem doesnt occurs always on same machine):</p>
  
  <pre><code>org.apache.coyote.http11.Http11Protocol pause
  INFO: Pausing Coyote HTTP/1.1 on http-50100
  INFO: Stopping service Catalina
  org.apache.coyote.http11.Http11Protocol destroy
  INFO: Stopping Coyote HTTP/1.1 on http-8080
  </code></pre>
  
  <p>After this moment the Tomcat process is shutted down and execution fails.
  No picks can be seen on memory or CPU use.
  No system messages can be seen under /var/logs.</p>
  
  <ol>
  <li><p>At first stage we suspected that an external signal is killing our Tomcat process and we tried to use the <code>-Xrs</code> flag for JVM arguments of the process but issue still occurred.</p>
  
  <p>The java flag (<code>-xrs</code> ) should prevent the Java runtime environment handling exception signals such <code>SIGSEGV</code> and <code>SIGABRT</code> not help</p>
  
  <p>Before adding <code>xrs</code> flag we saw <code>INFO: Pausing Coyote HTTP/1.1 on http-50100</code> error that happened because outcome signal</p>
  
  <p>After adding `xrs flag the kettle/tomcat behaves the same way and crushes without any errors in log files. </p></li>
  <li><p>Running starce command on relevant process gives us the following output:</p>
  
  <p>Process 12917 attached - interrupt to quit
  futex(0x7f6b9e8ab9d0, FUTEX_WAIT, 12918, NULLPANIC: attached pid 12917 exited with 143
  </p>
  
  <p>We cannot run <code>starce -f</code> to include all threads because it causes our application to be stuck.
   However, killing son process with <code>kill</code> command behaves similary - we see that parent JVM
   exit with error code 143 and we see also the signal sent to the son process.</p></li>
  <li><p>We are running also with java <code>-XX:+HeapDumpOnOutOfMemoryError</code> optind to create a heap dump for jvm crash due to out of memory error but no core dump or heap dumps are creted due to the crush.</p></li>
  </ol>
  
  <p>Any ideas?</p>