title: Tomcat dies suddenly
body: <p>Trying to diagnose some bizarre Tomcat (<code>7.0.21</code>) and/or JVM errors on a 64-bit linux (CentOS) machine.</p>
  
  <p>I'm load testing our server application and tried hitting it with 100K messages. Launched <code>jvisualvm</code> and kept my eye on the heap the whole time. Everything was looking great* (see below) until I got to about 93K processed messages and then Tomcat just died. Ran a <code>ps</code> on Tomcat's PID number to confirm it was dead.</p>
  
  <p>Up until this crash:</p>
  
  <ul>
  <li>Load test had been running for about 90 minutes; should have finished shortly thereafter since we were at 93K/100K)</li>
  <li>CPU was holding strong around 45%</li>
  <li>Used heap was around 2GB (plus or minus a bunch after GCs) but heap size grew from 4GB to <code>MAX_HEAP</code> after about 30 minutes</li>
  <li>Class loading/unloading was cycling normally</li>
  <li>Thread dumps were normal</li>
  </ul>
  
  <p>Nowhere in the server code are any calls to <code>System.exit()</code> - so we can rule that right out (and yes I've double-checked!!!).</p>
  
  <p>I'm not sure if this is Tomcat crashing or the JVM (<strong>how do I tell?</strong>). And even if I did know, I can't seem to find any indication of what went wrong:</p>
  
  <ul>
  <li>All of the server app's logs just stop without any ERROR messages (even though we have logging universally set to DEBUG and higher)</li>
  <li>Tomcat's <code>catalina.out</code> and respect <code>localhost_access_*</code> files just stop without any info</li>
  </ul>
  
  <p>I've heard it is possible to have Tomcat log a coredump when it does but not sure how to do that and online examples aren't helping much.</p>
  
  <p>How would SO go about diagnosing this? What steps should I take to start ruling out all of the possible factors?</p>
  
  <p>Thanks in advance!</p>