title: web application runs much faster in embedded tomcat than in standalone tomcat
body: <p>I have a spring-boot web application (mostly used through REST calls), that I can run using <code>mvn exec</code> that starts an embedded tomcat (8.5.11), or build a war and deploy it into a standalone tomcat (debian stock 8.5.14-1~bpo8+1). Both are configured the same way, using </p>
  
  <p>To our utmost surprise, the embedded tomcat seems to be much faster for high loads (a small test sequence with 200+ threads using jmeter). At 600 threads, for example:</p>
  
  <ul>
  <li>The standalone tomcat has very large response times, while having a relatively low load of 50-70 (the server has 64 cores and can run 128 threads), and a low IO usage.</li>
  <li>The embedded tomcat has a load of 150-200 and faster response times, and high I/O usage (it seems that the database is the limiting factor here, but it degrades gracefully: 600 threads results in double as slow as 300 threads).</li>
  </ul>
  
  <p>Supposedly, the configuration is the same for both tomcats, so currently I am quite troubled because of this. I really would not like to run embedded tomcat in production if I can help it.</p>
  
  <p>Does anyone have an idea:</p>
  
  <ul>
  <li>what the cause for this performance disparity may be, and</li>
  <li>how we can reliably compare the configuration for two tomcats?</li>
  </ul>
  
  <p><strong>Update</strong></p>
  
  <p>I ran some more tests and discovered a significant difference after looking through the Garbage Collector logs: with 600 jmeter threads, the embedded tomcat spent about 5% of its time GCing, while the standalone tomcat spent about 50% of its time GCing. I calculated these numbers with an awk script, so they may be a bit mis-parsed, but manually checking the GC logs seems to corroborate them. It still does not explain <em>why</em> one of them is GCing all the time and the other is not...</p>
  
  <p><strong>One more update</strong></p>
  
  <p>I managed to speed up the standalone tomcat by switching the garbage collector to G1. Now, it uses about 20% of elapsed time for garbage collection, and never exceeds 1s for any single GC run. Now the standalone tomcat is only 20-30% slower than the embedded tomcat. Interestingly, using G1 in the embedded tomcat had no real effect on its performance, GC overhead is still around 15% there.</p>
  
  <p>This is by no means a solution, but it helped to close the gap between the two tomcats and thus now the problem is not so critical.</p>