title: Weird Tomcat outage, possibly related to maxConnections
body: <p>In my company we experienced a serious problem today: our production server went down.  Most people accessing our software via a browser were unable to get a connection, however people who had already been using the software were able to continue using it. Even our hot standby server was unable to communicate with the production server, which it does using HTTP, not even going out to the broader internet. The whole time the server was accessible via ping and ssh, and in fact was quite underloaded - it's normally running at 5% CPU load and it was even lower at this time. We do almost no disk i/o.</p>
  
  <p>A few days after the problem started we have a new variation:  port 443 (HTTPS) is responding but port 80 stopped responding. The server load is very low. Immediately after restarting tomcat, port 80 started responding again.</p>
  
  <p>We're using tomcat7, with maxThreads="200", and using maxConnections=10000. We serve all data out of main memory, so each HTTP request completes very quickly, but we have a large number of users doing very simple interactions (this is high school subject selection). But it seems very unlikely we would have 10,000 users all with their browser open on our page at the same time.</p>
  
  <p>My question has several parts:</p>
  
  <ul>
  <li>Is it likely that the "maxConnections" parameter is the cause of our woes?</li>
  <li>Is there any reason not to set "maxConnections" to a ridiculously high value e.g. 100,000?  (i.e. what's the cost of doing so?)</li>
  <li>Does tomcat output a warning message anywhere once it hits the "maxConnections" message?  (We didn't notice anything).</li>
  <li>Is it possible there's an OS limit we're hitting?  We're using CentOS 6.4 (Linux) and "ulimit -f" says "unlimited".  (Do firewalls understand the concept of Tcp/Ip connections? Could there be a limit elsewhere?)</li>
  <li>What happens when tomcat hits the "maxConnections" limit? Does it try to close down some inactive connections?  If not, why not?  I don't like the idea that our server can be held to ransom by people having their browsers on it, sending the keep-alive's to keep the connection open.</li>
  </ul>
  
  <p>But the main question is, "How do we fix our server?"</p>
  
  <p>More info as requested by Stefan and Sharpy:</p>
  
  <ul>
  <li>Our clients communicate directly with this server</li>
  <li>TCP connections were in some cases immediately refused and in other cases timed out</li>
  <li>The problem is evident even connecting my browser to the server within the network, or with the hot standby server - also in the same network - unable to do database replication messages which normally happens over HTTP</li>
  <li>IPTables - yes, IPTables6 - I don't think so. Anyway, there's nothing between my browser and the server when I test after noticing the problem.</li>
  </ul>
  
  <p>More info:
  It really looked like we had solved the problem when we realised we were using the default Tomcat7 setting of BIO, which has one thread per connection, and we had maxThreads=200. In fact 'netstat -an' showed about 297 connections, which matches 200 + queue of 100. So we changed this to NIO and restarted tomcat. Unfortunately the same problem occurred the following day.  It's possible we misconfigured the server.xml.</p>
  
  <p>The server.xml and extract from catalina.out is available here:
  <a href="https://www.dropbox.com/sh/sxgd0fbzyvuldy7/AACZWoBKXNKfXjsSmkgkVgW_a?dl=0" rel="noreferrer">https://www.dropbox.com/sh/sxgd0fbzyvuldy7/AACZWoBKXNKfXjsSmkgkVgW_a?dl=0</a></p>
  
  <p>More info:
  I did a load test.  I'm able to create 500 connections from my development laptop, and do an HTTP GET 3 times on each, without any problem. Unless my load test is invalid (the Java class is also in the above link).</p>