title: Operation timed out and Connection reset by peer - 400 concurrent accesses to MySQL
body: <p><strong>Some background</strong></p>
  
  <p>I am working on project where we have a obsolete very simple bookshop web application from 1999 written in Java EE 1.4. However, the purpose is to modernize it in order to run it in cloud environment like Amazon AWS. We decided that we will rewrite it from start using Spring framework, Hibernate and c3p0 for connection pooling.</p>
  
  <p>Beside web application we also got a stress tool for web application called RBE (Remote Browser Emulator). You simply tell RBE how many users do you want and he create one thread for each user. Each thread is randomly generating url and it is doing so as long as user browse the page (we defined that user browse site for 180 seconds). After that, it shuts down a thread, and that's. For each request it takes HTML page source, parse out jsessionid and then go on next url.</p>
  
  <p><strong>The problem</strong></p>
  
  <p>When I run RBE for 400 users (that is 400 requests per second), I quickly get the following exception:</p>
  
  <pre><code>java.net.SocketException: Connection reset by peer
      at java.net.PlainSocketImpl.socketConnect(Native Method)
      at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:382)
      at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:241)
      at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:228)
      at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:431)
      at java.net.Socket.connect(Socket.java:527)
      at sun.net.NetworkClient.doConnect(NetworkClient.java:158)
      at sun.net.www.http.HttpClient.openServer(HttpClient.java:388)
      at sun.net.www.http.HttpClient.openServer(HttpClient.java:523)
      at sun.net.www.http.HttpClient.&lt;init&gt;(HttpClient.java:227)
      at sun.net.www.http.HttpClient.New(HttpClient.java:300)
      at sun.net.www.http.HttpClient.New(HttpClient.java:317)
      at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970)
      at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911)
      at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836)
      at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172)
      at rbe.EB.getHTML(EB.java:423)
      at rbe.EB.run(EB.java:324)
  </code></pre>
  
  <p>After a while, when RBEs try to browse other sub pages I get:</p>
  
  <pre><code>java.net.ConnectException: Operation timed out
      at java.net.PlainSocketImpl.socketConnect(Native Method)
      at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:382)
      at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:241)
      at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:228)
      at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:431)
      at java.net.Socket.connect(Socket.java:527)
      at sun.net.NetworkClient.doConnect(NetworkClient.java:158)
      at sun.net.www.http.HttpClient.openServer(HttpClient.java:388)
      at sun.net.www.http.HttpClient.openServer(HttpClient.java:523)
      at sun.net.www.http.HttpClient.&lt;init&gt;(HttpClient.java:227)
      at sun.net.www.http.HttpClient.New(HttpClient.java:300)
      at sun.net.www.http.HttpClient.New(HttpClient.java:317)
      at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970)
      at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911)
      at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836)
      at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172)
      at rbe.EB.getHTML(EB.java:423)
      at rbe.EB.run(EB.java:324)
  </code></pre>
  
  <p>The code at <code>EB.java:423)</code> (where the exception occur) is: </p>
  
  <pre><code>BufferedReader in = new BufferedReader( new InputStreamReader(connection.getInputStream() ) );
  </code></pre>
  
  <p>My <code>my.cnf</code> configuration:</p>
  
  <pre><code>[mysqld]
  log=/var/log/mysqld.log
  max_connections = 200
  log-slow-queries = /var/log/mysqld_slow_queries.log
  log-error = /usr/local/mysql/data/mysql-error.log
  back-log = 200 
  key_buffer_size=16M
  max_allowed_packet=100M
  interactive_timeout=30
  wait_timeout=30
  </code></pre>
  
  <p>Tomcat relevant settings in <code>server.xml</code>:</p>
  
  <pre><code>&lt;Connector connectionTimeout="20000" port="8080" protocol="HTTP/1.1" redirectPort="8443" acceptCount="500" /&gt;
  </code></pre>
  
  <p>I also did a test and served static html files with 2 seconds delay and did not do any iteraction with database. I did this test to see if Tomcat can't handle 400 concurrent requests. Tomcat passed this test successfully.</p>
  
  <p>What is the bottleneck for 400 concurrent accesses to mysql? It's this too much for MySQL?</p>