title: Reuse tomcat threads while waiting "long" time
body: <p><strong>THE CONFIGURATION</strong><br>
  Web server : Nginx<br>
  App server  : Tomcat with default configuration of 200 request serving threads<br>
  Expected response time for my server : ~30 seconds(There are lots of third party dependencies)</p>
  
  <p><strong>THE SCENARIO</strong><br>
  Every 10 seconds the application will need to generate the token for its use. The expected time for token generation is  around 5 seconds, but since its a third party system being contacted over network, this is obviously not consistent and can go up to 10 seconds.<br>
  During the token generation process, nearly 80% of the incoming requests per second will need to wait.</p>
  
  <p><strong>WHAT I BELIEVE SHOULD HAPPEN</strong><br>
  Since the requests waiting for the token generation will have to wait a "long" time, there is no reason for these request serving to be reused to serve other incoming requests while waiting for token generation process to complete.<br>
  Basically, it would make sense if my 20% to keep being served. If the waiting threads are not being utilized for other requests, tomcat request serving limit will be reached and server would essentially choke, not really something any developer will like.</p>
  
  <p><strong>WHAT I TRIED</strong><br>
  Initially I expected switching to tomcat NIO connector would do this job. But after looking at <a href="https://tomcat.apache.org/tomcat-7.0-doc/config/http.html#Connector_Comparison" rel="nofollow noreferrer">this</a> comparison, I was really not hopeful. Nevertheless, I tried by forcing the requests to wait for 10 second and it did not work.<br>
  Now I am thinking on the lines that I need to, sort of, shelve the request while its waiting and need to signal the tomcat that this thread  is free to reuse. Similarly, I will need tomcat to give me a thread from its threadpool when the request is ready to be moved forward. But I am blindsided on how to do it or even if this is possible.</p>
  
  <p>Any guidance or help?</p>