title: Can't find reason for progressive slow down in API response time despite extensive profiling, any ideas?
body: <p>Our application exposes a JSON-RPC API, using Jetty and Google Gson to parse/generate the JSON code.</p>
  
  <p>A remote system opens several thousand persistent HTTP connections to our application, and starts sending API requests at the rate of about 50 per second.  Our application generates a response using CPU only (ie. no disk or database access).  Our app is running on an EC2 virtual machine.</p>
  
  <p>When our app first starts its typical response time is 1-2ms, however over the course of several hours this increases steadily until eventually it gets as high as 80ms which is too slow for our application.  Here is a graph showing response time in nanoseconds, note the steady increase.</p>
  
  <p><img src="https://i.stack.imgur.com/jZPMO.png" alt="Graph of API response time in Nanoseconds"></p>
  
  <p>I have used YourKit profiler to capture CPU snapshots shortly after startup and then again later when it's significantly slower.  The problem is that no one method seems to account for the slowdown, everything just gets slower with time.</p>
  
  <p>The number of threads and memory usage don't seem to increase either, so I'm now at a loss as to the likely cause of the slowdown.</p>
  
  <p>Does anyone have any ideas as to what the cause might be, or a suggestion as to a more effective way to isolate the problem?</p>