title: Heroku Sporadic High Response Time
body: <p>This is very specific, but I will try to be brief:</p>
  
  <p>We are running a <strong>Django</strong> app on <strong>Heroku</strong>. Three servers:   </p>
  
  <ol>
  <li>test (1 web, 1 celery dyno)   </li>
  <li>training (1 web, 1 celery dyno)  </li>
  <li>prod (2 web, 1 celery dyno).  </li>
  </ol>
  
  <p>We are using <strong>Gunicorn</strong> with <strong>gevents</strong> and <strong>4 workers on each dyno</strong>.</p>
  
  <p>We are <strong>experiencing sporadic high service times</strong>. Here is an example from Logentries:  </p>
  
  <pre><code>High Response Time:
  heroku router - - at=info 
  method=GET 
  path="/accounts/login/" 
  dyno=web.1 
  connect=1ms 
  service=6880ms 
  status=200 
  bytes=3562
  </code></pre>
  
  <p>I have been Googling this for weeks now. We are unable to reproduce at will but experience these alerts 0 to 5 times a day. <strong>Notable points</strong>:</p>
  
  <ul>
  <li>Occurs on all three apps (all running similar code)  </li>
  <li>Occurs on different pages, including simple pages such as 404 and /admin  </li>
  <li>Occurs at random times </li>
  <li>Occurs with varying throughput. One of our instances only drives 3 users/day. It is not related to sleeping dynos because we ping with New Relic and the issue can occur mid-session</li>
  <li>Unable to reproduce at will. I have experienced this issue personally once. Clicking a page that normally executes in 500ms resulted in a 30 second delay and eventually an app error screen from Heroku's 30s timeout  </li>
  <li>High response times vary from 5000ms - 30000ms.  </li>
  <li>New Relic does not point to a specific issue. Here are the past few transactions and times: 
  
  <ul>
  <li>RegexURLResolver.resolve <code>4,270ms</code></li>
  <li>SessionMiddleware.process_request <code>2,750ms</code></li>
  <li>Render login.html <code>1,230ms</code></li>
  <li>WSGIHandler <code>1,390ms</code></li>
  <li>The above are simple calls and do not normally take near that amount of time  </li>
  </ul></li>
  </ul>
  
  <p><strong>What I have narrowed it down to</strong>:  </p>
  
  <ol>
  <li><s><a href="http://blog.etianen.com/blog/2014/01/19/gunicorn-heroku-django/" rel="noreferrer">This article on Gunicorn and slow clients</a></s> 
  
  <ul>
  <li>I have seen this issue happen with slow clients but also at our office where we have a fiber connection.   </li>
  </ul></li>
  <li><s>Gevent and async workers not playing nicely</s>
  
  <ul>
  <li>We've switched to gunicorn sync workers and problem still persists.</li>
  </ul></li>
  <li>Gunicorn worker timeout 
  
  <ul>
  <li>It's possible that workers are somehow being kept-alive in a null state.  </li>
  </ul></li>
  <li><s>Insufficient workers / dynos</s>
  
  <ul>
  <li>No indication of CPU/memory/db overutilization and New Relic doesn't display any indication of DB latency</li>
  </ul></li>
  <li>Noisy Neighbors
  
  <ul>
  <li>Among my multiple emails with Heroku, the support rep has mentioned at least one of my long requests was due to a noisy neighbor, but was not convinced that was the issue.</li>
  </ul></li>
  <li><s>Subdomain 301</s>  
  
  <ul>
  <li>The requests are coming through fine, but getting stuck randomly in the application.</li>
  </ul></li>
  <li><s>Dynos restarting</s>
  
  <ul>
  <li>If this were the case, many users would be affected. Also, I can see that our dynos have not restarted recently.</li>
  </ul></li>
  <li>Heroku routing / service issue
  
  <ul>
  <li>It is possible that the Heroku service is less than advertised and this is simply a downside of using their service.</li>
  </ul></li>
  </ol>
  
  <p>We have been having this issue for the past few months, but now that we are scaling it needs to be fixed. <strong>Any ideas would be much appreciated</strong> as I have exhausted nearly every SO or Google link.</p>