title: "Error L10 (output buffer overflow)" when writing to Splunk drain
body: <p>I'm pushing my logs to a local splunk installation. Recently I found that the following error repeats a lot (about once every minute):</p>
  
  <blockquote>
    <p>Error L10
    (output buffer overflow): 7150 messages dropped since
    2013-06-26T19:19:52+00:00.134 &lt;13>1 2013-07-08T14:59:47.162084+00:00
    host app web.1 - [\x1B[37minfo\x1B[0m] application - Perf - it took 31
    milliseconds to fetch row IDs ...</p>
  </blockquote>
  
  <p>The errors repeat quite a lot, and <a href="https://devcenter.heroku.com/articles/error-codes#l10-drain-buffer-overflow" rel="noreferrer">in the documentation</a> it is said that these errors happen when your application produces a lot of logs.</p>
  
  <p>Thing is, I barely have 20-30 logs per second, which isn't really considered a lot. I tested with other drains (added the built-in papertrail plugin), and these errors do not happen there - so they are specific to the outgoing splunk drain.</p>
  
  <p>I thought maybe the splunk machine was loaded and thus not accepting logs fast enough, but its CPU is idle, and it has plenty of disk &amp; memory.</p>
  
  <p>Also, I believe the app (Play 2 app) is auto-flushing logs to console all the time, so there is no big buildup of unflushed logs followed by a release.</p>
  
  <p>What can cause a slow drain speed for the outgoing splunk drain? How should I debug it?</p>