title: Causes of Linux UDP packet drops
body: <p>I have a Linux C++ application which receives sequenced UDP packets.  Because of the sequencing, I can easily determine when a packet is lost or re-ordered, i.e. when a "gap" is encountered.  The system has a recovery mechanism to handle gaps, however, it is best to avoid gaps in the first place.  Using a simple libpcap-based packet sniffer, I have determined that there are no gaps in the data at the hardware level.  However, I am seeing a lot of gaps in my application.  This suggests the kernel is dropping packets; it is confirmed by looking at the <strong>/proc/net/snmp</strong> file.  When my application encounters a gap, the <strong>Udp InErrors</strong> counter increases.</p>
  
  <p>At the system level, we have increased the max receive buffer:</p>
  
  <pre><code># sysctl net.core.rmem_max
  net.core.rmem_max = 33554432
  </code></pre>
  
  <p>At the application level, we have increased the receive buffer size:</p>
  
  <pre><code>int sockbufsize = 33554432
  int ret = setsockopt(my_socket_fd, SOL_SOCKET, SO_RCVBUF,
          (char *)&amp;sockbufsize,  (int)sizeof(sockbufsize));
  // check return code
  sockbufsize = 0;
  ret = getsockopt(my_socket_fd, SOL_SOCKET, SO_RCVBUF, 
          (char*)&amp;sockbufsize, &amp;size);
  // print sockbufsize
  </code></pre>
  
  <p>After the call to getsockopt(), the printed value is always 2x what it is set to (67108864 in the example above), but I believe that is to be expected.</p>
  
  <p>I know that failure to consume data quickly enough can result in packet loss.  However, all this application does is check the sequencing, then push the data into a queue; the actual processing is done in another thread.  Furthermore, the machine is modern (dual Xeon X5560, 8 GB RAM) and very lightly loaded.  We have literally dozens of identical applications receiving data at a <em>much higher rate</em> that do not experience this problem.</p>
  
  <p>Besides a too-slow consuming application, are there other reasons why the Linux kernel might drop UDP packets?</p>
  
  <p>FWIW, this is on CentOS 4, with kernel 2.6.9-89.0.25.ELlargesmp.</p>