title: UDP sendto performance over loopback
body: <h2>Background</h2>
  
  <p>I have a very high throughput / low latency network app (goal is &lt;&lt; 5 usec per packet) and I wanted to add some monitoring/metrics to it. I have heard about the statsd craze and seems a simple way to collect metrics and feed them into our time series database. Sending metrics is done via a small udp packet write to a daemon (typically running on same server). </p>
  
  <p>I wanted to characterize the effects of sending ~5-10 udp packets in my data path to understand how much latency it would add and was surprised at how bad it is. I know this is a very obscure micro-benchmark but just wanted to get a rough idea on where it lands.</p>
  
  <h2>The question I have</h2>
  
  <p>I am trying to understand why it takes so long (relatively speaking) to send a UDP packet to localhost versus a remote host. Are there any tweaks I can make to reduce the latency to send a UDP packet? I am thinking the solution for me to push metric collection to an auxiliary core or actually run the statsd daemon on a seperate host.</p>
  
  <hr>
  
  <h2>My setup/benchmarks</h2>
  
  <p>CentOS 6.5 with some beefy server hardware. <br/>
  The client test program I have been using is available here:  <a href="https://gist.github.com/rishid/9178261" rel="nofollow noreferrer">https://gist.github.com/rishid/9178261</a> <br/>
  Compiled with gcc 4.7.3 <code>gcc -O3 -std=gnu99 -mtune=native udp_send_bm.c -lrt -o udp_send_bm</code> <br/>
  The receiver side is running <code>nc -ulk 127.0.0.1 12000 &gt; /dev/null</code> (ip change per IF)<br/></p>
  
  <p>I have ran this micro-benchmark with the following devices. <br>
  Some benchmark results:</p>
  
  <ul>
  <li>loopback
  
  <ul>
  <li>Packet Size 500 // Time per sendto() 2159 nanosec // Total time 2.159518</li>
  </ul></li>
  <li>integrated 1 Gb mobo controller
  
  <ul>
  <li>Packet Size 500 // Time per sendto() 397 nanosec // Total time 0.397234</li>
  </ul></li>
  <li>intel ixgbe 10 Gb
  
  <ul>
  <li>Packet Size 500 // Time per sendto() 449 nanosec // Total time 0.449355</li>
  </ul></li>
  <li>solarflare 10 Gb with userspace stack (onload)
  
  <ul>
  <li>Packet Size 500 // Time per sendto() 317 nanosec // Total time 0.317229</li>
  </ul></li>
  </ul>