title: UDP client periodic sendto() delay in C in Linux
body: <p>I am developing a UDP client in C in Linux with glibc 2.23. My application sends small packets (11 bytes) in a loop with a rate of 20,000 packets/sec. </p>
  
  <p>To ensure that the client met its promised send rate, I instrumented my application to measure UDP send timings when I saw a strange behavior. The time it takes to send a packet (a call to <code>sendto()</code> function) is "usually" around 13 usec. </p>
  
  <p>But approximately every 30 msec, there is a call which takes around 50 usec and approximately every 1500 ms, there is a call which takes 160 usec. The numbers are approximate and are average of a long 1 minute run. </p>
  
  <p>The graph below shows the delay values for 5 second run of the application.
  <img src="https://i.stack.imgur.com/Wlhox.png" alt="sendto() call delay as a function of runtime"></p>
  
  <p>Here are the things that I tried:</p>
  
  <ul>
  <li>Using affinity to pin my application to a core to avoid context switch</li>
  <li>Changing the UDP send/receive buffer size from 100K to 1G (smaller buffer sizes slightly reduces the amount of delay but not the frequency)</li>
  <li>Changing form IPV4 to IPV6 (no difference)</li>
  </ul>
  
  <p>Also tried:</p>
  
  <ul>
  <li>Disabled <code>arp</code> (address resolution protocol) </li>
  <li>Ran the test while the server and client were at the same machine as well as two different machines (connected via a switch), no difference in the results</li>
  <li>switched between the 10G and 1G NICs, no difference in the results</li>
  <li>Ran client and server on two docker instances on the same machine connected via a virtual bridge, no difference in the results. </li>
  </ul>
  
  <p>I was wondering if anyone experienced such behavior? </p>
  
  <p>I appreciate any suggestion about the cause or solution to avoid the costly calls? </p>