anchor.diff:  
   |====================
   
  -For these benchmarks I chose to measure the round-trip time for UDP datagrams with a payload of 200 bytes. For comparison, the feedback packets of HEBI Robotic's X-Series actuators currently require 185-215 bytes, and requesting the state of all pins of an I/O board currently requires about 300 bytes.
  +For these benchmarks I chose to measure the round-trip time for UDP Datagrams with a payload of 200 bytes. For comparison, the feedback packets of HEBI Robotic's X-Series actuators currently require 185-215 bytes. Returning the state of all pins of an I/O board currently requires about 300 bytes.
   
   While the size is representative of a typical feedback packet, the round trip times in production will be slightly faster because the outgoing packets (commands) are usually significantly smaller than response packets (feedback).
   
  
positive.code: 
  == Micro-bursting
  
  When I get this far in a conversation, I usually get push back claiming that while this may be true for single devices, UDP will surely blow up once more than one thing is connected.
  
  In order to test this hypothesis we put together a test setup consisting of 40 HEBI Robotics I/O boards that are connected to a  http://www.downloads.netgear.com/files/GDC/GS748Tv1/GS748T_ds_03Feb05.pdf[GS748T] 48-port Ethernet switch. Each board runs the 'receiver' firmware. This enables us to send a single broadcast message that gets received by all devices at approximately the same time. This then triggers 40x266 byte (>10KB total) return packets that can arrive at the switch within less than 250 nanoseconds, resulting in a https://en.wikipedia.org/wiki/Micro-bursting_(networking)[Microburst]. This would be representative of a very high degree of freedom system such as a full body humanoid robot.
  
  image::udp/multiple-boards.jpg[]
  
  In order to keep the sender device from overloading, I changed the sender to a http://www.gigabyte.com/products/product-page.aspx?pid=4888#ov[Gigabyte Brix i7-4770R] computer running https://ennerf.github.io/2016/09/20/A-Practical-Look-at-Latency-in-Robotics-The-Importance-of-Metrics-and-Operating-Systems.html[Scientific Linux 6.6 with real-time kernel] with a disabled firewall. First, I looked at the jitter of the underlying operating system. The figure below shows the difference between an idealized signal that ticks every 10ms and the actual measurements of the loop start times. It shows that this setup suffers from more than an order of magnitude more jitter than the embedded OS. (Note that the corresponding jHiccup control chart looks identical as in the OS blog post.)
  
  image::udp/os-jitter-linux-rt.png[]
  
  Gathering data at 100Hz close to 90 minutes resulted in more than 20 million returned packets. I also ran some tests at 1KHz and the result looks the same.
  
  image::udp/rtt-linux-40x-zoomed.png[]
  
  It may be surprising, but there was again no packet loss or re-ordering of packets from a single source. Rather than blowing up, what actually happens is that all packets get stored in the internal buffer of the switch (1.6MB), queued, and forwarded to the target port as fast as possible. Since the original sender is connected via 1 Gigabit, the packets arrive every ~2us. All of them have the same start timestamp, so they show up as a vertical column in the graphs.
  
  image::udp/rtt-linux-40x.png[]
  
  What surprised me is that the latency is so much higher than the previous tests. Looking at the previous tests, I would have expected most columns to start at around ~180us and end at ~280us. While this is sometimes the case, the majority of columns start above 300 us. After some initial research I suspect that this is related to suboptimal device drivers and the Linux https://en.wikipedia.org/wiki/New_API[NAPI] using polling mode rather than interrupts. The installed network interface and driver are below.
  
  [source,shell]
  $ lspci | grep Ethernet
  
negative.code: 
  [.text-center]
  .Round Trip Time for 200 byte payload (10min)
  image::udp/rtt-baseline-zoomed.png[link="../images/udp/rtt-baseline-zoomed.png"]
  
  Overall this initial data looks very promising for being able to use UDP for many real-time control tasks. With more tuning and a better implementation (e.g. lwip with zero copy and tuned options) it seems likely that the maximum jitter could go down to below 6us and potentially even 1us.
  
  