anchor.diff:  
   The unexpectedly high latency and jitter that is commonly observed in network benchmarks seem to be caused almost entirely by the underlying operating system and it's device drivers. The two charts below show a comparable baseline setup with one sender running on an embedded device and one running on a RT Linux system over 10 hours and 10 minutes respectively. Note that the RT Linux system is connected using a Gigabit connection and should actually receive datagrams ~40us before the embedded device does.
   
  -image::udp/rtt-linux-1x-comparison-10h.png[]
  -
  -image::udp/rtt-linux-1x-comparison-10m.png[]
  -
  -I realize that this post still leaves many open questions, but it's already a bit too long as it is. I'm considering doing another post in the future that looks at additional cases such as differences in operating systems, network interfaces, device drivers, and the performance impacts of sending uncontrolled traffic (e.g. streaming video) through the same network. There may also be a blog post discussing clock drift and clock synchronization using IEEE 1588v2.
  -
  -
  +[.text-center]
  +.Baseline setup on embedded device vs Linux computer (10h)
  +image::udp/rtt-linux-1x-comparison-10h.png[link="../images/udp/rtt-linux-1x-comparison-10h.png"]
  +
  +[.text-center]
  +.Baseline setup on embedded device vs Linux computer (10min)
  +image::udp/rtt-linux-1x-comparison-10m.png[link="../images/udp/rtt-linux-1x-comparison-10m.png"]
  +
  +I realize that this post still leaves many open questions, but it's already quite long as it is. I'm considering doing another post in the future that looks at additional cases such as differences in operating systems, network interfaces, device drivers, and the performance impacts of sending uncontrolled traffic (e.g. streaming video) through the same network. There may also be a blog post discussing clock drift and clock synchronization using IEEE 1588v2.
  +
  +
  
positive.code: 
  As the sender and receiver devices I'm using two HEBI Robotics I/O Boards. It's a product that we typically don't advertise, but customers often use it to integrate actuators with external devices or to get various sensor input into MATLAB. It has 48 pins that serve a variety of functions (analog/digitial I/O, PWM, Encoder input, etc.) that can be accessed remotely via network. It contains a 168MHz ARM microcontroller (STM32f407) and a 100 Mbit network port.
  
  image::udp/io-boards.jpg[HEBI Robotics IO Board]
  
  I wrote and programmed custom firmware based on http://www.chibios.org/[ChibiOS 2.6.8] and http://savannah.nongnu.org/projects/lwip/[lwIP 1.4.1] to isolate the network stack. The relevant code pieces can be found https://gist.github.com/ennerf/36a57d432bcff20a58efcdee10f91bd9[here]. The elapsed time is measured using a hardware counter that was set to a 250ns resolution.
  
  
negative.code: 
  == Micro-bursting
  
  When I get this far in a conversation, I often get push back claiming that while this may be true for single devices, UDP will surely blow up once more than one thing is connected.
  
  In order to test the actual behavior we put together a test setup consisting of 40 HEBI Robotics I/O boards that are connected to a  http://www.downloads.netgear.com/files/GDC/GS748Tv1/GS748T_ds_03Feb05.pdf[GS748T] 48-port Ethernet switch. Each board runs the 'receiver' firmware. Sending a single broadcast message to all receiver devices triggers 40x266 byte (>10KB total) return packets that can arrive at the switch within less than 250 nanoseconds. This https://en.wikipedia.org/wiki/Micro-bursting_(networking)[Microburst] traffic pattern is representative of a very high degree of freedom system such as a full body humanoid robot.
  
  [.text-center]
  .Network test setup with 40 HEBI Robotics IO Boards
  image::udp/multiple-boards.jpg[link="../images/udp/multiple-boards.jpg"]
  
  [NOTE]
  TODO: Replace with a new picture that doesn't have a heater in the back?
  
  This setup also benefits from two convenient side effects of using standard networking:
  
  1. There is no need for any manual addressing because of https://en.wikipedia.org/wiki/Dynamic_Host_Configuration_Protocol[DHCP] and device specific globally unique mac addresses.
  
  2. It takes only 3-6 seconds to re-program the firmware of all 40 devices at once using a bootloader and TCP/IP.
  
  In order to keep the sender device from overloading, I changed the sender to a http://www.gigabyte.com/products/product-page.aspx?pid=4888#ov[Gigabyte Brix i7-4770R] computer running https://ennerf.github.io/2016/09/20/A-Practical-Look-at-Latency-in-Robotics-The-Importance-of-Metrics-and-Operating-Systems.html[Scientific Linux 6.6 with real-time kernel] with a disabled firewall. First, I looked at the jitter of the underlying operating system. The figure below shows the difference between an idealized signal that ticks every 10ms and the actual measurements of the loop start times. It shows that this setup suffers from more than an order of magnitude more jitter than the embedded OS. (Note that the corresponding jHiccup control chart looks identical as in the OS blog post.)
  
  [.text-center]
  .Operating system jitter of Scientific Linux 6.6 with MRG Realtime
  image::udp/os-jitter-linux-rt.png[link="../images/udp/os-jitter-linux-rt.png"]
  
  Gathering data at 100Hz close to 90 minutes resulted in more than 20 million returned packets. I also ran some tests at 1KHz and the result looks the same.
  
  [.text-center]
  .Receive pattern for incoming datagrams from 40 devices
  image::udp/rtt-linux-40x-zoomed.png[link="../images/udp/rtt-linux-40x-zoomed.png"]
  
  It may be surprising, but there was again no packet loss or re-ordering of packets from a single source. Rather than blowing up, what actually happens is that all packets get stored in the internal buffer of the switch (1.6MB), queued, and forwarded to the target port as fast as possible. Since the original sender is connected via 1 Gigabit, the packets arrive every ~2us. All of them have the same start timestamp, so they show up as a vertical column in the graphs.
  
  [.text-center]
  .Round-trip latency for datagrams from 40 devices (90 min)
  image::udp/rtt-linux-40x.png[link="../images/udp/rtt-linux-40x.png"]
  
  However, what did actually surprise me was how much worse the latency turned out to be compared to the embedded system. I expected most columns to start at around ~180us and end at ~280us. While this is sometimes the case, the majority of columns start above 300 us. After some initial research I suspect that this is related to suboptimal device drivers, and the Linux https://en.wikipedia.org/wiki/New_API[NAPI] using polling mode rather than interrupts. I remember reading about some of this in the past, but I didn't expect this order of magnitude. The installed network interface and driver are below.
  
  [source,shell]
  $ lspci | grep Ethernet
  