anchor.diff:  
   90% of all packets arrive within 194us and a jitter of less than 1 microsecond. Roughly 80us of this time is spent on the wire, so using chips that support Gigabit (rather than 100Mbit) could lower the round trip time to ~120us. Above the common case, there are three different periodically reoccuring modes that cause the round-trip-time to get up to a worst case that is 60us higher. 
   
  -* Mode 1 occurs consistently every 5.3 minutes and lasts for 15.01 seconds. During this time it adds up to 4 us latency.
  +* Mode 1 occurs consistently every ~5.3 minutes and lasts for ~15.01 seconds. During this time it adds up to 4 us latency.
   * Mode 2 occurs exactly once every 5 seconds and is always at 210us.
   * Mode 3 occurs roughly once an hour and adds linearly increasing latency up to 60us to 10 packets.
   
  -Below is a zoomed in view of a 10 minute timespan that better shows Mode 1 and 2.
  +So far I have not been successful in determining the root cause of these modes. All three modes seem to be related to actual time and independent of rate and packet count. However, after several tests, I strongly suspect that all of them occur on the firmware side rather than being tied to the switch or the protocol itself. Below is a zoomed in view of a 10 minute time span that better shows Modes 1 and 2.
   
   [.text-center]
   .Round Trip Time for 200 byte payload (10min)
   image::udp/rtt-baseline-zoomed.png[]
   
  -So far I have not been successful in determining the root cause of these modes. All 3 modes seem to be related to actual time and independent of rate and packet count. However, after several tests, I strongly suspect that all of them occur on the firmware side rather than being tied to the switch or the protocol itself. With more tuning and a better implementation (e.g. lwip with zero copy and tuned options) it seems likely that the maximum jitter could go down to below 6us and potentially even 1us.
  +Overall this initial data looks very promising for being able to use UDP for many real-time control tasks. With more tuning and a better implementation (e.g. lwip with zero copy and tuned options) it seems likely that the maximum jitter could go down to below 6us and potentially even 1us.
   
   // test IO board to IO board (100)
   
  
positive.code: // https://en.wikipedia.org/wiki/New_API
  // http://blog.packagecloud.io/eng/2016/06/22/monitoring-tuning-linux-networking-stack-receiving-data/
  
  Even for a single target, there is a lot of jitter in the Linux network driver.
  
  image::udp/rtt-linux-1x.png[]
  
  == Conclusion
  
  
negative.code: 
  == Benchmark Setup
  
  Most of the UDP benchmarks that I was able to find online setup two Desktop computers, where the sender sends a message, the receiver echoes it back, and the sender measures the time it took for a round trip. This works well, but there is a lot of 'magic' that happens in the underlying operating system and it's device drivers.
  
  In order to mitigate this potential source of jitter, I setup a benchmark that measures the round trip time between two dedicated embedded systems. Aside from being on an embedded system, the approach is the same, i.e.,
  
  1. Sender sends packet to receiver in fixed intervals
  2. Receiver echoes packet back to sender
  3. Sender measures the round trip time
  4. Measurement gets persisted to disk
  
  As the sender and receiver devices I'm using two HEBI Robotics I/O Boards. It's a product that we haven't advertised yet, but customers usually use it to integrate actuators with external devices or to get various sensor input into MATLAB. It has 48 pins that serve a variety of functions (analog/digitial I/O, PWM, Encoder input, etc.) that can be accessed remotely via network. It sports a STM32f407 (168MHz) microcontroller and a 100 Mbit network port. In order to make sure that the devices are not doing anything else, I programmed custom firmware specifically for testing the network stack. The relevant code pieces can be found https://gist.github.com/ennerf/36a57d432bcff20a58efcdee10f91bd9[here]. The hardware timer (counter) resolution was set to 250ns.
  
  image::udp/io-boards.jpg[HEBI Robotics IO Board]
  
  Since the resulting log files are too large to be stored directly on the device, I've setup a https://gist.github.com/ennerf/0ddc4396d15852d28e4eca4a8a923eb7[UDP server (Java)] that receives measurement data and persists the data to disk. The main socket handler writes incoming packets into a double buffered structure that can be persisted by a background thread without halting the packet handler. The synchronization between the threads is done using a http://stuff-gil-says.blogspot.com/2014/11/writerreaderphaser-story-about-new.html[WriterReaderPhaser], which is a synchronization primitive that is very useful for persisting events that are represented by a small amount of data.
  
  The resulting binary data was loaded into MATLAB(C) for analysis and to generate the plots. The code can be found https://gist.github.com/ennerf/19b48406a066f6e946a0567a1a4de1ed[here].
  
  == Ethernet Packet Structure
  
  