anchor.diff:  
   |====================
   
  -All of the benchmarks in this post measure the round-trip time for a UDP datagram with a payload of 200 bytes. After including all overhead, the actual frame size on the wire is 266 bytes. This may seem high to users to are used to dealing with serial protocols, but is representative of our standard use cases. For comparison, the feedback packets of HEBI Robotic's X-Series actuators currently require a payload of 185-215 bytes. Requesting the state of all pins of an I/O board currently requires about 300 bytes.
  +All of the benchmarks in this post measure the round-trip time for a UDP datagram with a payload of 200 bytes. After including all overhead, the actual frame size on the wire is 266 bytes. For comparison, the feedback packets of HEBI Robotic's X-Series actuators currently require a payload of 185-215 bytes. Requesting the state of all pins of an I/O board currently requires about 300 bytes.
  +
  +While this size is representative of our standard use case, the round trip times in practice are slightly faster because the outgoing packets (commands) are usually significantly smaller than response packets (feedback).
   
   The theoretical time to send 266 bytes over the wire is 20.3us over 100 Mbit/s Ethernet and 2.03us over 1 Gbit/s Ethernet.
   
  -Note that the round trip times in practice are slightly faster because the outgoing packets (commands) are usually significantly smaller than response packets (feedback).
  -
   == Baseline: Single Switch
   
  -We can establish a baseline of the best-case round trip time (RTT) by having two devices communicate with each other through a single switch that does not see any external traffic. It's possible to create a point-to-point connection without any switches, but the typical use case goes through at least one switch.
  +We can establish a baseline of the best-case round trip time (RTT) by having both devices communicate with each other through a single switch that does not see any external traffic. It's possible to create a point-to-point connection without any switches, but the typical use case goes through at least one switch.
   
   I set the frequency to 100Hz and logged data for close to 24 hours. I chose this frequency because it is a common control rate for sending high-level trajectories, and because 10ms is a safe deadline in case there are large outliers. Faster control rates are of course possible.
   
  
positive.code: 
  == Benchmark Setup
  
  In order to mitigate jitter caused by the operating system and it's device drivers, I've tried to setup a benchmark that measures the communications latency between two dedicated embedded systems.
  
  1. Sender sends packet to receiver in fixed intervals
  2. Receiver echoes packet back to sender
  3. Sender measures round trip time and sends results to logging server
  4. Logging server persists data to disk
  
  As the sender and receiver devices I'm using two HEBI Robotics I/O Boards. It's a product that we haven't advertised yet, but customers usually use it to integrate actuators with external devices or to get various sensor input into MATLAB. It has 48 pins that serve a variety of functions (analog/digitial I/O, PWM, Encoder input, etc.) that can be accessed remotely via network. It sports a STM32f407 microcontroller and a 100 Mbit network port. I programmed custom firmware to test the network stack. The relevant code pieces can be found https://gist.github.com/ennerf/36a57d432bcff20a58efcdee10f91bd9[here].
  
  image::udp/io-boards.jpg[HEBI Robotics IO Board]
  
  Since the logs are too large to be stored directly on the device, I've setup a https://gist.github.com/ennerf/0ddc4396d15852d28e4eca4a8a923eb7[UDP server (Java)] that receives measurement data and persists the data to disk. The main socket handler writes incoming packets into a double buffered structure that can be persisted by a background thread without halting the packet handler. The synchronization between the threads is done using a http://stuff-gil-says.blogspot.com/2014/11/writerreaderphaser-story-about-new.html[WriterReaderPhaser], which is a synchronization primitive that is very useful for persisting events that are represented by a small amount of data.
  
  The resulting binary data was loaded into MATLAB(C) for analysis and to generate the plots.
  
  == Ethernet Packet Structure
  
  
negative.code: 
  90% of all packets arrive within 194us and a jitter of less than 1 microsecond. Roughly 80us of this time is spent on the wire, so using chips that support Gigabit (rather than 100Mbit) could lower the round trip time to ~120us. Above the common case, there are three different periodically reoccuring modes that cause the round-trip-time to get up to a worst case that is 60us higher. 
  
  * Mode 1 occurs consistently every ~5.3 minutes and lasts for ~15.01 seconds. During this time it adds up to 4 us latency.
  * Mode 2 occurs exactly once every 5 seconds and is always at 210us.
  * Mode 3 occurs roughly once an hour and adds linearly increasing latency up to 60us to 10 packets.
  
  So far I have not been successful in determining the root cause of these modes. All three modes seem to be related to actual time and independent of rate and packet count. However, after several tests, I strongly suspect that all of them occur on the firmware side rather than being tied to the switch or the protocol itself. Below is a zoomed in view of a 10 minute time span that better shows Modes 1 and 2.
  
  [.text-center]
  .Round Trip Time for 200 byte payload (10min)
  image::udp/rtt-baseline-zoomed.png[]
  
  Overall this initial data looks very promising for being able to use UDP for many real-time control tasks. With more tuning and a better implementation (e.g. lwip with zero copy and tuned options) it seems likely that the maximum jitter could go down to below 6us and potentially even 1us.
  
  // test IO board to IO board (100)
  
  