anchor.diff:  
   For these benchmarks we chose to measure the round-trip time for a payload of 200 bytes. After including all overhead, the actual size on the wire is 266 bytes. The theoretical time it takes to transfer 266 bytes over 100 Mbit/s and 1Gbit/s Ethernet is 20.3us and 2.03us respectively.
   
  -Note that while the size is representative of a typical actuator feedback packet, the round trip times in production may be faster because outgoing packets (commands) tend to be significantly smaller than response packets (feedback).
  +Note that while the size is representative of a typical actuator feedback packet, the round-trip times in production may be faster because outgoing packets (commands) tend to be significantly smaller than response packets (feedback).
   
   == Baseline - Single Switch
   
  -We can establish a baseline of the best-case round trip time (RTT) by having both devices communicate with each other through a single switch that does not see any external traffic. It would be possible to create a point-to-point connection without any Switches, but the typical use case goes through at least one Switch.
  +We can establish a baseline of the best-case round-trip time by having both devices communicate with each other through a single switch that does not see any external traffic. It would be possible to create a point-to-point connection without any Switches, but the typical use case goes through at least one Switch.
   
   [.text-center]
   .Baseline setup using single Switch
   image::udp/setup-baseline.png[link="../images/udp/setup-baseline.png"]
   
  -We set the frequency to 100Hz and logged data for close to 24 hours. This frequency was chosen because it is a common control rate for sending high-level trajectories, and because 10ms is a safe deadline in case there are large outliers. Faster control rates are of course possible.
  -
  -First, let's look at the jitter of the underlying embedded real-time operating system (RTOS). The figure below shows the difference between an idealized signal that ticks every 10ms and the actual measurements of the loop start times. 99% are within the lowest measurement resolution (250ns), and the worst observed case is below 6us. Note that this is significantly better than the 150us worst case we observed in a similar test running on real-time Linux (see https://ennerf.github.io/2016/09/20/A-Practical-Look-at-Latency-in-Robotics-The-Importance-of-Metrics-and-Operating-Systems.html[The Importance of Metrics and Operating Systems]).
  +We set the frequency to 100Hz and logged data for ~24 hours. We chose this frequency because it is a common control rate for sending high-level trajectories, and because 10ms is a safe deadline in case there are large outliers. Faster control rates are of course possible.
  +
  +First, let's look at the jitter of the underlying embedded real-time operating system (RTOS). The figure below shows the difference between an idealized signal that ticks every 10ms and the actual measurements of the loop start times. 99% are within the lowest measurement resolution (250ns), and the total observed range is slightly below 6us. Note that this is significantly better than the 150us range we observed in a similar test on real-time Linux (see https://ennerf.github.io/2016/09/20/A-Practical-Look-at-Latency-in-Robotics-The-Importance-of-Metrics-and-Operating-Systems.html[The Importance of Metrics and Operating Systems]).
   
   [.text-center]
   .OS jitter of ChibiOS 2.6.8 on STM32F407 (24h)
   image::udp/os-jitter-embedded.png[link="../images/udp/os-jitter-embedded.png"]
   
  -The two figures below show the round trip time for all packets and the corresponding percentile distribution. There were a total of 8.5 million messages. None of them were lost and none of them arrived out of order.
  +The two figures below show the round-trip time for all packets and the corresponding percentile distribution. There were a total of 8.5 million messages. None of them were lost and none of them arrived out of order.
   
   [[img-rtt-24h]]
   [.text-center]
  -.Round Trip Time for 200 byte payload (24h)
  +.RTT for 200 byte payload (24h)
   image::udp/rtt-baseline.png[link="../images/udp/rtt-baseline.png"]
   
   [.text-center]
  -.Zoomed in view of Round Trip Time for 200 byte payload (10min)
  +.Zoomed in view of RTT for 200 byte payload (10min)
   image::udp/rtt-baseline-zoomed.png[link="../images/udp/rtt-baseline-zoomed.png"]
   
  -90% of all packets arrive within 194us and a jitter of less than 1 microsecond. Roughly 80us of this time is spent on the wire, so using chips that support Gigabit (rather than 100Mbit) could lower the round trip time to ~120us. Above the common case, there are three different periodically reoccuring modes that cause the round-trip-time to get up to a worst case that is 60us higher. 
  +90% of all packets arrive within 194us and a jitter of less than 1 microsecond. Roughly 80us of this time is spent on the wire, so using chips that support Gigabit (rather than 100Mbit) could lower the round-trip time to ~120us. Above the common case, there are three different periodically reoccuring modes that cause the round-trip-time to get up to a worst case that is 60us higher. 
   
   * Mode 1 occurs consistently every ~5.3 minutes and lasts for ~15.01 seconds. During this time it adds up to 4 us latency.
   * Mode 2 occurs exactly once every 5 seconds and is always at 210us.
   * Mode 3 occurs roughly once an hour and adds linearly increasing latency of up to 60us to 10 packets.
   
  -So far we have not been successful in determining the root cause of these modes. All three modes seem to be related to actual time and independent of rate and packet count. However, after several tests, we strongly suspect that all of them occur on the firmware side rather than being tied to the switch or the protocol itself. The zoomed in view of a 10 minute time span highlights Modes 1 and 2.
  +The zoomed in view of a 10 minute time span highlights Modes 1 and 2. So far we have not been successful in determining the root cause of these modes. All three modes seem to be related to actual time and independent of rate and packet count. However, after several tests, we strongly suspect that all of them occur on the firmware side rather than being tied to the switch or the protocol itself.
   
   Overall this initial data looks very promising for being able to use UDP for many real-time control tasks. With more tuning and a better implementation (e.g. lwip with zero copy and tuned options) it seems likely that the maximum jitter could be reduced to below 6us and maybe even below 1us.
   
  
positive.code: In total there were about 1 million packets. Again, we did not observe any packet loss or out-of-order delivery.
  
  [.text-center]
  .Comparison of round trip time through different switches (35min)
  image::udp/rtt-switch-comparison.png[link="../images/udp/rtt-switch-comparison.png"]
  
  The figure below shows a zoomed view of the time series highlighting the added jitter characteristics. Modes 1 and 3 do not seem to be affected by additional switches. Mode 2 remains constant at 210 us and disappears for higher round trip times, indicating an issue at the receiving step of the sender.
  
  [.text-center]
  .Zoomed in view of switch comparison (10min)
  image::udp/comparison-switch-latency.png[link="../images/udp/comparison-switch-latency.png"]
  
  Both KSZ8863 and the RB750Gr2 add a constant switching latency of 2.9 us and 3.6 us in addition to the wire time of 40.6 us and 4.06 us respectively to the RTT. The added jitter seems to be negligible at well below 1us. Surprisingly, the GS105 seems to have problems with this use case, resulting in higher latency and more jitter than the KSZ8863 even though it was connected using Gigabit. More details are in the table below.
  
negative.code: 
  == Benchmark Setup
  
  A common way to benchmark networks is to setup two computers and have a sender transmit a message to a receiver that echoes it back. That way the sender can measure the https://en.wikipedia.org/wiki/Round-trip_delay_time[round-trip time (RTT)] and gather statistics of the network. This generally works well, but large operating system stacks and device drivers can potentially add a lot of variation. In an attempt to reduce unwanted jitter, we decided to setup a benchmark using two embedded devices instead.
  
  [.text-center]
  .HEBI Robotics I/O Board
  image::udp/io-board.jpg[link="../images/udp/io-board.jpg"]
  
  As the sender and receiver devices we chose to use two HEBI Robotics I/O Boards.  We use them in conjunction with our actuators to interface with external devices, such as a gripper or pneumatic valve, or to get various sensor input into MATLAB. They have 48 pins that serve a variety of functions (analog/digitial I/O, PWM, Encoder input, etc.) that can be accessed remotely via network. Each device contains a 168MHz ARM microcontroller (STM32f407) and a 100 Mbit/s network port.
  
  In order to isolate the network stack we wrote and programmed custom firmware based on http://www.chibios.org/[ChibiOS 2.6.8] and http://savannah.nongnu.org/projects/lwip/[lwIP 1.4.1]. The relevant code pieces can be found https://gist.github.com/ennerf/36a57d432bcff20a58efcdee10f91bd9[here]. The elapsed time was measured using a hardware counter with a resolution of 250ns. 
  
  Since there was no way to store multiple Gigabytes on these devices, we decided to log data remotely using a UDP service that can receive measurement data and persist to disk (see  https://gist.github.com/ennerf/0ddc4396d15852d28e4eca4a8a923eb7[code]). In order to avoid stalls caused by disk I/O, the main socket handler wrote into a double buffered structure that got persisted by a background thread. The synchronization between the threads was done using a http://stuff-gil-says.blogspot.com/2014/11/writerreaderphaser-story-about-new.html[WriterReaderPhaser], which is a synchronization primitive that allows readers to flip buffers while keeping writers wait-free. We found it to be very useful for persisting events that are represented by small amounts of data.
  
  // Since storing multiple Gigabytes of data on an embedded device is challenging, we also setup a https://gist.github.com/ennerf/0ddc4396d15852d28e4eca4a8a923eb7[(Java) UDP server] that can receive measurement data and persist to disk. The main socket handler writes incoming packets into a double buffered structure that can be persisted by a background thread without halting the packet handler. The synchronization between the threads is done using a http://stuff-gil-says.blogspot.com/2014/11/writerreaderphaser-story-about-new.html[WriterReaderPhaser], which is a synchronization primitive that is very useful for persisting events that are represented by a small amount of data.
  
  