anchor.diff:  
   == Benchmark Setup
   
  -A common way to benchmark networks is to setup two computers and have a sender transmit a message to a receiver that echoes it back. That way the sender can measure the round trip time and gather statistics of the network. This generally works well, but large operating system stacks and device drivers can potentially add a lot of variation. In an attempt to reduce unwanted jitter, we decided to setup a benchmark using two embedded devices instead.
  +A common way to benchmark networks is to setup two computers and have a sender transmit a message to a receiver that echoes it back. That way the sender can measure the https://en.wikipedia.org/wiki/Round-trip_delay_time[round-trip time (RTT)] and gather statistics of the network. This generally works well, but large operating system stacks and device drivers can potentially add a lot of variation. In an attempt to reduce unwanted jitter, we decided to setup a benchmark using two embedded devices instead.
   
   [.text-center]
   .HEBI Robotics I/O Board
   image::udp/io-board.jpg[link="../images/udp/io-board.jpg"]
   
  -As the sender and receiver devices we chose to use two HEBI Robotics I/O Boards.  We use them in conjunction with our actuators to interface with external devices, such as a gripper or pneumatic valve, or to get various sensor input into MATLAB. It has 48 pins that serve a variety of functions (analog/digitial I/O, PWM, Encoder input, etc.) that can be accessed remotely via network. It contains a 168MHz ARM microcontroller (STM32f407) and a 100 Mbit/s network port.
  +As the sender and receiver devices we chose to use two HEBI Robotics I/O Boards.  We use them in conjunction with our actuators to interface with external devices, such as a gripper or pneumatic valve, or to get various sensor input into MATLAB. They have 48 pins that serve a variety of functions (analog/digitial I/O, PWM, Encoder input, etc.) that can be accessed remotely via network. Each device contains a 168MHz ARM microcontroller (STM32f407) and a 100 Mbit/s network port.
   
   In order to isolate the network stack we wrote and programmed custom firmware based on http://www.chibios.org/[ChibiOS 2.6.8] and http://savannah.nongnu.org/projects/lwip/[lwIP 1.4.1]. The relevant code pieces can be found https://gist.github.com/ennerf/36a57d432bcff20a58efcdee10f91bd9[here]. The elapsed time was measured using a hardware counter with a resolution of 250ns. 
   
  -Since there was no way to store multiple Gigabytes on these devices, we decided to log data remotely using a UDP service (see  https://gist.github.com/ennerf/0ddc4396d15852d28e4eca4a8a923eb7[code]) that can receive measurement data and persist to disk. In order to avoid stalls caused by disk I/O, the main socket handler wrote into a double buffered structure that got persisted by a background thread. The synchronization between the threads was done using a http://stuff-gil-says.blogspot.com/2014/11/writerreaderphaser-story-about-new.html[WriterReaderPhaser], which is a synchronization primitive that allows readers to flip buffers while keeping writers wait-free. We found it to be very useful for persisting events that are represented by small amounts of data.
  +Since there was no way to store multiple Gigabytes on these devices, we decided to log data remotely using a UDP service that can receive measurement data and persist to disk (see  https://gist.github.com/ennerf/0ddc4396d15852d28e4eca4a8a923eb7[code]). In order to avoid stalls caused by disk I/O, the main socket handler wrote into a double buffered structure that got persisted by a background thread. The synchronization between the threads was done using a http://stuff-gil-says.blogspot.com/2014/11/writerreaderphaser-story-about-new.html[WriterReaderPhaser], which is a synchronization primitive that allows readers to flip buffers while keeping writers wait-free. We found it to be very useful for persisting events that are represented by small amounts of data.
   
   // Since storing multiple Gigabytes of data on an embedded device is challenging, we also setup a https://gist.github.com/ennerf/0ddc4396d15852d28e4eca4a8a923eb7[(Java) UDP server] that can receive measurement data and persist to disk. The main socket handler writes incoming packets into a double buffered structure that can be persisted by a background thread without halting the packet handler. The synchronization between the threads is done using a http://stuff-gil-says.blogspot.com/2014/11/writerreaderphaser-story-about-new.html[WriterReaderPhaser], which is a synchronization primitive that is very useful for persisting events that are represented by a small amount of data.
   
  
positive.code: // GS105: 244.8 us
  // RB750Gr2: 201.5 us
  
  According to the http://www.downloads.netgear.com/files/GDC/GS105/GS105_datasheet_04Sept03.pdf[GS105 spec sheet], the added network latency should be below 10us (1 Gbit) and 20us (100 Mbit). We did additional tests and it did seem to perform according to spec when using exclusively 100 Mbit/s or 1 Gbit/s on all ports. 
  
  We also conducted another baseline test that replaced the GS105 with a RB750Gr2. While we found a consistent improvement of 0.5us, we did not consider this significant enough to rerun all tests.
  
  == Scaling to Many Devices
  
  Another major concern about UDP is how performs with multiple devices on the network, rather than single point-to-point connections.
  
  [.text-center]
  .Multiple devices connected to a single Switch
  image::udp/setup-bursting.png[link="../images/udp/setup-bursting.png"]
  
  In order to test the actual behavior we put together a test setup consisting of 40 HEBI Robotics I/O boards that are connected to a  http://www.downloads.netgear.com/files/GDC/GS748Tv1/GS748T_ds_03Feb05.pdf[GS748T] 48-port Ethernet switch. Each board runs the 'receiver' firmware. Sending a single broadcast message to all receiver devices triggers 40x266 byte (>10KB total) return packets that can arrive at the switch within less than 250 nanoseconds. This https://en.wikipedia.org/wiki/Micro-bursting_(networking)[Microburst] traffic pattern is representative of a high degree of freedom system such as a full body humanoid robot without doing any daisy-chaining.
  
  [.text-center]
  .Network test setup with 40 HEBI Robotics I/O Boards
  image::udp/multiple-boards.jpg[link="../images/udp/multiple-boards.jpg"]
  
  This setup also benefits from two convenient side effects of using a standard networking stack to communicate with robotic hardware:
  
  1. There is no need for any manual addressing because of https://en.wikipedia.org/wiki/Dynamic_Host_Configuration_Protocol[DHCP] and device specific globally unique mac addresses.
  
  2. It takes only 3-6 seconds to re-program the firmware of all 40 devices at once using a bootloader and TCP/IP.
  
  In order to keep the sender device from overloading, we changed the sender to a http://www.gigabyte.com/products/product-page.aspx?pid=4888#ov[Gigabyte Brix i7-4770R] computer running https://ennerf.github.io/2016/09/20/A-Practical-Look-at-Latency-in-Robotics-The-Importance-of-Metrics-and-Operating-Systems.html[Scientific Linux 6.6 with real-time kernel] with a disabled firewall. First, we looked at the jitter of the underlying operating system. The figure below shows the difference between an idealized signal that ticks every 10ms and the actual measurements of the loop start times. It shows that this setup suffers from more than an order of magnitude more jitter than the embedded OS. (Note that the corresponding jHiccup control chart looks identical as in the OS blog post.)
  
  [.text-center]
  .Operating system jitter of Scientific Linux 6.6 with MRG Realtime
  image::udp/os-jitter-linux-rt.png[link="../images/udp/os-jitter-linux-rt.png"]
  
  Gathering data at 100Hz close to 90 minutes resulted in more than 20 million returned packets. We also ran some tests at 1KHz and the result looks the same.
  
  [.text-center]
  .Receive pattern for incoming datagrams from 40 devices
  image::udp/rtt-linux-40x-zoomed.png[link="../images/udp/rtt-linux-40x-zoomed.png"]
  
  It may be surprising, but there was again no packet loss or re-ordering of packets from a single source. Rather than blowing up, what actually happens is that all packets get stored in the internal buffer of the switch (1.6MB), queued, and forwarded to the target port as fast as possible. Since the original sender is connected via 1 Gigabit, the packets arrive every ~2us. All of them have the same start timestamp, so they show up as a vertical column in the graphs.
  
  [.text-center]
  .Round-trip latency for datagrams from 40 devices (90 min)
  image::udp/rtt-linux-40x.png[link="../images/udp/rtt-linux-40x.png"]
  
  However, what did actually surprise us was how much worse the latency turned out to be compared to the embedded system. We expected most columns to start at around ~180us and end at ~280us. While this is sometimes the case, the majority of columns start above 300 us. After some initial research we suspect that this is related to suboptimal device drivers, and the Linux https://en.wikipedia.org/wiki/New_API[NAPI] using polling mode rather than interrupts. We remember reading about some of this in the past, but we didn't expect this order of magnitude. The installed network interface and driver are below.
  
  [source,shell]
  $ lspci | grep Ethernet
  
negative.code: 
  == Conclusion
  
  Even consumer-grade Ethernet networks can exhibit very deterministic performance with regards to latency. In the more than 100 million packets that were sent for this blog post, we also didn't see any packet loss or out-of order delivery. While we still believe that large and dangerous industrial robots should be controlled using specialized industrial networking equipment, we feel that standard UDP is more than sufficient for most robotic applications. 
  
  We found that most of the observed latency and jitter were caused by the underlying operating systems and their device drivers. To further illustrate this point we also compared the baseline setup with the sender running on different operating systems. The configurations were as follows:
  
  * ChibiOS 2.6.8 with lwIP 1.4.1 on 168 MHz STM32F407
  * Windows 10 on Gigabyte Brix-i7-4470R with Realtek NIC
  * Scientific Linux 6.6 with MRG Realtime on Gigabyte Brix-i7-4470R with Realtek NIC
  
  The two charts below show the round trip time for each system communicating with a single device over a single Switch. Note that Linux and Windows were connected to the Switch via Gigabit and should have received datagrams ~40us before the embedded device.
  
  [.text-center]
  .Baseline RTT comparing RTOS vs RT-Linux vs Windows (10h)
  image::udp/rtt-linux-1x-comparison-10h.png[link="../images/udp/rtt-linux-1x-comparison-10h.png"]
  
  [.text-center]
  .Zoomed in baseline RTT comparing RTOS vs RT-Linux vs Windows (10min)
  image::udp/rtt-linux-1x-comparison-10m.png[link="../images/udp/rtt-linux-1x-comparison-10m.png"]
  
  We realize that there are many more interesting questions that would go beyond the scope of this work. We are currently considering the following networking-related topics for future blog posts:
  
  * Comparison of device drivers and network interfaces from various vendors
  * Performance impact of uncontrolled traffic (e.g. streaming video)
  * Controlling through wireless networks
  * Clock drift and time synchronization using IEEE 1588v2
  
  If there are other topics that you think would be worth covering, please leave a note in the comment section. If you are working for a hardware vendor that specializes in low-latency networking equipment and would be willing to provide samples for evaluation, please contact us through our  http://hebirobotics.com/contact/[website].
  
  