anchor.diff:  // GS105: 244.8 us
   // RB750Gr2: 201.5 us
   
  -According to the http://www.downloads.netgear.com/files/GDC/GS105/GS105_datasheet_04Sept03.pdf[GS105 spec sheet], the added network latency should be below 10us (1 Gbit) and 20us (100 Mbit). We did additional tests and it did seem to perform according to spec when using exclusively 100 Mbit/s or 1 Gbit/s on all ports. 
  +According to the http://www.downloads.netgear.com/files/GDC/GS105/GS105_datasheet_04Sept03.pdf[GS105 spec sheet], the added network latency should be below 10us for 1 Gbit/s and 20us for 100 Mbit/s connections. We did additional tests and the GS105 did seem to perform according to spec when using exclusively 100 Mbit/s or 1 Gbit/s on all ports. 
   
   We also conducted another baseline test that replaced the GS105 with a RB750Gr2. While we found a consistent improvement of 0.5us, we did not consider this significant enough to rerun all tests.
   
   == Scaling to Many Devices
   
  -Another major concern about UDP is how performs with multiple devices on the network, rather than single point-to-point connections.
  +So far all tests were measuring the round-trip time between one sender and a single target device. Since real robotic systems can contain many devices, e.g., one per axis or degree of freedom, we also looked at how UDP performs with multiple devices on the same network. In conversations with other researchers we often found an expectation that there would be significant packet loss if multiple packets were to arrive at a Switch at the same time. The worst case would be if all devices were connected to a single Switch as shown below.
   
   [.text-center]
   .Multiple devices connected to a single Switch
   image::udp/setup-bursting.png[link="../images/udp/setup-bursting.png"]
   
  -In order to test the actual behavior we put together a test setup consisting of 40 HEBI Robotics I/O boards that are connected to a  http://www.downloads.netgear.com/files/GDC/GS748Tv1/GS748T_ds_03Feb05.pdf[GS748T] 48-port Ethernet switch. Each board runs the 'receiver' firmware. Sending a single broadcast message to all receiver devices triggers 40x266 byte (>10KB total) return packets that can arrive at the switch within less than 250 nanoseconds. This https://en.wikipedia.org/wiki/Micro-bursting_(networking)[Microburst] traffic pattern is representative of a high degree of freedom system such as a full body humanoid robot without doing any daisy-chaining.
  +In order to test the actual behavior we setup a test consisting of 40 HEBI Robotics I/O boards that were connected to a single 48-port Ethernet Switch (http://www.downloads.netgear.com/files/GDC/GS748Tv1/GS748T_ds_03Feb05.pdf[GS748T]). All devices were running the same (receiver) firmware as before, so sending a single broadcast message triggered 40 response packets that caused more than 10 KB of total traffic to arrive at the Switch within occasionally less than 250 nanoseconds. These https://en.wikipedia.org/wiki/Micro-bursting_(networking)[Microbursts] were well beyond the sustainable bandwidth of Gigabit Ethernet. The setup shown below was representative of a high degree of freedom system such as a full body humanoid robot without daisy-chaining.
   
   [.text-center]
   .Network test setup with 40 HEBI Robotics I/O Boards
   image::udp/multiple-boards.jpg[link="../images/udp/multiple-boards.jpg"]
   
  -This setup also benefits from two convenient side effects of using a standard networking stack to communicate with robotic hardware:
  -
  -1. There is no need for any manual addressing because of https://en.wikipedia.org/wiki/Dynamic_Host_Configuration_Protocol[DHCP] and device specific globally unique mac addresses.
  -
  -2. It takes only 3-6 seconds to re-program the firmware of all 40 devices at once using a bootloader and TCP/IP.
  -
  -In order to keep the sender device from overloading, we changed the sender to a http://www.gigabyte.com/products/product-page.aspx?pid=4888#ov[Gigabyte Brix i7-4770R] computer running https://ennerf.github.io/2016/09/20/A-Practical-Look-at-Latency-in-Robotics-The-Importance-of-Metrics-and-Operating-Systems.html[Scientific Linux 6.6 with real-time kernel] with a disabled firewall. First, we looked at the jitter of the underlying operating system. The figure below shows the difference between an idealized signal that ticks every 10ms and the actual measurements of the loop start times. It shows that this setup suffers from more than an order of magnitude more jitter than the embedded OS. (Note that the corresponding jHiccup control chart looks identical as in the OS blog post.)
  +We would also like to mention that this setup heavily benefited from two side effects that came with using a standard Ethernet stack. First, there was no need for any manual addressing because of https://en.wikipedia.org/wiki/Dynamic_Host_Configuration_Protocol[DHCP] and device specific globally unique mac addresses. Second, we were able to re-program the firmware on all 40 devices simultaneously within 3-6 seconds due to the fact that we had a bootloader with TCP/IP support. It would have been very tedious to setup such a system if any step would have required manual intervention.
  +
  +Since the combined responses result in more traffic than we can handle on the embedded device, we changed the sender to a http://www.gigabyte.com/products/product-page.aspx?pid=4888#ov[Gigabyte Brix i7-4770R] desktop computer running Scientific Linux 6.6 with a real-time kernel. We setup the system as described in https://ennerf.github.io/2016/09/20/A-Practical-Look-at-Latency-in-Robotics-The-Importance-of-Metrics-and-Operating-Systems.html[The Importance of Metrics and Operating Systems] and disabled the firewall. 
  +
  +Running the benchmark at 100Hz for ~90 minutes resulted in more than 20 million measurements.
  +
  +First, we looked at the jitter of the underlying operating system. The figure below shows the difference between an idealized signal that ticks every 10ms and the actual measurements of the loop start times. It shows that this setup suffers from more than an order of magnitude more jitter than the embedded OS. Note that the corresponding jHiccup control chart looks identical as in the OS blog post.
   
   [.text-center]
   .Operating system jitter of Scientific Linux 6.6 with MRG Realtime
   image::udp/os-jitter-linux-rt.png[link="../images/udp/os-jitter-linux-rt.png"]
   
  -Gathering data at 100Hz close to 90 minutes resulted in more than 20 million returned packets. We also ran some tests at 1KHz and the result looks the same.
  -
  -[.text-center]
  -.Receive pattern for incoming datagrams from 40 devices
  +The two figures below show the round-trip time for each measurement. It may be surprising, but there was again no packet loss or re-ordering of packets from any single source. 
  +
  +Rather than packets being dropped, what actually happened was that all packets got stored in the internal buffer of the switch (1.6MB), queued, and forwarded to the target port as fast as possible. Since the original sender was connected via Gigabit, the packets arrived roughly every ~2us. The time axis in the chart is based on the start timestamp, so each cycle shows up as vertical column in the graphs. We also conducted the same test at 1KHz and found identical results.
  +
  +[.text-center]
  +.Zoomed in RTT for 40 devices
   image::udp/rtt-linux-40x-zoomed.png[link="../images/udp/rtt-linux-40x-zoomed.png"]
   
  -It may be surprising, but there was again no packet loss or re-ordering of packets from a single source. Rather than blowing up, what actually happens is that all packets get stored in the internal buffer of the switch (1.6MB), queued, and forwarded to the target port as fast as possible. Since the original sender is connected via 1 Gigabit, the packets arrive every ~2us. All of them have the same start timestamp, so they show up as a vertical column in the graphs.
  -
  -[.text-center]
  -.Round-trip latency for datagrams from 40 devices (90 min)
  +[.text-center]
  +.RTT for 40 devices (90 min)
   image::udp/rtt-linux-40x.png[link="../images/udp/rtt-linux-40x.png"]
   
  -However, what did actually surprise us was how much worse the latency turned out to be compared to the embedded system. We expected most columns to start at around ~180us and end at ~280us. While this is sometimes the case, the majority of columns start above 300 us. After some initial research we suspect that this is related to suboptimal device drivers, and the Linux https://en.wikipedia.org/wiki/New_API[NAPI] using polling mode rather than interrupts. We remember reading about some of this in the past, but we didn't expect this order of magnitude. The installed network interface and driver are below.
  +However, the amount of latency and jitter turned out to be worse than we anticipated. We expected most columns to start at around ~180us and end at ~280us. While this was sometimes the case, the majority of columns started above 300 us. After some initial research we suspected that this delay was mostly caused by the Linux https://en.wikipedia.org/wiki/New_API[NAPI] using polling mode rather than interrupts, and by using a low-cost network interface paired with suboptimal device drivers. While we expected the OS and driver stack to introduce additional latency and jitter, we were surprised by the order of magnitude.
  +
  +The installed network interface and driver are below.
   
   [source,shell]
   $ lspci | grep Ethernet
  
positive.code: 1. Sender wakes up in fixed intervals
  2. Sender measures time and sends packet to receiver
  3. Receiver echoes packet back to sender
  4. Sender receives packet and measures round trip time
  5. Sender sends measurement to logging server
  6. Logging server receives measurement and persists to disk
  
  
negative.code: 
  == Benchmark Setup
  
  A common way to benchmark networks is to setup two computers and have a sender transmit a message to a receiver that echoes it back. That way the sender can measure the https://en.wikipedia.org/wiki/Round-trip_delay_time[round-trip time (RTT)] and gather statistics of the network. This generally works well, but large operating system stacks and device drivers can potentially add a lot of variation. In an attempt to reduce unwanted jitter, we decided to setup a benchmark using two embedded devices instead.
  
  [.text-center]
  .HEBI Robotics I/O Board
  image::udp/io-board.jpg[link="../images/udp/io-board.jpg"]
  
  As the sender and receiver devices we chose to use two HEBI Robotics I/O Boards.  We use them in conjunction with our actuators to interface with external devices, such as a gripper or pneumatic valve, or to get various sensor input into MATLAB. They have 48 pins that serve a variety of functions (analog/digitial I/O, PWM, Encoder input, etc.) that can be accessed remotely via network. Each device contains a 168MHz ARM microcontroller (STM32f407) and a 100 Mbit/s network port.
  
  In order to isolate the network stack we wrote and programmed custom firmware based on http://www.chibios.org/[ChibiOS 2.6.8] and http://savannah.nongnu.org/projects/lwip/[lwIP 1.4.1]. The relevant code pieces can be found https://gist.github.com/ennerf/36a57d432bcff20a58efcdee10f91bd9[here]. The elapsed time was measured using a hardware counter with a resolution of 250ns. 
  
  Since there was no way to store multiple Gigabytes on these devices, we decided to log data remotely using a UDP service that can receive measurement data and persist to disk (see  https://gist.github.com/ennerf/0ddc4396d15852d28e4eca4a8a923eb7[code]). In order to avoid stalls caused by disk I/O, the main socket handler wrote into a double buffered structure that got persisted by a background thread. The synchronization between the threads was done using a http://stuff-gil-says.blogspot.com/2014/11/writerreaderphaser-story-about-new.html[WriterReaderPhaser], which is a synchronization primitive that allows readers to flip buffers while keeping writers wait-free. We found it to be very useful for persisting events that are represented by small amounts of data.
  
  // Since storing multiple Gigabytes of data on an embedded device is challenging, we also setup a https://gist.github.com/ennerf/0ddc4396d15852d28e4eca4a8a923eb7[(Java) UDP server] that can receive measurement data and persist to disk. The main socket handler writes incoming packets into a double buffered structure that can be persisted by a background thread without halting the packet handler. The synchronization between the threads is done using a http://stuff-gil-says.blogspot.com/2014/11/writerreaderphaser-story-about-new.html[WriterReaderPhaser], which is a synchronization primitive that is very useful for persisting events that are represented by a small amount of data.
  
  